Sitelet https://github.com/NodeOps-app/treg2/commit/37e7e78f74796c60c23d80fb6dd92a73973ead0d
Skip to content

Commit 37e7e78

Browse files
committed
Merge remote-tracking branch 'origin/main' into feat/platform-balance
# Conflicts: # src/treg/api.py
2 parents e2550e1 + eba8823 commit 37e7e78

21 files changed

Lines changed: 3857 additions & 57 deletions

‎USAGE.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,107 @@ sudo treg setup-local-run --run-proof "$TREG_RUN_PROOF" # Linux admin, once
178178
treg runs --limit 20
179179
```
180180

181+
## Catching your own calls
182+
183+
`treg run` covers a vendor **CLI**, and `treg call` covers an HTTP call you ask treg to make. Neither
184+
helps when a program makes its own request to `api.stripe.com` — treg is invisible to it and it has no
185+
key.
186+
187+
### The normal way: `treg <command>`
188+
189+
Put `treg` in front of any command:
190+
191+
```bash
192+
pip install "tools-registry[proxy]" # the certificate library; not in the light CLI
193+
194+
treg claude # a Claude Code session using the team's shared credentials
195+
treg codex # same for Codex
196+
treg node server.js # your app, with the team's keys, without holding any
197+
treg python train.py
198+
treg with -- npm test # the explicit form, for anything that confuses the parser
199+
```
200+
201+
**This is opt-in per command, and that is the point.** treg is the parent process, so the setting
202+
applies to that command and its children only. `treg claude` uses the team's shared access; plain
203+
`claude` is completely untouched and uses your own local keys. Nothing is written to any config file,
204+
so there is nothing to undo and no session is ever changed behind your back.
205+
206+
While it runs, an HTTPS call to a **registered** host goes through the registry, which adds the
207+
credential **on the server**. Every other address — including your agent's own calls to
208+
`api.anthropic.com` or `api.openai.com` — goes straight out and cannot be read by us.
209+
210+
If a `treg serve` proxy is already running it is used and left alone; otherwise treg starts a private
211+
one on a port the operating system picks (so two sessions never collide) and stops it when your command
212+
exits.
213+
214+
**One caveat if you write Node.** Node's built-in `fetch` ignores proxy settings until **Node 24**. On
215+
older Node a plain `fetch()` is not captured and the call goes out with no credential. Use a client that
216+
reads the environment (`axios`, `got`, undici's `ProxyAgent`) or upgrade. curl, git, Python
217+
`requests`/`httpx` and Deno work at any version. There is a runnable example in
218+
[`examples/proxy-demo/`](examples/proxy-demo/) that shows both, and explains the workaround for older
219+
Node.
220+
221+
### The subshell: `treg shell --proxy`
222+
223+
`treg run` covers a vendor **CLI**, and `treg call` covers an HTTP call you ask treg to make. Neither
224+
helps when an agent writes its own script that talks to `api.stripe.com` directly — treg is invisible to
225+
it and the script has no key.
226+
227+
`treg shell start --proxy` closes that gap. Inside that shell, an HTTPS call to a **registered** host is
228+
routed through the registry, which adds the credential **on the server** and returns the vendor's answer
229+
unchanged. Your code needs no key and no treg-specific lines:
230+
231+
```bash
232+
pip install "tools-registry[proxy]" # the certificate library; not in the light CLI
233+
treg shell start --proxy # the banner lists the hosts being captured
234+
curl https://api.stripe.com/v1/balance # no key anywhere — treg injected it server-side
235+
python my_script.py # same for anything the script calls
236+
exit # the proxy stops with the shell
237+
```
238+
239+
How it works: the shell sets `HTTPS_PROXY` and a trust bundle **for that shell only**, using a
240+
certificate authority generated on your machine (private key `0600`, valid two years, never shared).
241+
**The system trust store is never modified** — nothing outside that shell trusts it, not your browser
242+
and not the operating system.
243+
244+
What it does **not** touch: every address that is not a registered tool, including your agent's own
245+
calls to `api.anthropic.com` or `api.openai.com`. Those are tunnelled without being read, and no
246+
certificate is ever generated for them.
247+
248+
| Option | What it does |
249+
|---|---|
250+
| `--proxy` | turn it on (off by default) |
251+
| `--proxy-port N` | listen somewhere other than 127.0.0.1:18791 |
252+
| `--renew-ca` | regenerate this machine's certificate authority before starting |
253+
254+
### As a background service (`treg serve`)
255+
256+
If you would rather keep your own shell than enter a subshell, run the same proxy as a service:
257+
258+
```bash
259+
treg serve start # starts in the background, prints the next line for you
260+
eval "$(treg serve env)" # point THIS terminal at it (repeat in any other terminal)
261+
curl https://api.stripe.com/v1/balance
262+
treg serve status # port, team, and the hosts being captured
263+
eval "$(treg serve env --unset)" # stop using it in this terminal
264+
treg serve stop # stop the service
265+
```
266+
267+
`treg serve start --foreground` stays attached instead of detaching, which is what you want for a
268+
service manager or when reading its log (`~/.treg/proxy/serve.log`).
269+
270+
One difference worth knowing. `treg shell --proxy` keeps its access token in the subshell's
271+
environment and both disappear together. A service has to be findable by other terminals, so it writes
272+
its port and token to `~/.treg/proxy/proxy.json` (owner-readable only, mode `0600`). That file holds
273+
the proxy's own token, never a vendor key — but it is a file on disk, which the subshell version does
274+
not have. Choose accordingly.
275+
276+
Limits worth knowing: a client that pins its certificate will refuse the interception (use `treg run` or
277+
`treg call` for that one); every captured call takes one extra hop through the registry, so it fails if
278+
the registry is down; a **hosted** MCP server makes its calls on someone else's machine, so it is not
279+
covered. If a call fails, treg says so in its own words — "no tool is registered for this host", "ask an
280+
admin" — rather than leaving you to read a bare 404 as the vendor's answer.
281+
181282
## Skills (bundles)
182283

183284
| Command | Options | What it does |

‎docs/LOCAL-PROXY-PLAN.md‎

Lines changed: 116 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Local proxy — catch the agent's calls without the agent knowing (`treg serve`)
22

3-
**Status:** planned · **Depends on:** nothing new server-side · **v1 scope:** capture locally, inject on
4-
the server.
3+
**Status:** shipped (P0–P5 + `treg serve`, branch `feat/local-proxy`, 2026-07-30) · **Depends on:** one
4+
small server addition, the `X-Treg-Error` marker · **v1 scope:** capture locally, inject on the server.
55

66
## The problem this solves
77

@@ -56,18 +56,20 @@ is no second policy engine to write — the reason oneCLI needed 24k lines of Ru
5656
enter the base install.
5757

5858
**Decision: a third extra, `[proxy]` → `cryptography` only.** The proxy itself uses `asyncio` + `ssl`
59-
from the standard library and `httpx` (already base). `treg serve` without the extra exits with a clear
60-
install hint. `[server]` already includes `cryptography`, so a self-hoster gets it free.
59+
from the standard library and `httpx` (already base). Without the extra the proxy raises
60+
`ProxyDependencyError` with the exact install line. `[server]` already includes `cryptography`, so a
61+
self-hoster gets it free. **Shipped** — see `[project.optional-dependencies].proxy` in `pyproject.toml`.
6162

6263
## Components
6364

6465
| Piece | Where | Notes |
6566
|---|---|---|
6667
| Proxy server | `src/treg/localproxy.py` (new) | named for `localrun.py`; **not** `proxy.py`, which is the server relay |
6768
| CA + leaf certs | same module | ECDSA P-256, leaf cached per host in memory |
68-
| CLI command | `cmd_serve` in `cli.py` | `treg serve [--port] [--stop] [--status] [--print-env]` |
69-
| Shell wiring | `shell.py` | `treg shell` starts the proxy and sets the env in the subshell |
70-
| State | `~/.treg/proxy/` | `ca-key.pem` 0600 · `ca-cert.pem` 0644 · `ca-bundle.pem` 0644 · `proxy.json` |
69+
| CLI flags | `cmd_shell_start` in `cli.py` | `treg shell start --proxy [--proxy-port] [--renew-ca]` |
70+
| Daemon | `cmd_serve_*` in `cli.py` | `treg serve start\|stop\|status\|env` — `eval "$(treg serve env)"` |
71+
| Shell wiring | `shell.py` | `start_session(extra_env=…, on_close=…)` publishes the env, stops the proxy |
72+
| State | `~/.treg/proxy/` | `ca-key.pem` 0600 · `ca-cert.pem` 0644 · `ca-bundle.pem` 0644 · `proxy.json` 0600 (**daemon only** — port + proxy token, so another terminal can find it; `--proxy` writes none of it) · `serve.log`. Follows `TREG_CONFIG` when set |
7173

7274
### The environment we set
7375

@@ -110,28 +112,35 @@ system roots; append to them.
110112

111113
## Phases — each independently testable
112114

113-
**P0 · Skeleton (no interception).** Listen, accept `CONNECT`, blind-tunnel everything both ways.
114-
*Proof:* `curl -x http://127.0.0.1:PORT https://example.com` works exactly as without the proxy.
115-
116-
**P1 · Certificates.** Generate + persist the CA, build the bundle, generate and cache leaf certs.
117-
`treg serve --print-env`.
118-
*Proof:* a unit test signs a leaf for `example.com` and validates it against the CA; `curl
119-
--cacert ca-bundle.pem` through the proxy trusts an intercepted host.
120-
121-
**P2 · Intercept and forward.** For allow-listed hosts: terminate TLS, read the request, re-address to
122-
`{base}/call/https://{host}{path}`, add `X-Treg-Token` / `X-Treg-Org` / `X-Treg-Client`, stream the
123-
response back.
124-
*Proof:* a registered tool answers correctly with **no key anywhere on the machine**.
125-
126-
**P3 · Allow-list + errors.** Fetch hosts from `GET /tools` at start, refresh on a timer. Map treg's
127-
failures to something readable: a 404 from `_resolve_call` means "no tool registered for this host or
128-
path"; a 403 means "you do not have access"; treg unreachable must not look like the vendor being down.
129-
130-
**P4 · Wiring UX.** `treg shell` starts the proxy and exports the env for the subshell — it already runs
131-
a controlled subshell, so this is the natural home. Standalone `treg serve` for people who want it in
132-
their own shell.
133-
134-
**P5 · Docs.** `docs/context/architecture/local-proxy.md` fragment (new subsystem, so it needs one),
115+
**P0 · Skeleton (no interception). — DONE.** Listen, authenticate, blind-tunnel everything both ways.
116+
*Proof:* `curl -x http://treg:<token>@127.0.0.1:PORT https://example.com` works exactly as without the
117+
proxy (verified: 200, `ssl_verify_result=0`); without the token it is a 407.
118+
119+
**P1 · Certificates. — DONE.** Generate + persist the CA, build the bundle, sign and cache leaf certs
120+
(`ensure_ca`, `build_bundle`, `CertAuthority.leaf_pem` / `context_for`), plus `proxy_env()`, the exact
121+
environment P4 will publish.
122+
*Proof:* unit tests sign a leaf and complete a real TLS handshake against it; `curl --cacert
123+
ca-bundle.pem` accepts it, and curl with the system roots alone rejects it.
124+
125+
**P2 · Intercept and forward. — DONE.** For allow-listed hosts: terminate TLS (`_intercept`), read the
126+
request, re-address to `{base}/call/https://{host}{path}` (`_treg_url`), swap the caller's transport and
127+
`x-treg-*` headers for our own (`_forward_headers`), and stream the answer back (`_write_response`).
128+
Keep-alive is honoured, so several calls share one handshake.
129+
*Proof:* `test_the_agent_calls_the_vendor_and_treg_injects_the_key` runs a plain `httpx.Client` — no treg
130+
awareness, no key — through the proxy against the **real** FastAPI app; the credential arrives at the
131+
upstream because the server injected it.
132+
133+
**P3 · Allow-list + errors. — DONE.** Fetch hosts from `GET /tools` at start, refresh on a timer — `ProxyConfig.hosts`
134+
is the seam, currently filled by the caller. Map treg's failures to something readable: a 404 from
135+
`_resolve_call` means "no tool registered for this host or path"; a 403 means "you do not have access".
136+
(The treg-unreachable case is already handled — a 502 that names treg and says the vendor call was **not**
137+
made, so an agent does not blame a healthy vendor and retry.)
138+
139+
**P4 · Wiring UX. — DONE.** `treg shell start` mints a token, starts the proxy and merges `handle.env(treg_host)`
140+
into the subshell environment (`start_session` already builds that dict), stopping it on teardown. The
141+
banner says which hosts are captured. No standalone command — decision 1.
142+
143+
**P5 · Docs. — DONE.** `docs/context/architecture/local-proxy.md` fragment (new subsystem, so it needs one),
135144
plus `/llms.txt` and the dashboard's agent instructions.
136145

137146
## Testing
@@ -162,12 +171,81 @@ plus `/llms.txt` and the dashboard's agent instructions.
162171
Local injection under `treg-run` · system trust store install · Windows · WebSocket interception ·
163172
per-agent proxy tokens (one per session is enough) · request/response inspection beyond routing.
164173

165-
## Open decisions
166-
167-
1. **Command shape** — `treg serve` as its own daemon, or only inside `treg shell`? (I suggest both:
168-
`serve` is the engine, `shell` is the convenient front door.)
169-
2. **CA lifetime** — oneCLI uses 10 years. Shorter is safer but expires; suggest 2 years plus
170-
`treg serve --renew-ca`.
171-
3. **Default port.**
172-
4. **Ship `[proxy]` as an extra** (recommended) or fold `cryptography` into base (rejected: breaks the
173-
light CLI).
174+
## Decisions (settled 2026-07-30, with Unclecode)
175+
176+
1. **Command shape — `treg shell start --proxy` first, `treg serve` added after.** The original
177+
decision was shell-only, because the proxy then starts and dies with a session and its token never
178+
has to be written to disk. After testing it live, Unclecode asked for the daemon too, so both
179+
shipped: `--proxy` for a subshell, `treg serve start|stop|status|env` for a background service that
180+
other terminals point at with `eval "$(treg serve env)"`. The daemon's cost is exactly the thing the
181+
first decision avoided — a `proxy.json` (0600) holding the port and the proxy token — and that is
182+
stated in USAGE so the choice is informed. `_start_proxy_handle` is the single shared code path.
183+
2. **CA lifetime — 2 years**, regenerated automatically inside its last 30 days
184+
(`_RENEW_WITHIN_DAYS`), plus an explicit renew. Not oneCLI's 10 years: a key sitting on a laptop
185+
for a decade is a long time for something that can impersonate any site.
186+
3. **Default port — 18791**, next to the dev server's 18790.
187+
4. **`[proxy]` extra — yes**, `cryptography` alone. Folding it into base was rejected: it is compiled,
188+
and `pip install tools-registry` must stay the light CLI.
189+
190+
## Build log
191+
192+
- **P0 + P1 shipped** (2026-07-30) — `src/treg/localproxy.py`, `tests/test_localproxy.py` (34 tests).
193+
Verified with a real `curl`: `https://example.com` and `https://api.github.com` tunnel through
194+
untouched with certificate verification intact; a leaf signed by the CA is accepted by
195+
curl/OpenSSL when pointed at `ca-bundle.pem` and **rejected** with system roots only — the proof
196+
that non-negotiable #2 holds.
197+
- The `_is_self` loop guard and the plain-`http://` forward path (which strips `Proxy-Authorization`
198+
before the upstream sees it) were added beyond the P0 sketch — `HTTP_PROXY` is set alongside
199+
`HTTPS_PROXY`, so an http caller must not simply break.
200+
- **P2 shipped** (2026-07-30) — 21 more tests, 55 in the file. Beyond the sketch:
201+
- **The edge-WAF retry.** Render's edge 403s a body that looks like injection. The CLI's
202+
`_RegistryClient` already re-sends such a body base64-encoded under `X-Treg-Body-Encoding`; an
203+
intercepted call has to do the same, or a legitimate SQL query fails invisibly. Bodies up to 1 MiB
204+
are buffered so a retry is possible at all; larger ones stream and cannot be retried.
205+
- **`_CALLER_MUST_NOT_SET`.** An agent that sets its own `X-Treg-Token` must not be able to spend
206+
another member's quota through our proxy. The caller's `x-treg-*` headers are dropped; the proxy's
207+
identity is the only one that reaches treg.
208+
- **`intercepts()` refuses without a credential.** With no CA or no treg token we would terminate TLS
209+
and then have no way to finish the call — worse than not intercepting, because the agent's own
210+
request breaks for a reason it cannot see.
211+
- **`_body_chunks`.** `aiter_raw()` raises `StreamConsumed` on a response a transport already loaded,
212+
which would silently drop the body. Both shapes are handled.
213+
- **Framing is re-derived, not copied** — `Content-Length` when treg gave one, chunked when it did
214+
not — because our hop must be self-consistent even when the hop into treg was framed differently.
215+
216+
- **`treg serve` shipped** (2026-07-30, after the live test) — `cmd_serve_start/_stop/_status/_env` +
217+
the state helpers in `localproxy.py` (`write_state`, `read_state`, `running`, `pid_alive`). Two
218+
findings the tests produced, both real:
219+
- `pid_alive(0)` returned **True**, because `os.kill(0, …)` signals the caller's whole process
220+
group. A truncated pid in the state file would have read as "running", and `treg serve stop` would
221+
have signalled the user's terminal. Non-positive pids are now rejected before the call.
222+
- The parent said only "see the log" when the detached child died. It now prints the child's last
223+
log line — which is how the port-collision case reads as a sentence instead of a scavenger hunt.
224+
- Verified live end to end: `serve start` → `eval "$(treg serve env)"` → a plain `curl` to a captured
225+
host arrived at the vendor with `Authorization: Bearer …` injected server-side, an uncaptured host
226+
went straight out, `--unset` cleared the shell, a second `start` was refused, `stop` removed the
227+
state file, and a stale file with a dead pid was cleaned up on the next call.
228+
229+
- **Agent hook shipped** (2026-07-30) — `treg serve hook [--install] [--agent]`. Unclecode asked how
230+
oneCLI avoids the eval; reading their code, it does not: `onecli run` is the parent, containers get
231+
`-e HTTPS_PROXY=…`, and their Claude plugin sets **`BASH_ENV`** to a generated `env.sh`
232+
(`plugins/claude/hooks/session-start.mjs:236`). We now do the third one, reusing the harness
233+
registry in `agents.py`. Proven with a bare `BASH_ENV=… bash -c 'curl …'`: credentialed while the
234+
proxy runs, plain after `serve stop`.
235+
- **The hook was replaced by `treg <command>`, same day.** Unclecode rejected the global installer, and
236+
he was right: writing `BASH_ENV` into `~/.claude/settings.json` captures every session of that agent
237+
on the machine, forever, whether or not you wanted treg that day — and leaves no easy way to use your
238+
own personal key instead. The launcher (`cmd_with`) gives the same "no eval ever" result with none of
239+
the reach: treg is the parent of the one command it runs. `treg claude` uses team access; plain
240+
`claude` is untouched. Nothing is written to any config file. `treg serve hook`, `env.sh` and the
241+
three config installers were deleted — do not re-add them.
242+
- **Next, requested but not built:** `treg claude` could also pre-load the team's skills into the
243+
session instead of the agent downloading and installing them.
244+
- **Measured, and it corrects the plan:** `NODE_USE_ENV_PROXY` only exists from **Node 24**. On the dev
245+
machine (Node 23.11) `node --use-env-proxy` is a "bad option" and a plain `fetch()` through the proxy
246+
arrives uncredentialed. Setting the variable is still right for Node 24+, but the plan's line about it
247+
read as "Node works now", which is only true on 24. Documented in the fragment, USAGE and /llms.txt.
248+
- **`examples/proxy-demo/`** — a zero-dependency Node app with a small web UI: click a button, it calls
249+
`api.openai.com` (registered) or `example.com` (not). `node server.js` gets a 401; `treg node
250+
server.js` gets the real answer, same code. It speaks CONNECT by hand because of the Node caveat
251+
above, and its README says why real apps do not need to.

0 commit comments

Comments
 (0)