Sitelet https://github.com/NodeOps-app/treg2/commit/2730691bbeca61062c5c893657ada7b0950d985c
Skip to content

Commit 2730691

Browse files
committed
feat(archive): key own-credential answers by org or connection; sharing is an endpoint judgment
Storage and sharing are now separate dimensions. The licence still decides whether bytes are kept; `archive.sharing` decides whose question the answer is, and the answer is confined by the KEY it is recorded under rather than by a read-time filter: - treg's platform key: public, as before. - the org's own credential (API key or OAuth token, metered or not) on an `any_account` endpoint: an org-scoped key (`org:<id>` folded into the hash); the caller reads its org key first, then the public one, so a platform fetch still serves an own-key caller free. - on an `own_account` endpoint: a connection-scoped key (`conn:<org>:<bound secret ids>`), only that key is consulted - two Google accounts in one team never see each other's sites, a reconnect starts a fresh history, and pre-existing public rows are never served. - crossing teams requires the ENDPOINT to declare `cache.sharing: public`; the store refuses it on a provider header, on an `own_account` endpoint, and any value but `public`. A provider's storage licence no longer implies sharing (`judged_storable` and the `own_key_scoped` miss are gone). `ArchiveKey.scope` (migration 0034, still unreleased) lets the refresh worker skip private keys; `tool_called` carries `cache_sharing` and, on a hit, `cache_scope`. Fragments: archive ("Sharing"), data-model; AGENTS.md, SECURITY.md.
1 parent e9952ca commit 2730691

11 files changed

Lines changed: 440 additions & 174 deletions

File tree

‎AGENTS.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -111,11 +111,12 @@ agents then built against a constitution that was wrong.
111111
- **Money.** Everything is **integer micro-USD** - never floats, never cents. The Stripe SDK lives
112112
only in `infra/stripe.py`, orchestration in `application/billing.py`, and `reconcile.py` is
113113
read-only. See `docs/context/architecture/money.md`.
114-
- **The archive serves every tier.** Own-key answers are recorded (bounded read, never a prefix)
115-
and a hit on an own key is free; a metered hit settles through the same hold, at
116-
`archive_hit_repeat_price_percent` once the team has paid for that question. An own-key answer
117-
crosses teams only where the provider's licence was judged. See
118-
`docs/context/architecture/archive.md`.
114+
- **The archive serves every tier, keyed by whose question it is.** Own-credential answers are
115+
recorded (bounded read, never a prefix) under an org-scoped key, or a connection-scoped key on
116+
an `own_account` endpoint, and reach other teams only where the endpoint itself declares
117+
`cache.sharing: public`; a provider's storage licence never decides that. A hit on an own key
118+
is free; a metered hit settles through the same hold, at `archive_hit_repeat_price_percent`
119+
once the team has paid for that question. See `docs/context/architecture/archive.md`.
119120

120121
### Security guards that look redundant on purpose
121122

‎SECURITY.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,10 @@ the team's CallRecord before resolving its archive reference. A hash is an ident
6262
access credential. Upstream answers are retained under the existing archive policy. A provider
6363
that echoes treg's PLATFORM credential in a 2xx body is not sanitized by these guards (judge that
6464
provider's `cache` with it in mind); an answer that echoes a team's OWN credential is refused by
65-
the recorder (`_echoes_own_credential`) so it can never be served to another team.
65+
the recorder (`_echoes_own_credential`). An own-credential answer is stored under a cache key that
66+
folds in the org (or the connection, on an `own_account` endpoint), so another team never computes
67+
the hash that would find it; only an endpoint-level `cache.sharing: public` declaration puts such an
68+
answer on the public key.
6669

6770
Object I/O never holds the calling task's database connection. Startup refuses incomplete R2
6871
configuration when archive mode is enabled and any storage switch selects R2. The manual smoke

‎docs/context/architecture/archive.md‎

Lines changed: 49 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -74,20 +74,44 @@ compresses anyway is not recorded (`_identity_encoded`). A caller body the own-k
7474
read (streamed, no small `Content-Length`) cannot key the question: no lookup, no recording.
7575
Own-TOOL calls (no catalog entry) stay untouched. **An answer that quotes the team's own
7676
credential back is never recorded** (`_echoes_own_credential`: the exact renderings the error
77-
masking uses, failing closed when they cannot be rendered) — on a judged provider it would be
78-
served to another team. The same guard and the same `origin_org_id` apply to a metered call on an
79-
org credential that rides treg's pay-per-use OAuth app (`billed_oauth`): the token is the team's.
80-
81-
The snapshot carries `origin_org_id` (migration 0034; NULL = treg's platform key). Who it may
82-
serve is decided at READ time in `lookup`: its own team always; another team only when the
83-
provider's licence was JUDGED to allow storage (`judged_storable`: an explicit
84-
`cache: transient|archive`, never the unjudged default) — the data was fetched under that team's
85-
vendor contract. Otherwise the miss is `own_key_scoped` and the vendor answers; the platform's
86-
own recording of that answer then serves everyone. A platform-key snapshot serves own-key
87-
callers too. **A hit on an own key is free**: nothing is reserved or settled (non-negotiable 1),
88-
the response carries the same `X-Treg-Cache: hit` headers and the audit row `cached: true`, and
89-
`cache_price` reports `free`. Own-key observations train the timer and the result state like any
90-
other; the refresh worker still re-asks on treg's platform key.
77+
masking uses, failing closed when they cannot be rendered). The same guard, the same
78+
`origin_org_id` provenance and the same sharing rules apply to a metered call on an org
79+
credential that rides treg's pay-per-use OAuth app (`billed_oauth`): the token is the team's.
80+
81+
**A hit on an own key is free**: nothing is reserved or settled (non-negotiable 1), the response
82+
carries the same `X-Treg-Cache: hit` headers and the audit row `cached: true`, and `cache_price`
83+
reports `free`. Own-key observations train the timer and the result state of THEIR key like any
84+
other.
85+
86+
### Sharing: whose question is it (2026-09-15)
87+
88+
Storage and sharing are two dimensions, judged separately. The licence (`cache.mode`) says
89+
whether bytes may be KEPT; `archive.sharing(entry, own_credential=…)` says whose question the
90+
answer is — "the vendor allows storage" never implies "the answer does not depend on who asked".
91+
92+
| Who asked | `scope: any_account` | `scope: own_account` |
93+
|---|---|---|
94+
| treg's platform key | `public` | (never: own_account needs the caller's credential) |
95+
| the org's own credential (API key or OAuth token, metered or not) | `org`, or `public` only where the ENDPOINT declares `cache.sharing: public` | `connection` |
96+
97+
Sharing is enforced by the KEY, not by a filter at read time: `scope_tags` folds `org:<id>` or
98+
`conn:<id>:<sorted bound secret ids>` into `cache_key`, so a private history is under a hash
99+
nobody else ever computes, its timer learns only from its own answers, and two Google accounts
100+
in one team never see each other's sites (a reconnect is a new secret, hence a fresh history).
101+
A caller consults its scopes most specific first — connection only; org then public (a
102+
platform-key fetch may serve an own-key caller free); public only — and records under the first.
103+
`ArchiveKey.scope` (`org` | `conn` | NULL = public, including every key from before the column)
104+
lets the refresh worker skip private keys: treg's platform key cannot re-ask them. History from
105+
before this rule (platform and billed-OAuth rows, all NULL-origin, all on public keys) is
106+
therefore never served to an `own_account` caller and only ever served on `any_account`
107+
endpoints, where it was a public question anyway. `cache_sharing` and, on a hit, `cache_scope`
108+
are on `tool_called`.
109+
110+
`cache.sharing: public` is an ENDPOINT declaration (`_validate_cache` refuses it on a provider
111+
header, refuses any value but `public`, and refuses it on an `own_account` endpoint): it says
112+
this endpoint's answer is identical whoever asks — treg's own service OAuth reading public data
113+
is the intended case — and it is judged per endpoint, never inherited from a provider's licence.
114+
Nothing in the shipped catalog declares it yet.
91115

92116
## Pricing a hit (2026-09-14)
93117

@@ -528,7 +552,7 @@ produce hypothetical hit counts or fresh-answer comparisons.
528552
(`_buffer_response` needs the provider's reported cost), so recording adds no latency. An
529553
own-key catalog answer is read whole only when it fits the archive's size cap (see "Own-key
530554
answers"); larger ones stream untouched. Own-tool calls (no catalog entry) are never touched.
531-
Who an own-key answer may serve is decided at READ time by `origin_org_id`, not at write time.
555+
Who an own-credential answer may serve is decided by the KEY it is recorded under ("Sharing").
532556

533557
Gates 1+2 are `archive.policy(entry)`; gate 3 is the hook site's own context.
534558

@@ -539,10 +563,11 @@ buffer's 8 MiB limit fail before recording and cannot populate a cache or idempo
539563

540564
## The cache key
541565

542-
`archive.cache_key(method, endpoint_id, upstream_url, body, headers)` → sha256 over the canonical
543-
request: uppercased method, catalog endpoint id (a provider URL reshuffle starts a fresh history),
544-
sorted query pairs, canonical-JSON body hash (raw hash for non-JSON), plus only `Accept` and
545-
`Accept-Language` from the caller's headers. Auth/cookies/tracing/encodings never enter the key —
566+
`archive.cache_key(method, endpoint_id, upstream_url, body, headers, scope="")` → sha256 over the
567+
canonical request: uppercased method, catalog endpoint id (a provider URL reshuffle starts a fresh
568+
history), sorted query pairs, canonical-JSON body hash (raw hash for non-JSON), plus only `Accept`
569+
and `Accept-Language` from the caller's headers, plus the sharing scope when the answer is not
570+
public (`org:<id>` / `conn:<id>:<secret ids>`, see "Sharing"; a public key hashes as it always did). Auth/cookies/tracing/encodings never enter the key —
546571
and credentials could not anyway: injection happens after the key is taken.
547572

548573
## Tables (migration 0002)
@@ -551,9 +576,11 @@ and credentials could not anyway: injection happens after the key is taken.
551576
`policy`, AIMD timer state (`ttl_s`, grow ×1.5 capped on stable refetch / shrink ×0.5 floored on
552577
change — the learner lands in PR 5), change statistics (`change_seen`/`stable_seen`/
553578
`last_changed_at`), legacy `volatile_paths` (retained for schema compatibility, no longer read or updated), and demand (`heat`, `last_requested_at`). Platform-scoped, no `org_id`:
554-
one team's fetch may warm another team's hit; an own-key answer's reach is the SNAPSHOT's
555-
`origin_org_id` (migration 0034), and `ArchiveKeyOrg` (same migration) is the per-(org, key)
556-
"has paid for this question" mark that prices a repeat hit — see "Pricing a hit".
579+
one team's fetch may warm another team's hit; an own-credential answer's reach is its KEY's
580+
scope (`ArchiveKey.scope`, migration 0034, and the scope folded into the hash — see "Sharing"),
581+
the snapshot's `origin_org_id` (same migration) is provenance, and `ArchiveKeyOrg` (same
582+
migration) is the per-(org, key) "has paid for this question" mark that prices a repeat hit —
583+
see "Pricing a hit".
557584

558585
`ArchiveSnapshot` — one version: unique `(key_id, version)`, verbatim `body` bytes, `content_hash`
559586
(raw sha256) for dedup — an identical consecutive answer stores a version row with `body=NULL,

‎docs/context/architecture/data-model.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -83,10 +83,14 @@ verified upload finishes outside any DB session; `content_hash` is the object na
8383
backfill, body-column removal or destructive migration occurs. Double-write rows retain their DB
8484
body/carrier; R2-only rows require no carrier pointer. See [archive](archive.md#body-storage-and-r2-double-writing).
8585
Migration `0034` adds nullable `ArchiveSnapshot.origin_org_id` (the team whose own credential
86-
fetched the answer; NULL = treg's platform key, every row before it) and the `ArchiveKeyOrg`
87-
table, unique on `(org_id, key_hash)`: which teams have paid for which archived question, written
88-
by archive inside the metered settle transaction and read by lookup to price a repeat hit. See
89-
[archive](archive.md#own-key-answers-2026-09-14) and [pricing a hit](archive.md#pricing-a-hit-2026-09-14).
86+
fetched the answer; NULL = treg's platform key, every row before it - provenance), nullable
87+
`ArchiveKey.scope` (`org` | `conn` for a key private to an org or a connection; NULL = public,
88+
every key before it - the sharing scope is also folded into the key hash) and the
89+
`ArchiveKeyOrg` table, unique on `(org_id, key_hash)`: which teams have paid for which archived
90+
question, written by archive inside the metered settle transaction and read by lookup to price
91+
a repeat hit. See [archive](archive.md#own-key-answers-2026-09-14),
92+
[sharing](archive.md#sharing-whose-question-is-it-2026-09-15) and
93+
[pricing a hit](archive.md#pricing-a-hit-2026-09-14).
9094

9195
Revision `0033` adds nullable `User.email_verified_at` and non-null `signup_promo_available`,
9296
with a retained database default of false for existing rows and old writers. New application users

‎src/treg/alembic/versions/0034_archive_own_key_and_repeat_pricing.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@
1111
def upgrade() -> None:
1212
# NULL = fetched on treg's platform key (every row before this revision).
1313
op.add_column("archivesnapshot", sa.Column("origin_org_id", sa.Integer(), nullable=True))
14+
# "org" | "conn" for a key private to an org or a connection; NULL = public (every key
15+
# before this revision, whose hash carries no scope).
16+
op.add_column("archivekey", sa.Column("scope", sa.String(), nullable=True))
1417
op.create_table(
1518
"archivekeyorg",
1619
sa.Column("id", sa.Integer(), primary_key=True),
@@ -29,4 +32,5 @@ def downgrade() -> None:
2932
op.drop_index("ix_archivekeyorg_key_hash", table_name="archivekeyorg")
3033
op.drop_index("ix_archivekeyorg_org_id", table_name="archivekeyorg")
3134
op.drop_table("archivekeyorg")
35+
op.drop_column("archivekey", "scope")
3236
op.drop_column("archivesnapshot", "origin_org_id")

‎src/treg/application/call/service.py‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -881,6 +881,18 @@ def _audit(status_code: int, *, observed_micro: int | None = None, charged_micro
881881
own_key_archivable = (
882882
own_key_cacheable and archive.recording()
883883
and archive.storable(catalog_store.load().by_id.get(mk.endpoint_id)))
884+
# Whose question this is (archive.sharing): the org's own credential - an API key or
885+
# an OAuth token, metered or not - confines the answer to the org, or to the
886+
# connection on an `own_account` endpoint, by keying it. The tags are the scopes this
887+
# caller may READ, most specific first; the first is where its answer is recorded.
888+
own_credential = mk is not None and mk.tier in ("tool", "credential")
889+
cache_scopes = archive.scope_tags(
890+
archive.sharing(catalog_store.load().by_id.get(mk.endpoint_id) if mk else None,
891+
own_credential=own_credential),
892+
caller.org_id, secrets) if mk is not None else [""]
893+
if mk is not None:
894+
cache_diagnostics["cache_sharing"] = archive.sharing(
895+
catalog_store.load().by_id.get(mk.endpoint_id), own_credential=own_credential)
884896
# A probe must reach the vendor: an archived answer proves nothing about capacity.
885897
if (mk is not None and mk.probe_lock_id is None and archive.serving()
886898
and ((mk.metered and not mk.streamable_free_result) or own_key_cacheable)):
@@ -893,7 +905,7 @@ def _audit(status_code: int, *, observed_micro: int | None = None, charged_micro
893905
drop_params or set()),
894906
caller_body=caller_body, request_headers=request.headers,
895907
cohort=str(audit_org_id), diagnostics=cache_diagnostics,
896-
org_id=caller.org_id, price_repeat=mk.metered)
908+
org_id=caller.org_id, price_repeat=mk.metered, scopes=cache_scopes)
897909
except Exception: # noqa: BLE001 — lookup swallows internally; this catches even a
898910
served = None # fault in its own plumbing. Cache trouble must cost a vendor
899911
# call, never a 500.
@@ -959,7 +971,7 @@ def _audit(status_code: int, *, observed_micro: int | None = None, charged_micro
959971
# in memory here for the settle, so observing it costs nothing on-request. Metered
960972
# 2xx only — gate 3 of eligibility is exactly 'this fact, at this line'. Off unless
961973
# TREG_ARCHIVE_MODE says otherwise; record() is fire-and-forget and never raises.
962-
own_credential = mk.tier in ("tool", "credential") # billed OAuth: the org's token
974+
# `own_credential` here means billed OAuth: the org's token, treg's bill.
963975
if (mk.metered and archive.recording() and 200 <= response.status < 300
964976
and not (own_credential and _echoes_own_credential(tool, secrets, body))):
965977
_ct = next((v.decode("latin-1") for k, v in response.raw_headers
@@ -976,7 +988,8 @@ def _audit(status_code: int, *, observed_micro: int | None = None, charged_micro
976988
headers={k: request.headers.get(k, "") for k in ("accept", "accept-language")},
977989
status_code=response.status, media_type=_ct, body=body,
978990
observation=body_observation,
979-
origin_org_id=caller.org_id if own_credential else None)
991+
origin_org_id=caller.org_id if own_credential else None,
992+
scope=cache_scopes[0])
980993
elif (served is None and own_key_archivable and 200 <= response.status < 300
981994
and _identity_encoded(response)):
982995
# An own-key answer: read whole when it fits the archive's cap (bounded, before
@@ -999,7 +1012,8 @@ def _audit(status_code: int, *, observed_micro: int | None = None, charged_micro
9991012
caller_body=caller_body,
10001013
headers={k: request.headers.get(k, "") for k in ("accept", "accept-language")},
10011014
status_code=response.status, media_type=_ct, body=body,
1002-
observation=body_observation, origin_org_id=caller.org_id)
1015+
observation=body_observation, origin_org_id=caller.org_id,
1016+
scope=cache_scopes[0])
10031017
elif response.status >= 400:
10041018
# Preserve streaming for own-key and own-tool calls while retaining only the small
10051019
# diagnostic head. The replacement response replays every consumed byte verbatim.

0 commit comments

Comments
 (0)