Sitelet https://github.com/NodeOps-app/treg2/commit/9f671301694d02a6584238f7db9c3ff7cc3e1461
Skip to content

Commit 9f67130

Browse files
stonexerstonexer
committed
feat(archive): add R2 body storage and gated double writes
1 parent 2ec1006 commit 9f67130

26 files changed

Lines changed: 1543 additions & 81 deletions

‎.agents/skills/tools-registry-context/MAP.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ Regenerate via `scripts/build-map.py`.
4040
| `scripts/indexnow_submit.py` | interface/seo.md |
4141
| `scripts/minimax_plugin.py` | interface/skill.md |
4242
| `scripts/provider_balances.py` | ops/capacity.md |
43+
| `scripts/smoke_archive_r2.py` | architecture/archive.md |
4344
| `src/treg/__main__.py` | ops/deploy.md |
4445
| `src/treg/adsconv.py` | architecture/ads-conversions.md |
4546
| `src/treg/agent_pages.py` | interface/seo.md |
@@ -68,6 +69,7 @@ Regenerate via `scripts/build-map.py`.
6869
| `src/treg/alembic/versions/0025_feedback.py` | architecture/feedback.md |
6970
| `src/treg/alembic/versions/0026_callreview.py` | architecture/feedback.md |
7071
| `src/treg/alembic/versions/0027_enrich_arena.py` | architecture/data-model.md, interface/enrich-arena.md |
72+
| `src/treg/alembic/versions/0028_archive_body_storage.py` | architecture/archive.md, architecture/data-model.md |
7173
| `src/treg/alembic/versions/0028_arena_insights.py` | architecture/data-model.md, interface/enrich-arena.md |
7274
| `src/treg/alembic/versions/0029_arena_verification_snapshot.py` | architecture/data-model.md, interface/enrich-arena.md |
7375
| `src/treg/alembic/versions/0030_feedback_handling.py` | architecture/feedback.md |
@@ -107,6 +109,7 @@ Regenerate via `scripts/build-map.py`.
107109
| `src/treg/application/referrals.py` | architecture/money.md, interface/api.md |
108110
| `src/treg/application/signup.py` | architecture/ads-conversions.md, architecture/money.md, architecture/multi-tenancy.md, interface/api.md |
109111
| `src/treg/archive.py` | architecture/archive.md |
112+
| `src/treg/archive_bodies.py` | architecture/archive.md |
110113
| `src/treg/audit.py` | architecture/data-model.md, ops/deploy.md |
111114
| `src/treg/bootstrap.py` | architecture/archive.md, architecture/composition.md, interface/enrich-arena.md |
112115
| `src/treg/bootstrap_handlers.py` | architecture/composition.md, architecture/data-model.md, interface/api.md |
@@ -205,7 +208,7 @@ Regenerate via `scripts/build-map.py`.
205208
| `src/treg/cli.py` | architecture/instagram-oauth.md, interface/cli.md, interface/onboarding.md, interface/shell.md |
206209
| `src/treg/cli_analytics.py` | interface/cli.md |
207210
| `src/treg/client_identity.py` | architecture/import-boundaries.md, architecture/proxy-model.md, interface/api.md |
208-
| `src/treg/config.py` | architecture/auth-secrets.md, architecture/feedback.md, architecture/super-admin.md, guides/expanding-a-category.md, ops/deploy.md |
211+
| `src/treg/config.py` | architecture/archive.md, architecture/auth-secrets.md, architecture/feedback.md, architecture/super-admin.md, guides/expanding-a-category.md, ops/deploy.md |
209212
| `src/treg/convert.py` | interface/cli.md |
210213
| `src/treg/crypto.py` | architecture/auth-secrets.md |
211214
| `src/treg/domain/__init__.py` | architecture/import-boundaries.md |
@@ -270,6 +273,7 @@ Regenerate via `scripts/build-map.py`.
270273
| `src/treg/infra/kv.py` | architecture/feedback.md |
271274
| `src/treg/infra/oauth_exchange.py` | architecture/auth-secrets.md, architecture/instagram-oauth.md, guides/expanding-a-category.md |
272275
| `src/treg/infra/oauth_refresh.py` | architecture/auth-secrets.md |
276+
| `src/treg/infra/object_store.py` | architecture/archive.md |
273277
| `src/treg/infra/stripe.py` | architecture/money.md |
274278
| `src/treg/infra/upstream/__init__.py` | architecture/import-boundaries.md |
275279
| `src/treg/infra/upstream/aggregators/__init__.py` | architecture/import-boundaries.md, ops/capacity.md |
@@ -362,13 +366,15 @@ Regenerate via `scripts/build-map.py`.
362366
| `src/treg/web/vendor/README.md` | interface/dashboard.md |
363367
| `src/treg/web/vendor/vue-3.5.41.global.prod.js` | interface/dashboard.md |
364368
| `src/treg/worker.py` | ops/capacity.md, ops/deploy.md |
369+
| `tests/fake_object_store.py` | architecture/archive.md |
365370
| `tests/fixtures/aggregators/verification/contactout.json` | architecture/contactout.md |
366371
| `tests/js/arena-bench.test.cjs` | interface/enrich-arena.md |
367372
| `tests/js/arena-template.test.cjs` | interface/enrich-arena.md |
368373
| `tests/js/enrich-arena.test.cjs` | interface/enrich-arena.md |
369374
| `tests/test_aigc_pr_b.py` | architecture/catalog.md |
370375
| `tests/test_alembic_expand_safety.py` | architecture/data-model.md |
371376
| `tests/test_app_roles.py` | architecture/composition.md |
377+
| `tests/test_archive_r2.py` | architecture/archive.md |
372378
| `tests/test_arena_insights.py` | interface/enrich-arena.md |
373379
| `tests/test_arena_verification_insights.py` | interface/enrich-arena.md |
374380
| `tests/test_asynctasks.py` | architecture/money.md, architecture/multi-tenancy.md, architecture/proxy-model.md |
@@ -419,12 +425,12 @@ Regenerate via `scripts/build-map.py`.
419425
| Fragment | Sources |
420426
|---|---|
421427
| `architecture/ads-conversions.md` | `adsconv.py`, `signup.py`, `adtrack.js`, `gtag.js` |
422-
| `architecture/archive.md` | `archive.py`, `results.py`, `0031_archive_result_admission.py`, `test_cache_result_admission.py`, `0002_archive_tables.py`, `0003_callrecord_cached.py`, `0004_archivekey_request_shape.py`, `0011_callrecord_archive_link.py`, `service.py`, `backfill_call_archive_links.py`, `api.py`, `bootstrap.py`, `admin.py`, `asynctasks.py` |
428+
| `architecture/archive.md` | `archive.py`, `results.py`, `0031_archive_result_admission.py`, `test_cache_result_admission.py`, `archive_bodies.py`, `config.py`, `object_store.py`, `0028_archive_body_storage.py`, `test_archive_r2.py`, `fake_object_store.py`, `smoke_archive_r2.py`, `0002_archive_tables.py`, `0003_callrecord_cached.py`, `0004_archivekey_request_shape.py`, `0011_callrecord_archive_link.py`, `service.py`, `backfill_call_archive_links.py`, `api.py`, `bootstrap.py`, `admin.py`, `asynctasks.py` |
423429
| `architecture/auth-secrets.md` | `injectors.py`, `ssrf.py`, `crypto.py`, `oauth.py`, `__init__.py`, `authorization.py`, `oauth_flow.py`, `refresh.py`, `oauth_exchange.py`, `oauth_refresh.py`, `oauth_providers.py`, `health.py`, `connect.py`, `connections.py`, `resources.py`, `__init__.py`, `bindings.py`, `bundles.py`, `test_oauth_refresh.py`, `config.py` |
424430
| `architecture/catalog.md` | `quickenrich.yaml`, `quickenrich.extended.yaml`, `quickenrich.companies.search.json`, `quickenrich.people.email.find.json`, `quickenrich.people.enrich.json`, `quickenrich.people.phone.find.json`, `quickenrich.people.search.domain.json`, `quickenrich.people.search.json`, `quickenrich.x.company-services.json`, `quickenrich.x.country-codes.json`, `quickenrich.x.employee-ranges.json`, `quickenrich.x.industries.json`, `quickenrich.x.revenue-ranges.json`, `trykitt.yaml`, `trykitt.people.email.find.json`, `trykitt.people.email.verify.json`, `contracts.yaml`, `millionverifier.yaml`, `millionverifier.people.email.verify.json`, `millionverifier.account.usage.json`, `adapters.yaml`, `test_route_cost_ceiling.py`, `tomba.yaml`, `tomba.people.email.verify.json`, `findymail.search.business-profile.json`, `__init__.py`, `contracts.py`, `paths.py`, `plan.py`, `synthetic.py`, `route.py`, `test_routing.py`, `catalog-drift.yml`, `catalog_drift.py`, `catalog_ingest.py`, `catalog_validate.py`, `aliases.yaml`, `fx.yaml`, `aviato.yaml`, `crustdata.yaml`, `aviato.companies.acquisitions.json`, `aviato.companies.employees.json`, `aviato.companies.enrich.bulk.json`, `aviato.companies.enrich.json`, `aviato.companies.founders.json`, `aviato.companies.funding_rounds.json`, `aviato.companies.investments.json`, `aviato.companies.outbound_investments.json`, `aviato.companies.search.json`, `aviato.linkedin.company.posts.json`, `aviato.linkedin.post.comments.json`, `aviato.linkedin.post.reactions.json`, `aviato.linkedin.post.reposts.json`, `aviato.linkedin.user.posts.json`, `aviato.people.contact.get.json`, `aviato.people.email.find.json`, `aviato.people.enrich.bulk.json`, `aviato.people.enrich.json`, `aviato.people.phone.find.json`, `aviato.people.search.json`, `aviato.people.search.simple.json`, `crustdata.companies.autocomplete.json`, `crustdata.companies.enrich.json`, `crustdata.companies.identify.json`, `crustdata.companies.jobs.search.json`, `crustdata.companies.search.json`, `crustdata.people.autocomplete.json`, `crustdata.people.enrich.json`, `crustdata.people.search.json`, `google-search-console.yaml`, `google-search-console.extended.yaml`, `google-tag-manager.yaml`, `google-tag-manager.extended.yaml`, `instagram.yaml`, `instagram.extended.yaml`, `justoneapi.extended.yaml`, `minimax.yaml`, `apify.yaml`, `brightdata.yaml`, `companyenrich.yaml`, `oceanio.yaml`, `akta.extended.yaml`, `dataforseo.extended.yaml`, `tikhub.extended.yaml`, `minimax.video-gen.result.retrieve.json`, `minimax.video-gen.from_image.json`, `minimax.video-gen.task.status.json`, `openrouter.yaml`, `openrouter.extended.yaml`, `openrouter.x.alibaba-wan-3-0.json`, `replicate.yaml`, `replicate.extended.yaml`, `replicate.image-gen.flux-schnell.json`, `__init__.py`, `store.py`, `settlement.py`, `stats.py`, `catalog_observations.py`, `catalog.py`, `test_aigc_pr_b.py`, `test_catalog_api.py`, `test_catalog_validate.py` |
425431
| `architecture/composition.md` | `bootstrap.py`, `bootstrap_handlers.py`, `bootstrap_http.py`, `call_surface.py`, `connect.py`, `mcp_oauth.py`, `session.py`, `admin.py`, `auth.py`, `billing.py`, `call.py`, `connections.py`, `onboard.py`, `orgs.py`, `resources.py`, `referrals.py`, `web.py`, `dump_surface.py`, `test_app_roles.py` |
426432
| `architecture/contactout.md` | `contactout.yaml`, `adapters.yaml`, `contactout.people.email.verify.json`, `test_routing.py`, `contactout.companies.search.json`, `contactout.companies.enrich.json`, `resolve.py`, `contactout.py`, `contactout.svg`, `test_marketplace_call.py`, `test_key_providers.py`, `test_capacity_collectors.py`, `test_catalog_validate.py`, `test_capacity_overflow.py`, `contactout_overflow_verify.py`, `contactout.json` |
427-
| `architecture/data-model.md` | `alembic.ini`, `env.py`, `0001_baseline_current_schema.py`, `0002_archive_tables.py`, `0003_callrecord_cached.py`, `0004_archivekey_request_shape.py`, `0005_capacity_policy_snapshot.py`, `0006_overflow_route.py`, `0007_overflow_spend.py`, `0008_org_platform_overflow_disabled.py`, `0009_callrecord_hit.py`, `0017_async_task_record.py`, `0018_async_resource_ownership.py`, `0019_async_poll_failures.py`, `0020_callrecord_created_at_indexes.py`, `0021_ledgerentry_org_created_at_index.py`, `0022_org_spent_today_counter.py`, `0023_callrecord_org_user_created_at_index.py`, `0024_membership_calls_today_counter.py`, `0027_enrich_arena.py`, `0028_arena_insights.py`, `0029_arena_verification_snapshot.py`, `0011_callrecord_archive_link.py`, `0015_idempotentcall_membership_cascade.py`, `maintenance.py`, `sitetrack.js`, `models.py`, `0031_archive_result_admission.py`, `timeutil.py`, `db.py`, `referrals.py`, `audit.py`, `analytics.py`, `bootstrap_handlers.py`, `ratestore.py`, `auth.py`, `test_postgres_reset.py`, `test_alembic_expand_safety.py` |
433+
| `architecture/data-model.md` | `alembic.ini`, `env.py`, `0001_baseline_current_schema.py`, `0002_archive_tables.py`, `0003_callrecord_cached.py`, `0004_archivekey_request_shape.py`, `0005_capacity_policy_snapshot.py`, `0006_overflow_route.py`, `0007_overflow_spend.py`, `0008_org_platform_overflow_disabled.py`, `0009_callrecord_hit.py`, `0017_async_task_record.py`, `0018_async_resource_ownership.py`, `0019_async_poll_failures.py`, `0020_callrecord_created_at_indexes.py`, `0021_ledgerentry_org_created_at_index.py`, `0022_org_spent_today_counter.py`, `0023_callrecord_org_user_created_at_index.py`, `0024_membership_calls_today_counter.py`, `0027_enrich_arena.py`, `0028_arena_insights.py`, `0029_arena_verification_snapshot.py`, `0011_callrecord_archive_link.py`, `0015_idempotentcall_membership_cascade.py`, `maintenance.py`, `sitetrack.js`, `models.py`, `0031_archive_result_admission.py`, `0028_archive_body_storage.py`, `timeutil.py`, `db.py`, `referrals.py`, `audit.py`, `analytics.py`, `bootstrap_handlers.py`, `ratestore.py`, `auth.py`, `test_postgres_reset.py`, `test_alembic_expand_safety.py` |
428434
| `architecture/feedback.md` | `feedback_contract.py`, `__init__.py`, `reports.py`, `reviews.py`, `hints.py`, `config.py`, `call.py`, `invite.py`, `kv.py`, `feedback.py`, `feedback.py`, `0025_feedback.py`, `0026_callreview.py`, `0030_feedback_handling.py`, `test_feedback_handling_schema.py`, `feedback.md`, `test_feedback.py`, `test_reviews.py`, `test_hints.py`, `test_kv.py` |
429435
| `architecture/import-boundaries.md` | `pyproject.toml`, `ci.yml`, `__init__.py`, `__init__.py`, `access.py`, `authorize.py`, `idempotency.py`, `overflow.py`, `route.py`, `__init__.py`, `intake.py`, `resolve.py`, `reserve.py`, `settle.py`, `evidence.py`, `service.py`, `types.py`, `client_identity.py`, `__init__.py`, `__init__.py`, `access.py`, `budgets.py`, `publicdemo.py`, `teams.py`, `usage.py`, `__init__.py`, `__init__.py`, `authorization.py`, `oauth_flow.py`, `refresh.py`, `__init__.py`, `__init__.py`, `__init__.py`, `__init__.py`, `__init__.py`, `injectors.py`, `relay.py`, `__init__.py`, `limiter.py`, `test_call_architecture.py`, `test_import_lightness.py` |
430436
| `architecture/instagram-oauth.md` | `catalog_ingest.py`, `access.py`, `resolve.py`, `service.py`, `instagram.yaml`, `instagram.extended.yaml`, `cli.py`, `store.py`, `authorization.py`, `oauth_flow.py`, `oauth_exchange.py`, `mcp.py`, `call.py`, `index.html`, `0010_oauth_authorization_method.py`, `test_instagram_oauth_architecture.py` |

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@ jobs:
106106
tests/test_feedback.py
107107
tests/test_archive.py
108108
tests/test_cache_result_admission.py
109+
tests/test_archive_r2.py
109110
tests/test_call_cancellation.py
110111
tests/test_call_pool_discipline.py
111112
tests/test_postgres_reset.py

‎AGENTS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,8 @@ Everything else in this file is guidance; these are the contract, and they win o
1414
1. A team's own key always wins over treg's, is never metered, and is never routed or overflowed.
1515
2. A hold (the balance `reserve` sets aside for one call) is settled or released exactly once, on
1616
every path: timeout, cancellation and exceptions included.
17-
3. Zero database connections are held while an upstream request is in flight. This is why
18-
`reserve` and `settle` are two transactions; never merge them.
17+
3. A request holds zero database connections while upstream or object-storage I/O is in flight.
18+
Keep `reserve` and `settle` separate; read archive pointers, close the session, then fetch bytes.
1919
4. Plain `/call/` is a faithful relay: the injected credential, the transport headers listed in
2020
`src/treg/infra/upstream/relay.py`, and (on treg's shared key only) the per-org re-scoping of the
2121
caller's `Idempotency-Key` are the only rewrites. Never add upstream-specific modeling or body

‎SECURITY.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,28 @@ environment) and enforcement happens server-side and in the operating system.
3939
a speed bump, not a fix: a new domain costs the other side minutes, so treat the variable as
4040
something to edit during an incident, and suspend the accounts already created separately.
4141

42+
## Archive object storage credentials
43+
44+
R2 credentials are server environment settings (`TREG_ARCHIVE_OBJECT_STORE_ACCESS_KEY_ID` and
45+
`TREG_ARCHIVE_OBJECT_STORE_SECRET_ACCESS_KEY`), never catalog credentials or caller input. Use a token scoped
46+
to the private archive bucket with object read/write access; bucket administration is unnecessary.
47+
The configured endpoint must be an HTTPS Cloudflare R2 account endpoint. Object names are always
48+
SHA-256 hashes computed by treg; clients cannot supply an object path, bucket, host or URL.
49+
The SDK exists only in the server extra and is initialized by bootstrap. SDK exceptions, signed
50+
requests, credentials and response bodies are not included in archive telemetry.
51+
52+
PUT transmits an explicit SHA-256 checksum, then HEAD confirms the digest metadata and byte size
53+
before any R2 pointer is committed. GET verifies the returned raw bytes against the persisted
54+
hash. The bucket is not public and no object URL is exposed: call-history authorization checks
55+
the team's CallRecord before resolving its archive reference. A hash is an identity, not an
56+
access credential. Upstream answers are retained under the existing archive policy; these guards
57+
do not sanitize an upstream provider that echoes a credential in its response.
58+
59+
Object I/O never holds the calling task's database connection. Startup refuses incomplete R2
60+
configuration when archive mode is enabled and any storage switch selects R2. The manual smoke
61+
script accepts only `treg-archive-dev`, refuses CI, and skips absent credentials. It creates no
62+
cloud resources apart from one small test object. Production configuration belongs in the private ops project.
63+
4264
## Known limitations (by design, documented on purpose)
4365

4466
Honesty is part of the model. Two items are deliberately deferred:

0 commit comments

Comments
 (0)