Sitelet https://github.com/awesome-lang-auth/awesome-lambda-auth/pull/13
Skip to content

feat: outgoing webhooks delivered from SQS with a dead-letter queue (D9b) - #13

Merged
nik2208 merged 34 commits into
mainfrom
d9b/webhook-queue
Sep 30, 2026
Merged

nik2208 merged 34 commits into
mainfrom
d9b/webhook-queue

Conversation

@nb-camelot

@nb-camelot nb-camelot commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Dipende dalla PR #12: unire prima quella, poi questa con Squash and merge.

Stacked on PR #12 (4025f04); this PR's own commits are 4025f04..3774a0b. The review fixes (33 findings) and the line-up on #12 are listed in the PR comments; where this description and the comments disagree, the comments are current.

What this does

D9a left WebhookSender.Deliverer at the core's in-process HTTP deliverer, which on Lambda races the response freeze (outgoing-webhook-delivery-races-the-response). This block fills that one seam, opt-in:

  • tools.outboundWebhooks.queueUrl ([new], AWESOME_AUTH_TOOLS_OUTBOUND_WEBHOOKS_QUEUE_URL). Empty (the default) keeps D9a unchanged. When set, the sender's transport is awsintegration.SQSWebhookDeliverer: the core's fully built, signed, numbered attempt goes onto SQS whole (URL, full header set with the signature, body bytes base64'd) in a product-owned, schema-numbered envelope. The core's envelope, headers, signature and numbering are unchanged.
  • The response waits for the enqueue. webhookDefaults.FindByEvent runs synchronously on the request goroutine before the emitter spawns its goroutines, so it counts the settlements the request owes. App.Handle waits for them (at most 2 s) before handing the response to the runtime. Without this, "at-least-once" would still race the freeze.
  • cmd/webhook-worker is a new binary and artifact. It runs a 128 MB arm64 Lambda on the queue (batch 10, records concurrent, ReportBatchItemFailures). Per record: claim in a ledger → POST with the core's own HTTPWebhookDeliverer, bytes verbatim (plus X-Correlation-Id, as in process) → then either ack, back off with ChangeMessageVisibility = RetryDelay × 2^attempt (rounded up to whole seconds, capped just under 12 h), or dead-letter with a DeadLetterReason (exhausted / receive-ceiling / malformed) and LastStatus.
  • The delivery ledger (internal/store/dynamodb/webhook_deliveries.go) is designed first in data-model.md §1.9 and closes §8.10. The item is PK=IDEM#webhook#<deliveryId>, SK=IDEM, with a 24 h TTL. It keeps the reserved key shape, but not the bare attribute_not_exists(PK): every retry of one queued delivery carries the same id, so a boolean key would forbid every retry. Instead it has four states (claimed / failed / delivered / abandoned) and a lease equal to the invocation deadline. A duplicate that finds a live claim is not acked. A lost claim is classified from ReturnValuesOnConditionCheckFailure with no read.
  • Stack: EnableWebhookQueue (default false; takes effect only with EnableTools). A Rule refuses it without EnableTools.

Retry-count decision

  • Count: the attempt the deliverer normally gets is attempt 0 with Remaining == Retries(), so the count travels in the body.
  • Delay: RetryDelay() is not on the attempt. It is taken from the configuration the core itself resolved: webhookDefaults.FindByEvent records each returned config's RetryDelay() by ID just before the sender runs, and the deliverer carries it as the RetryDelayMs attribute. Two alternatives were rejected. The stack-wide default would silently ignore per-row values, which would be a deviation. A store read has no by-id getter in the core, needs the tenant, and could disagree with Remaining. An attempt the snapshot never saw falls back to tools.outboundWebhooks.defaults.retryDelayMs.
  • Counter: the worker counts attempts with the ledger's claims, not ApproximateReceiveCount. A duplicate receive is not an attempt.
  • Queue ceiling: the queue's maxReceiveCount (WebhookQueueMaxReceiveCount, default 12 = the schema's max 10 retries + 1 + 1 slack) is a ceiling/backstop. The worker enforces the per-subscription count and dead-letters explicitly. A row wanting more attempts than the ceiling allows is dead-lettered at the ceiling with receive-ceiling. Only messages the worker crashed on are redriven silently.

Alarm decision (rule 10)

Option 1: only the alarm that catches this function's failure mode. WebhookDeadLetterAlarm fires on DLQ ApproximateNumberOfMessagesVisible ≥ 1 (Maximum, 300 s, notBreaching). It is gated on WebhookQueueAlarmed: !And [AlarmsEnabled, WebhookQueueEnabled]. That gives nine alarms without the queue and ten with it, still inside the free ten.

template_test.go's counting rule is unchanged. Its condition check is widened to accept a feature condition defined as !And [!Condition AlarmsEnabled, …], because an alarm on a conditional queue cannot be gated on AlarmsEnabled alone. The worker's errors/throttles/duration/concurrency are deliberately left unalarmed: a crash loop ends in the DLQ, and throttling only delays delivery (14-day retention).

With this block alone the set is exactly 10. D9d (#14) and D9c (#15), lined up on top of this PR, change the counting rule to "alarms enabled by default" (9) and assert the all-on total (12) against cost-model §3.3; this PR keeps the rule as it was.

New AWS resources (only with EnableWebhookQueue + EnableTools)

resource USD / month at rest
WebhookQueue (SSE-SQS, visibility 180 s = 6 × worker timeout, 14-day retention, redrive → DLQ) 0.00
WebhookDLQ (SSE-SQS, 14-day retention) 0.00
WebhookWorkerFunction (128 MB arm64, 30 s, own IAM) 0.00
WebhookWorkerLogGroup (/aws/lambda/<stack>-webhook-worker, LogRetentionDays) ~0.00
WebhookDeadLetterAlarm 0.00 free / 0.10 if the allowance is spent
event-source empty receives (~650 000/month idle, estimate) 0.00 in the free million / ~0.26 beyond

The per-attempt cost is ≈ USD 4.60 per million attempts, more than half of it the ledger's 2 WCU. The full table is in docs/cost-model.md §3.3.

IAM:

  • Auth function: EnqueueOutgoingWebhooks (sqs:SendMessage on the queue).
  • Worker: ConsumeWebhookQueue (receive/delete/change-visibility/get-attributes on the queue), DeadLetterWebhooks (sqs:SendMessage on the DLQ), WebhookDeliveryLedger (dynamodb:UpdateItem on the table only for dynamodb:LeadingKeys IDEM#webhook#*).

Deviations

  • New queued-webhooks-are-delivered-at-least-once: a guarantee the reference does not give (duplicates are possible, there is a DLQ, and the schedule has transport limits).
  • New queued-webhook-retries-reuse-the-delivery-id: retries carry the id the core minted. The reference mints one per attempt.
  • Amended, not retired outgoing-webhook-delivery-races-the-response: it now applies only while queueUrl is unset, because the queue is opt-in.

RS rules: none taken. RS-19 stays unused. A queueUrl with the tools block or webhook store off is reported as an unwired knob, not refused.

Shared files touched (each edit contiguous, marked D9b)

  • cmd/auth/tools.go: toolsWiring.queue, a newToolsWiring param, one block after tw :=, the webhookDefaults.queue hook, the log line, and a header paragraph.
  • cmd/auth/app.go: Options.WebhookDeliverer, the call site, Handle flush, and unwiredKnobs.
  • cmd/auth/tools_test.go: a D9b cases block in TestUnwiredKnobsIsExactlyTheDocumentedList.
  • cmd/auth/deviations.go, deviations_test.go.
  • infra/sam/template.yaml: parameter group, three params, two conditions, a Rule, the auth env var and IAM, a resources block, the alarm, and two outputs.
  • infra/sam/template_test.go: the condition check and a D9b section.
  • .github/workflows/go.yml: LAMBDAS plus a size check for both zips.
  • scripts/deploy.sh: checks/builds every CodeUri artifact the template names.
  • scripts/toolchain.sh: two contract env vars.
  • internal/config/{config,env,validate,defaults_test}.go.
  • Docs: docs/spec/{config-schema,data-model}.md, docs/{config-reference,cost-model,deviations}.md, infra/sam/README.md, test/contract/README.md.
  • go.mod/go.sum: service/sqs v1.52.0, no core SDK bump.

Tests

  • Message shape: pinned byte for byte, and a round trip of the signed bytes.
  • Worker vs recorded receiver: core-built bytes are byte-identical at the receiver and the signature verifies. 500 × 4 gives waits of 1/2/4 s with the same delivery id, then DLQ exhausted + LastStatus. Timeout is tested with a log that never quotes the URL. Also covered: visibility arithmetic, malformed → DLQ, receive-ceiling, and a busy duplicate not acked.
  • Ledger: state machine plus a 30×8 claim race on DynamoDB Local. Worker idempotency race on DynamoDB Local: two copies of one message → exactly one POST.
  • cmd/auth: the enqueue has finished when Handle returns. The defaults snapshot is tested, and a failed last attempt does not hold the response.
  • Contract suite: an opt-in case, tools/outgoing-webhook-reaches-the-receiver-signed, runs behind AWESOME_AUTH_CONTRACT_WEBHOOK_RECEIVER_URL (+ _LISTEN). The suite runs the receiver behind a tunnel and chose the secret, so it verifies the signature itself.

Gate green on ./... with no skips. LAMBDAS="auth webhook-worker" builds both: auth 9.7 MB, worker 5.0 MB, both under the 12 MiB budget. No AWS command was run.

Concerns not settled

  • RedrivePolicy.maxReceiveCount: !Ref WebhookQueueMaxReceiveCount (a Number parameter inside a JSON-typed property) is validated by reading only. It should be checked at the first deploy.
  • Operability gap: replaying a dead-lettered message within 24 h of its last attempt bounces straight back to the DLQ, because the ledger holds abandoned. §17.4 documents deleting the IDEM#webhook#<id> item first.
  • The idle empty-receive figure is an estimate. Check NumberOfEmptyReceives after a day.
  • A failed enqueue spends one attempt of the row's budget. This is the core's numbering contract; it is documented and not engineered around.

For C1 (reference 1.10.0–1.10.8, not implemented here)

src/tools/webhook-sender.ts itself did not change in 1.10.x. Related items:

  • 1.10.5: (req.body ?? {}) across the tools router, so a body-less POST /track / /notify answers 200/400 instead of 500.
  • 1.10.0: AuthToolsOptions.sseDistributor, plus the warning when both sse: true and a distributor are set (D9c's transport).
  • 1.10.0: automatic event publication from the routers, with payload caps that shape bridged webhook envelopes (correlationId 1–128 chars of [A-Za-z0-9_.:-], email ≤ 320, AUTH_OAUTH_CONFLICT fields limited).
  • 1.10.0: the new event identity.user.email.changed ({ oldEmail, newEmail }), which webhooks can subscribe to.
  • 1.10.5 docs: node:vm is not a secure sandbox for inbound jsScript (D9d's).

Upstream: nothing needed. The core's WebhookDeliverer seam was sufficient.

🤖 Generated with Claude Code

nb-camelot and others added 15 commits September 30, 2026 15:01
Both blocks were merged by the owner on 2026-09-28, D8 first and D9a
rebased onto it. With them no domain is gated any more. The task board
marks P6 and X1 green, and the in-flight paragraph states the current
upstream and product position, the integration PR and wave 2 that
follow, and the owner's 2026-09-30 decision to freeze the parity target
on @awesome-lang-auth/node 1.10.8 with a catch-up block before v1.0.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the admin console merged, `tools.auth: admin` no longer has to be
refused at cold start. toolsAccess returns
auth.ToolsProtected(core.AdminGuard(base).Protect) - the same value the
adapter guards <admin>/api/* with - so the callers are exactly the
console's, and its refusals are the console's 401/403. checkToolsSupport
and its call in New are gone; internal/config still requires
admin.enabled for the posture, and toolsAccess refuses a Config that
bypassed the loader with no console mounted.

No double-submit is layered on it: the reference's admin guard has
none, and RS-18 keeps SameSite off none under a session policy.

TestToolsAccessPostures drives the bootstrap secret (202), an anonymous
caller (401 {"error":"Unauthorized"}), an ordinary user by bearer and by
cookie (403), and toolsAccess's own refusal with the console off.
docs/config-reference.md 17.6 gains the admin posture paragraph and the
apiKey paragraph names the console as the key store's other consumer;
the SAM ToolsAuth description says why admin is not offered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reference's CORS layer is router.use inside the auth router
(auth.router.ts:512-527) and createToolsRouter emits no Access-Control
header. Mounted beside the prefix - the shape tools.router.ts:114
documents, and this product's default - the tools router never meets
that layer, so corsExemptMounts now exempts it as it exempts the admin
console. Mounted under the prefix - as the Angular demo does,
router.use('/tools', ...) on the router served at /api/auth
(ng-awesome-node-auth src/server/auth.routes.ts:98-99, src/server.ts:65)
- every request passes the auth router's CORS first, so there the
mount stays wrapped.

TestToolsMountFollowsTheReferenceCORSGeometry pins both shapes; one
sentence each in docs/config-reference.md 17.6 and 16.2. Conformance,
not a deviation, so no register entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and webhooks

toolsKnobGaps reported stores.enable.apiKeys and stores.enable.webhooks
as "nothing in this build reads it" whenever the tools block was off
(or, for apiKeys, under a posture other than apiKey) - even with the
admin console mounted, whose credential routes mint and revoke keys and
manage webhook subscriptions through those same stores. A mounted
console (httpConfig(cfg).AdminMounted()) now silences both rows;
telemetry, which reaches a route through ToolsOptions alone, stays
inert with the block off.

Two rows in TestUnwiredKnobsIsExactlyTheDocumentedList pin it; the
docs/config-reference.md 17 and 17.1 lists and the driverStores comment
say the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…number

docs/spec/config-schema.md section 2 still had the placeholder row
"RS-14-RS-16 reserved: the tools block (D9a) numbered its rules first",
although both blocks are merged. It now carries D9a's three real rows
(RS-14 distributor, RS-15 inbound webhooks, RS-16 unset tools.auth).
internal/config/errors.go says the same: the RS-16 comment no longer
claims the rule is absent from the table, and the RS-14..16 note is in
the present tense. decisions.md has a single D-21, as required.

docs/config-reference.md had two sections numbered 17 ("Two worked
postures" and "tools.*, knob by knob"); the worked postures move to the
end as section 18. Nothing referenced the old number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ense

Both blocks are on main, but several comments written on one side still
spoke of the other as future work: ui.go (the console's settings routes
are what can write settings.UI), internal/store/dynamodb/webhooks.go
(the console's admin route defaults events and isActive), admin.go
(driverStores lists telemetry because the tools block hands it over),
the coreOptionSets wantEmpty comment (both remaining slots are empty on
purpose, nothing is waiting), and the phase-gate tests in
internal/config/phases_test.go (the empty set is the state, not a flip
the parent still owes; the skipped tests say why they skip).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… costs at 2.8

The index row for templates-dir-only-seeds-absent-ids was lost when D9a
was rebased onto the admin console: the register entry and its pin in
deviations_test.go stayed, and TestDeviationsIndexIsComplete kept
passing because the runtime-settings row cites the ID. The row is back
as it stood before the merge, and the test now requires every product
deviation to open an index row of its own.

main's docs/cost-model.md numbers uploads 2.6, the console 2.7 and the
tools block 2.8, but every tools reference still said 2.6 (two register
Spec fields and their index rows, the bridge comment in tools.go,
config-reference 17.2, the SAM tools banner). They now say 2.8; the
uploads references stay at 2.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…claim

TestUnwiredKnobsIsExactlyTheDocumentedList covered only the tools
block's rows; it now also drives the admin block's two knobs over core
constants, ui.uploadDir in its filesystem spelling, and both blocks at
once, so the one list the cold start logs is pinned as a union.

TestTemplatesAreBackedByEveryDriver said its second half requires the
early, named refusal of an unbacked store, but after the merge it only
re-checked that telemetry is accepted. With every stores.enable flag
handed over there is no unbacked key left, so it now pins the other side
of the claim: both drivers back the same set, and an unknown driver is
refused early and by name.

idp.go: restore the paragraph break before the adminRL paragraph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- TestDriverStoresListTheConsoleStores no longer speaks of a flag that stays
  out: all six are asserted present.
- TestTemplatesAreBackedByEveryDriver says the per-flag refusal of
  checkStoreSupport is unreachable through either driver, and asserts both
  drivers are known and non-empty so the symmetry loop cannot pass vacuously.
- TestUnwiredDomainViaEnvIsAlsoRefused's skip message says what its comment
  says: it speaks again the day a domain is gated.

Review findings (PR #12, conventions 5 and 9a/9c).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The console is mounted, so "will expose once the admin surface is mounted"
is stale, and the users note no longer names the first-user access policy,
which RS-17 refuses on every driver. The notes and their byte-for-byte quotes
in docs/deviations.md change together, as TestDeviationsIndexIsComplete
requires.

Review finding (PR #12, conventions 8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…RS gives the tools mount

validateMounts refused a tools or admin mount at "/" or equal to
http.apiPrefix, but not one above it (tools.basePath /api under
http.apiPrefix /api/auth). corsExemptMounts exempts such a tools mount as
"beside the prefix", and the exemption matches everything below the mount,
so every auth route lost the CORS layer: no Allow-Origin, no Vary, no
preflight answer. It is now refused with the same diagnostic shape as the
prefix itself, for either knob, and pinned in TestRefuseToStartRules.

Under the prefix the tools mount stays inside the layer, as the reference's
geometry requires; §17.6 and the corsExemptMounts comment now say what that
means under tools.auth: admin -- an allow-listed origin gets credentialed CORS
on routes that take a console credential. The assembleHandler layer list and
the SAM ToolsBasePath description name the tools mount and the new refusal.

Review findings (PR #12, security 3 and 4, wire 4 / conventions 3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p header and SAM wrap

- "a tools store with the block off" now switches on stores.enable.webhooks
  too, so the row toolsKnobGaps reports for it with no console mounted is
  pinned; deleting that branch now fails the case.
- The toolsKnobGaps header no longer speaks of a flag's "one consumer": the
  flags are consumed by the tools block and, for two of them, by the console.
- The ToolsAuth description in the SAM template is re-wrapped from the
  over-long line to the paragraph's width; the rendered value is unchanged.

Review findings (PR #12, conventions 1, 4 and 7).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n on it behind an open console, register the core's redirect

Three things about tools.auth: admin the review found unsaid or unguarded.

CSRF. Under a session policy the console's guard reads the same accessToken
cookie the session posture does, and nothing checked the double-submit in
front of it, so an admitted administrator's cookie could carry POST
<tools>/track from a same-site page. The reference's documented guard for the
tools router is auth.middleware() (tools.router.ts:114), which performs it
(auth.middleware.ts:33-41). The vendored console calls no tools route, so the
guard is now wrapped as `session` is: a cookie caller needs the matching
X-CSRF-Token, a bearer caller does not, and the cookie looked for is the one
the guard reads (<admin.cookiePrefix>accessToken when that knob is set).
The legacy guard and `open` read no cookie and are not wrapped.

Open console. Behind admin.accessPolicy: open the guard reads no credential,
so the pair is `none` by another name. It now gets `none`'s load-time warning
(collectWarnings) and cold-start Warn, and a plain Info line under every
other policy; §17.1, §17.6 and the tools.go comments no longer say "nobody
else". The `none` remedy names admin too.

Redirect. Under a session policy with admin.loginPath set, an unauthenticated
text/html request to a tools route gets the core's 302 to
<loginPath>?redirect=<admin mount><tools path>, a path nothing serves; the
reference answers 401 on every route but the panel since 1.10.0. Registered
as the core-caused tools-admin-login-redirect-points-into-the-admin-mount and
pinned by TestToolsAdminPostureRedirectsIntoTheAdminMount, which fails the day
the core answers 401 there.

Tests: a flagged is-admin-flag user reaches track by bearer (202); by cookie
it is 403 CSRF_INVALID without the header and 202 with it, also under a
cookie prefix; the refused non-admin session now sends the header and asserts
the console's {"error":"Forbidden"} body, so the session guard or a CSRF
refusal cannot pass for it; the legacy-secret comment says the guard compares
strings. TestWireDeviationIDsArePinned and the deviations index gain the new id, and the
index test's header says each product id opens its own row.

Review findings (PR #12: wire 1, 2, 3, 4; security 1, 2; conventions 2, 6, 9b).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… first-user policy

RS-17 refuses admin.accessPolicy: first-user on every driver, so the
usage text no longer names it as a reader of the user directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nb-camelot

Copy link
Copy Markdown
Collaborator Author

Review fixes (33 findings), head a4da656

Eight commits were added on top of 1a3d24a. There was no rebase and no force-push. The gate is green on ./... and the DynamoDB Local race ran. LAMBDAS="auth webhook-worker" builds auth at 9,683,686 B and the worker at 4,985,392 B, both under the 12,582,912 B budget. No AWS command was run.

Security

  • 1 (high) fixed. WebhookWorkerRole is now an explicit AWS::IAM::Role with no managed policy and no RoleName. It has four statements, each with a Sid and each on one resource: queue consume, DLQ send, ledger UpdateItem under IDEM#webhook#*, and logs:CreateLogStream/PutLogEvents on the worker's own log group. The function sets Role: and has no Policies:, so SAM attaches no AWSLambdaSQSQueueExecutionRole. template_test.go asserts the explicit Role, that the function has no Policies, that the role has no ManagedPolicyArns, and the exact action list of each statement. The "no wider" texts were corrected in main.go, the SAM README and the test.
  • 2 (high) fixed. On ErrWebhookTooLargeToQueue the flush count settles at attempt 0 and is never settled again for that delivery. Settling again would release another delivery's wait early. TestTooLargeEnvelopeDoesNotHoldTheResponse drives the core's real sender and asserts the flush returns in under 900 ms, against the old 2 s. It also asserts that the core's later retry does not settle the next count. §17.4 and cost-model §3.3 now say what a track caller can still cost.
  • 3 fixed. Cost model §3.3 now says concurrency bounds the rate, not the total. It gives the worker-duration ceiling (5 × 30 s × 128 MB continuously, about USD 21.60/month) and names who fills the queue without a credential (login.failed, and track under none/session). It also notes the 64 KB billing unit. The WebhookWorkerMaxConcurrency description was reworded to match.
  • 4 fixed, worker path plus documentation. A failing message whose next attempt would come within 12 h of the queue's retention is dead-lettered with reason expiring. The retention comes from the new env var AWESOME_AUTH_WEBHOOK_QUEUE_RETENTION_SECONDS, which template_test.go pins equal to MessageRetentionPeriod. A message that is never received within 14 days (a backlog, or a worker that cannot start) is still dropped by SQS without a trace. The template comment, §17.4 and §3.3 now say that plainly. No alarm was added.
  • 5 fixed. The DLQ comments, the alarm and output descriptions and §17.4 say "14 days from the hand-off". A message SQS redrove itself keeps its original enqueue time.
  • 6 fixed. A receive that finds the ledger abandoned now dead-letters as abandoned-earlier, not exhausted. The ledger does not keep the first reason, and changing the store's signature to keep it was not worth it.
  • 7 declined (documented). The TTL stays at 24 h. The store comment and data-model §1.9 now describe the backlog gap. A 15-day window would make a replayed dead-letter bounce straight back for a fortnight.
  • 8 fixed. The deny-list was replaced by exact action lists (see 1).
  • 9 fixed. §17.4 and the template now say what a queued or dead-lettered message holds and for how long.
  • 10 declined. Following redirects matches the in-process path and the reference's fetch.

Wire fidelity

  • 1 (high) fixed, text only. The register (both entries), both deviations.md rows, §17.4 and data-model §1.9 now say "one id per queued message". They name the duplicate where SQS stored the message but the client deadline fired, which gives two deliveries with two ids. An idempotent enqueue is not available on a standard queue.
  • 2 fixed. The contract Doc now names library-events-are-bridged-into-the-tools-fan-out as what is pinned and cites webhook-sender.ts for the delivery wire only. The case asserts at least one delivery.
  • 3 / conventions 10 fixed as skips; the literal CapTools fix is declined. The case skips with its reason when an anonymous POST <tools>/track answers 404, and when POST <admin>/api/webhooks answers 404. CapTools was not added: it is absent under the default apiKey posture, where the bridged login is still delivered.
  • 4 / security 6 fixed. Every path that hands a record back to the queue now checks the receive ceiling. On the last receive the worker dead-letters with receive-ceiling, busy-at-ceiling or ledger-unavailable. A DLQ message with no reason now means only that the worker did not finish its last receive (a crash, a timeout, or a failed hand-off). All texts were updated.
  • 5 fixed. The register now says "alarm when EnableAlarms is on".
  • 6 fixed. The size check adds the attributes' names, types and values (MessageAttributesSize), with a test.
  • 7 fixed. The correlation function is named (queuedCorrelationID). The test runs it on a bridged login's captured context, and asserts that the uninjected SQS deliverer is wired to it.
  • 8 fixed. The contract case now asserts a v4 id, the toISOString timestamp equal to the body's, the envelope's member set and event equal to its header. The regexes were checked against the core's output.
  • 9 declined as a code change; text qualified. "Never earlier" is now qualified for an SQS duplicate copy in worker.go, the register and §17.4. A retryAfter in the ledger was not added.
  • 10 fixed. The warning text is a constant that both the code and the test use.

Conventions

  • 1 fixed. The README build line is now LAMBDAS="auth webhook-worker", and the build-lambda.sh comment was rewritten.
  • 2 fixed. webhookDeliveryLedger and var _ webhookDeliveryLedger = (*Store)(nil) now sit in webhook_deliveries.go.
  • 3 fixed. The contract README has the REQUIRE list entry and the case row.
  • 4 fixed. The localStore comment says exactly what sets DYNAMODB_ENDPOINT and that the helper differs from the harness default.
  • 5 fixed. LAMBDAS is job-level with a D9b marker, and the size loop reads from it.
  • 6 fixed. The D9b Rule was moved after AdminRootUserNeedsItsPasswordHash.
  • 7 fixed. "(ten with EnableWebhookQueue)" was added in the template, the SAM README, the cost model and the README.
  • 8 fixed. §9 now lists the ledger.
  • 9 fixed. The alarm gate now requires !And [!Condition AlarmsEnabled, as the first term. The counting rule is unchanged.
  • 10 is covered with wire 3 above.
  • 11 fixed. The flush warning is on the request's own log line.
  • 12 fixed. D9b markers were added and the header retitled.
  • 13 fixed. teardown.sh names the webhook queues and their dead letters.

Deviations: none added. The text of queued-webhooks-are-delivered-at-least-once and queued-webhook-retries-reuse-the-delivery-id was amended in deviations.go and docs/deviations.md.

Merge notes for D9c/D9d:

  • README.md build line
  • go.yml job-level LAMBDAS
  • tools.go seam header
  • the "(ten with …)" parentheticals outside the D9b regions
  • gatedOnAlarmsEnabled now needs AlarmsEnabled as the first !And term

🤖 Generated with Claude Code

nb-camelot and others added 13 commits September 30, 2026 18:51
…t (D9b)

Closes §8.10: the SQS consumers own idempotency. The reserved
IDEM# shape is kept but the bare attribute_not_exists(PK) is not,
because every retry of one queued delivery carries the same id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Four states and a lease on IDEM#webhook#<deliveryId>: a claim is
granted when nobody has tried, the last try failed, or the last
lease lapsed; delivered and abandoned are absorbing. A lost claim
is classified from the pre-image, with no read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The attempt crosses whole — URL, the signed header set, the body
bytes base64-encoded — in a product-owned envelope with a schema
number; the resolved retry delay and the correlation id ride as
message attributes so the worker never reads the webhook store.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hedule (D9b)

A Lambda on the webhook queue with ReportBatchItemFailures: claim
the delivery in the ledger, POST with the core's own HTTP deliverer
and the bytes the core signed, then acknowledge, back off by
RetryDelay x 2^attempt through ChangeMessageVisibility, or hand the
message to the DLQ with a reason. Attempts are counted by the
ledger, not by ApproximateReceiveCount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er (D9b)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Url is set (D9b)

WebhookSender.Deliverer becomes the SQS deliverer; webhookDefaults
records each config's resolved RetryDelay for it and counts the
attempts a request owes, and App.Handle waits (bounded) for those
enqueues before the response reaches the runtime. Unset, D9a's
in-process deliverer stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed delivery id (D9b)

outgoing-webhook-delivery-races-the-response is narrowed to the
default, unqueued configuration rather than retired: the queue is
opt-in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… alarm (D9b)

Gated on EnableWebhookQueue (with EnableTools); off adds nothing.
The only new alarm is the DLQ's depth, gated on its own switch as
well as EnableAlarms, so the set is nine without the queue and ten
with it. deploy.sh checks every CodeUri artifact the template names;
CI builds and size-checks both zips.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ver URL (D9b)

The suite runs the receiver and chose the secret, so it can verify
X-Webhook-Signature; the stack reaches it through a tunnel named by
AWESOME_AUTH_CONTRACT_WEBHOOK_RECEIVER_URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the webhook queue (D9b)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… time (D9b)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… managed SQS policy (D9b)

A function with an SQS event source and no Role gets a SAM-generated role
with AWSLambdaSQSQueueExecutionRole attached, which grants receive and
delete on every queue in the account. WebhookWorkerRole now holds the
worker's four statements (queue, DLQ, ledger partition, own log group),
each with a Sid, and no managed policy. template_test.go asserts the
explicit Role, no Policies on the function, no ManagedPolicyArns, and the
exact action list of every statement instead of a deny-list (review:
security 1, security 8). The alarm-gate shape check now requires
AlarmsEnabled as the first term of the !And (conventions 9).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ue, and count attributes in the size (D9b)

ErrWebhookTooLargeToQueue is permanent, so waiting on the core's back-off
held the response for the full two-second flush bound; the count now
settles at attempt 0 and never again for that delivery, so the core's
later retries cannot release another delivery's wait. The size check adds
the attributes' names, types and values, as SQS counts them. The flush
warning is a constant the test matches on, and the production correlation
function is named so the test runs it on a queued delivery's context.
(review: security 2, wire 6, wire 7, wire 10)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nb-camelot and others added 6 commits September 30, 2026 18:51
… a truthful reason for an abandoned duplicate (D9b)

Every path that hands a record back to the queue now checks the receive
ceiling, not only a refused POST: on the last receive a duplicate bounce
is dead-lettered as busy-at-ceiling and a ledger outage as
ledger-unavailable, so a DLQ message without a reason means the worker
did not finish its last receive (review: wire 4). A failing message whose
next attempt would come within twelve hours of the queue's retention is
dead-lettered as expiring, because SQS deletes an expired message without
redriving it; the worker learns the retention from
AWESOME_AUTH_WEBHOOK_QUEUE_RETENTION_SECONDS, pinned equal to the queue's
MessageRetentionPeriod (security 4). A receive that finds the ledger
abandoned hands its copy over as abandoned-earlier instead of claiming
exhausted (security 6). The template's comments, the alarm description,
the DLQ output and the concurrency parameter now say what the DLQ keeps
and for how long, what a message carries, what the alarm cannot see, and
that concurrency bounds the rate and not the total (security 3, 5, 9).
The "never earlier" claim is qualified for an SQS duplicate copy (wire 9).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te cause, and price who fills the queue (D9b)

The register, config reference §17.4 and data-model §1.9 said a receiver
sees one delivery id per event and subscription; an enqueue whose client
deadline fires after SQS stored it is re-enqueued by the core under a
fresh id, so the id is one per queued message and both entries now say
so (review: wire 1). The at-least-once entry names that duplicate, the
early retry of an SQS duplicate copy (wire 9), the receive ceiling that
bounces and ledger outages spend (wire 4), the expiring hand-off and the
silent loss past retention (security 4), and the alarm only when
EnableAlarms is on (wire 5). §17.4 lists every DeadLetterReason, what a
queued message holds and for how long (security 5, 9), what a track
caller can still cost (security 2) and that the flush warning is on the
request's own log line (conventions 11). Cost model §3.3: concurrency
bounds the rate and not the total, the monthly worker-duration ceiling,
who can fill the queue without a credential, and the 64 KB billing unit
(security 3). The ledger's TTL comment and §1.9 say where a backlog can
outlive the window (security 7); §9 lists the ledger among the items
written (conventions 8); the store file pins its own shape (conventions
2); the stale "nine alarms" counts say ten with the queue (conventions
7); and the D9b lines in tools.go are marked (conventions 12).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ch one it builds, teardown names the queues (D9b)

deploy.sh now needs every CodeUri the template names, so the README's
build line passes LAMBDAS="auth webhook-worker" and build-lambda.sh's
comment no longer says the default serves deploy.sh (review: conventions
1). go.yml's LAMBDAS is job-level with a D9b marker, and the size check
loops over it, so a function cannot be built and not checked
(conventions 5). The D9b Rule no longer splits the admin-root comment
from its Rule (conventions 6). teardown.sh names the webhook queues and
the dead-lettered deliveries a stack delete destroys (conventions 13).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…out tools or a webhook store, and checks the wire (D9b)

The case's trigger is a login, which reaches a webhook only because of
the product deviation library-events-are-bridged-into-the-tools-fan-out;
its Doc now says so and cites webhook-sender.ts for the delivery's wire
alone, and "one POST per login" is gone: the assertion is at least one,
as at-least-once delivery requires (review: wire 2). It skips with the
reason on an anonymous POST <tools>/track answering 404 and on a 404 from
POST <admin>/api/webhooks (wire 3, conventions 10; CapTools is not added,
because it is absent under the default apiKey posture where the bridged
login is delivered all the same). It asserts a v4 delivery id, the
toISOString timestamp equal to the body's, the envelope's member set and
the event equal to its header (wire 8). The suite README lists
webhook-receiver among the REQUIRE capabilities and has the case's row
(conventions 3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s correlation id; log a retry only when it is one (D9b)

The correlation test now also builds the webhook queue without an
injection and asserts its SQS deliverer reads the login's
X-Correlation-Id from the queued delivery's context (review: wire 7).
The worker logs "retrying" inside handBack, on the hand-back branch
alone, so a record dead-lettered at the ceiling or as expiring no longer
logs both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nb-camelot

Copy link
Copy Markdown
Collaborator Author

Rebased onto the current head of #12 (4025f04, integ/d8-d9a-composition). It was built on 54994c9, an older head of #12.

  • New head 3774a0b. The same 19 commits, authors and messages unchanged: git rebase --onto 4025f04 54994c9.
  • No conflicts. The branch's own change is line-for-line the same as before (git diff 54994c9 a4da656 and git diff 4025f04 3774a0b add and remove the same lines). The only difference between the old and the new tree is feat(tools): compose the admin console and the tools surface (D8 × D9a) #12's own later commits.
  • Gate green: gofmt, vet, and go test -v -race ./... with DynamoDB Local reachable. No DynamoDB test skipped. The only skips are the retired phase tests and the contract suite, which has no stack to run against.
  • Zips: auth 9,686,062 B, webhook-worker 4,985,520 B, both under the 12 MiB budget.

This is the first of three branches rebased into a line: #13 goes on #12, #14 on #13, and #15 on #14. The shared-file reconciliations between blocks are made on #14 and #15, not here. After #12 is squash-merged, this branch needs git rebase --onto <squash> 4025f04.

🤖 Generated with Claude Code

nb-camelot added a commit that referenced this pull request Sep 30, 2026
…list, one statement of the counts

D9d now sits on D9b (#13) rather than beside it, so the shared pieces each
block wrote on its own are made one:

- infra/sam/template_test.go: one Conditions parser (load's tpl.conditions);
  D9b's conditionDefinition is gone and gatedOnAlarmsEnabled reads the parsed
  map, requiring !And [!Condition AlarmsEnabled, !Condition <feature>] with
  AlarmsEnabled first. offByDefaultAlarmGates lists WebhookQueueAlarmed (D9b)
  beside ScriptRunnerAlarmsEnabled (D9d), so the default count stays nine and
  the gate test checks both switches default to off.
- docs/cost-model.md §3.3: one opening paragraph states the rule and the
  numbers once — nine by default, one per optional function, **All-on total:
  11 alarm metrics** — and every "ten with ..." / "the tenth" / "the
  eleventh" in the cost model, the SAM README, the template's comments, the
  README and the config reference now points there.
- The seam header in cmd/auth/tools.go names both filled seams.
- scripts/build-lambda.sh's example builds the three functions that exist
  (it named a cmd/sse that does not).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nb-camelot added a commit that referenced this pull request Sep 30, 2026
…st model §3.1 and §3.3, SAM README, contract cases

docs/sse.md states what a client receives and is promised across a
reconnect. The config reference argues the SSE function, the event
log's two knobs, AuthType NONE against AWS_IAM behind OAC with the
cost of each, the reservation of 20, the CloudFront behaviour and the
URL clients open, and CORS by the tools mount's rule. The cost model's
§3.1 is now what is deployed: 128 MB, the counted per-poll read, the
publishing writes, refused requests; §3.3 prices the alarm set (nine by
default, ten with SSE, eleven with SSE and the webhook queue).

The contract suite gains sse/stream-emits-named-frames-with-rawData and
sse/resume-replays-after-last-event-id, read over plain net/http with a
bounded deadline, and run against the Function URL as well when
AWESOME_AUTH_CONTRACT_SSE_URL is set.

Every shared-file edit is placed away from D9b's (PR #13) hunks, so the
two merge mechanically; the data-model design moves into §1.5 for that
reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nb-camelot added a commit that referenced this pull request Sep 30, 2026
…, the counts stated once, the stale D9c lines refreshed

D9c now sits on D9d (#14), and through it on D9b (#13), so:

- infra/sam/conditions_test.go, the verbatim copy of D9b's helpers, is
  deleted. The template test has one Conditions parser, and SseAlarmsEnabled
  is in offByDefaultAlarmGates with EnableSse. D9c's own default count and
  TestFeatureGatedAlarmsAreOffByDefault were dropped during the replay for
  D9d's mechanism, which checks every listed gate defaults to off.
- docs/cost-model.md §3.3: the one opening paragraph names all three
  optional alarms. **All-on total: 12 alarm metrics**, asserted. The SSE
  row, §1, §3.1 and §5 point there instead of saying "the tenth" or "ten
  with SSE". The SAM README and the template's comments do the same, as do
  D9b's leftover "ten with the queue" / "$1.00 with the webhook queue".
- SseFunction's environment carries the auth function's inbound-webhook trio
  line for line. D9d changed AWESOME_AUTH_TOOLS_INBOUND_WEBHOOKS from 'false'
  to the EnableInboundWebhooks switch and added the runner and timeout lines,
  and D9c's TestTheSseFunctionIsASubsetOfTheAuthFunction holds the SSE
  environment to the auth function's. The SSE function never mounts the
  route (streamToolsOptions), and naming the runner grants it nothing, but
  RS-15 would refuse its cold start with the route on and no runner.
- TestCIBuildsEveryFunctionTheTemplateDeploys (D9d) compares the template's
  CodeUri list as a set: the SSE function deploys the auth zip.
- The seam header in cmd/auth/tools.go lists the three seams as filled.
- The three lines left stale on purpose are refreshed: the config
  reference §17.8 D9c row (and its heading), the RS-14 note in
  internal/config/errors.go, and the tools-surface log line's `stream`
  attribute, which now says where the stream is served. So is the
  tools.sse.enabled knob-gap remedy that still said "or for D9c".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik2208
nik2208 merged commit a081d67 into main Sep 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants