Check subscription resume positions against retained history, and reset durable MQTT sessions that can no longer catch up (#2946) * Let subscribe check a resume position against the database generation and retained history A subscription request can now name the database generation its position came from (`databaseGeneration`, with a finite `startTime`). Such a request is refused before it registers when the database was replaced (409 DatabaseGenerationChangedError, always on LMDB) or the history after the position is no longer retained (410 ResumeHistoryUnavailableError). The replay is checked again as it runs: a collection replay at every yield and at the end, against the position reached at the previous check, and a record's history walk once it has read back to the cursor, refusing a walk its 10,000-version cap cut short. The returned subscription carries `resumeVerified`, which resolves true only once the replay is complete and checked, and every subscription reports its `databaseGeneration`. Requests without the field behave as before. Refs #2448 Co-Authored-By: Claude Opus 5.5 * Refuse previousCount 0 with a resume position, and check a replay yield against the record in hand `if (request.previousCount)` let a zero through, and a zero count stops a record's history walk after one version while its check still passed. A collection replay's yield check now advances to the key of the record in hand rather than the last key it has passed: a prune deletes only keys below the floor, and every key below the record in hand has been read, so a yield inside one transaction no longer holds the check back to before it. This also drops the per-record key tracking. Co-Authored-By: Claude Opus 5.5 * Say that resumeVerified cannot see a transaction committing below the position RocksDB assigns a transaction's key when the transaction is created, so one can commit after a resume position was recorded with a key below it, and the replay after that position never includes it. The check covers retention and database replacement only; the ordering gap is harper#2928. Co-Authored-By: Claude Opus 5.5 * Verify a record's resume by its own history walk instead of the database floor A record subscription's history walk reads every version back to the resume position, so it is direct evidence that nothing after the position is gone. A walk that reaches the position with every version present now verifies even when the database-wide floor has passed it, so a quiet record, or a device offline longer than retention whose record's recent history is still there, resumes instead of resetting. A missing version refuses, as does the version cap. A record with no entry still falls back to the floor, since a pruned tombstone takes its history with it, and the pre-check for a record checks the generation only. Co-Authored-By: Claude Opus 5.5 * Fall back to the floor when a record's walk ends at a first version Cleanup removes a deleted record's tombstone when retention prunes its delete, so a record put again afterwards starts a new chain with no predecessor. A walk from an older position then ends at that first version having skipped the delete, with nothing missing to refuse it. A walk that ends at a first version before reaching the position is now checked against the floor, which the prune raised past the position. Co-Authored-By: Claude Opus 5.5 * Replay the first-version control from history rather than live Co-Authored-By: Claude Opus 5.5 * Refuse a checked replay across a failed log read or a reload marker The transaction log store ends a failed or corrupt log's iteration quietly and records it on the range (failedLogs, corruptFrameStop), so a replay could skip that log's records and still verify. A checked replay now inspects its range at each check and refuses with the 410 when it recorded a failure. A checked replay that crosses a reload marker for its table refuses too: the rows the reload back-filled have no audit history, so no replay can deliver them. Co-Authored-By: Claude Opus 5.5 * Refuse a checked replay across an audit entry that fails to decode The decoder turns an entry it cannot read into a sentinel with no table, type or record id, and a lazily decoded record id that fails reads as undefined. The replay's table filter skipped both, so a resume across one still verified. A checked collection replay now refuses on a sentinel before the table filter (its table cannot be known) and on an in-table entry whose record id failed to decode, and a record walk treats an undecodable version as missing. DESIGN.md also records that a record resume, and any resume while a replication base copy runs, cannot see the rows the copy back-fills (harper-pro#944). Co-Authored-By: Claude Opus 5.5 * Drop two test comments that restate their helpers Co-Authored-By: Claude Opus 5.5 * Let a subscription certify how far it has delivered (opt-in reportProgress) The broadcaster keeps a dispatched-through watermark per database: the key of the newest transaction every record of which it has dispatched. It advances only when a new key begins or a pass exhausts the log, never at a yield inside a transaction. It is tracked only while a subscription asked for progress, and it stops for good at a failed or corrupt log read or at an entry or record id that fails to decode. A subscription requested with `reportProgress` carries `progress()`: the key at or below which every history event it will deliver has already been sent. It follows the checked replay while resuming, holds at the resume position while a record's history is walked, holds at the registration watermark while a snapshot or buffered live events are pending, and follows the watermark once live. Such a subscription counts its sends, keeps every event buffered during its snapshot (the snapshot filter can drop events the scan never covered, harper#2933), tags scan events `fromScan`, and ends rather than re-snapshotting on a reload marker or when it fails to build a live event. Co-Authored-By: Claude Opus 5.5 * Resume MQTT durable sessions through the checked subscription A persisted QoS 1/2 subscription now records the database generation of a position certified by the subscription's progress, and resumes with it, so Table.subscribe refuses a position whose database was replaced or whose history is gone instead of replaying short. - Before CONNACK, a session whose wildcard entry fell below the floor, or whose entry names another generation, is deleted and reported as not present. - A refusal after CONNACK (a failed resume subscribe, or a refusal delivered as a subscription's final value, observed through its close even when the consumer is blocked on the socket) discards the session: writers are fenced, the connection closes without waiting on storage, and the record is deleted if this connection still owns it. Any other subscription failure saves positions and closes the connection so the client resumes. - A topic's position is the older of its subscription's progress and the newest transaction before its oldest unacknowledged message, and nothing past a resumed position is saved until the replay verifies. Positions are taken on acknowledgements, on a timer, and at disconnect before subscriptions end, so a quiet topic keeps up with the database. - Out-of-order acks are no longer saved as a list that filters by key; a legacy list is ignored and dropped. - Each connection stamps an incarnation on the record before CONNACK and writes only while the record still carries it; an older connection on the same thread is closed. Error values are never forwarded as messages. - saveSubscriptions keeps the persisted positions (harper#2908). Closes #2448 Co-Authored-By: Claude Opus 5.5 * Test certified progress and MQTT durable-session resume, in-process and over the wire Co-Authored-By: Claude Opus 5.5 * Document certified subscription progress and the MQTT durable-session lifecycle Co-Authored-By: Claude Opus 5.5 * Keep acknowledgements advancing uncertified topics, and coalesce and retry session saves A topic whose subscription reports no progress (LMDB, or a resource that is not a table) never moved off its first position, so every reconnect replayed everything since the SUBSCRIBE. It now advances on acknowledgements alone, still only to a transaction the client has taken and moved past. Session saves are coalesced into one write in flight. A failed save leaves the session dirty so the next checkpoint retries it, and rejects to a caller that awaits it, so a SUBSCRIBE whose record could not be written is not granted. Co-Authored-By: Claude Opus 5.5 * Freeze a topic whose certificate is unavailable, and save each SUBSCRIBE that is granted Only a resource that offers no progress at all (LMDB, or one that is not a table) advances on acknowledgements alone, now to the last delivered key once everything is acknowledged, and such a position is never bound to a database generation. A table subscription whose certificate is unavailable no longer advances. The save loop clears its in-flight marker together with its last dirty check, so a save requested at that moment starts a new write instead of joining one that has finished. A SUBSCRIBE whose record could not be saved ends the subscription it started, and every resume failure goes through the terminal path, which closes the connection. Co-Authored-By: Claude Opus 5.5 * Roll back only what a failed SUBSCRIBE started, and keep QoS 0 subscriptions with the session A failed SUBSCRIBE removed whichever subscription held its topic, so an older request failing after a newer one to the same topic had registered ended the newer one and deleted its state. It now removes only the subscription it started, plus the one it replaced, which replacing already ended. It saves the record once a topic is gone. Saved sessions carry QoS 0 subscriptions again, as main's record did. They are saved without a position and resume live. The resume uses retain handling 2 and awaits no acknowledgement. Co-Authored-By: Claude Opus 5.5 * Test a durable MQTT session across a hard kill and restart Co-Authored-By: Claude Opus 5.5 * Drop a test helper comment that narrates its body Co-Authored-By: Claude Opus 5.5 * Document QoS 0 entries and failed-SUBSCRIBE rollback in the durable-session design Co-Authored-By: Claude Opus 5.5 * Keep an uncertified topic before a key a queued message may share A resource that certifies nothing still advances on acknowledgements. Once nothing was unacknowledged, it saved the last delivered key. A custom Resource wrapping a RocksDB table keeps that table's shared transaction keys, so a queued sibling of the last acknowledged message could be skipped. Now only a topic whose events each have their own log key, as on LMDB, saves the delivered key. Any other topic saves the key before it. Co-Authored-By: Claude Opus 5.5 * Drop test helper comments that restate their bodies Co-Authored-By: Claude Opus 5.5 * Take an older durable connection over on a clean start too A clean-start connect deletes the client's durable session record. It left an older durable connection with the same client id running on this thread, and that connection kept receiving messages until its next save found the record gone. A connection that had never saved could write the record back. Every connection with a client id now closes an older durable connection on its thread (v5 0x8E), as a durable one already did. Co-Authored-By: Claude Opus 5.5 * Drop the durable session's unused copy of its loaded record Co-Authored-By: Claude Opus 5.5 * Apply a durable session's subscription changes one at a time MQTT packets are handled concurrently. Two SUBSCRIBEs to one topic could both miss each other in the replacement lookup and leave two live subscriptions, one of them writing its deliveries into the other's topic state. SUBSCRIBE, UNSUBSCRIBE and resume now change a durable session's topics in order. A superseded session also stops writing, so a clean start's deletion cannot be undone by the older connection's later saves. Co-Authored-By: Claude Opus 5.5 * Describe serial subscription changes and clean-start takeover in the durable-session design Co-Authored-By: Claude Opus 5.5 * Take the session over before reading or deleting its record, and send UNSUBACK once removal happens A takeover superseded the older connection only after the new connection had read the record, or a clean start had deleted it. So the older connection's save already in flight could still land afterwards, and write a clean-started session back. The takeover now runs once the client is authenticated, before getSession. It waits for the older connection's saves to settle, and a save that is superseded during its read no longer writes. The clean-start deletion is awaited. UNSUBSCRIBE now waits for each removal, which a durable session queues behind its other subscription changes. UNSUBACK used to treat the removal's Promise as its result, so it always reported success. It now reports 0x11 for a topic that had no subscription and 0x80 for a removal that fails. Co-Authored-By: Claude Opus 5.5 * Continue a durable topic's position when a SUBSCRIBE replaces its subscription Many clients send SUBSCRIBE again every time they connect. Replacing the resumed subscription started a new one at the current watermark, so the rest of the replay was lost, along with every delivery not yet acknowledged. On main the same resubscribe saved the current time. A replacement for a QoS 1 or 2 topic now continues from the topic's position, through the same checked replay a reconnect uses, as MQTT requires ("Application Messages MUST NOT be lost due to replacing the Subscription"). A refusal there resets the session, as it would at reconnect. Co-Authored-By: Claude Opus 5.5 * Take the session over in turn when CONNECTs for one client id arrive together Each CONNECT looked for an older connection before either one had registered its session. So two CONNECTs for one client id handled together on a thread could both stay live until one of them next saved. A CONNECT now claims its client id before it waits on anything, and waits for the CONNECT before it to register, so it always takes over from that one. Co-Authored-By: Claude Opus 5.5 * Settle an MQTT acknowledgement once the position it allows is saved main wrote the session record inside acknowledge(), and mqtt.ts emitted 'acknowledged' only after that write. Checkpoints now coalesce, so the event fired before the position was saved. A client that reconnected quickly could then read the earlier position and get an acknowledged message again. The apitest "subscribe to retained record with patch operations" failed that way. acknowledge() now returns the scheduled checkpoint's save, so the event waits for it as before, and acks arriving in one turn still share one save. Co-Authored-By: Claude Opus 5.5 * Describe only a resume refusal, or a 403 or 404, in the DISCONNECT a failed session closes with closeConnection() put the message of any 4xx error into the DISCONNECT reason string. A resource that throws a 400, 409 or 422 with an internal message would then hand that message to the client. The PUBACK/PUBREC path in this file deliberately withholds such messages. The DISCONNECT now follows the same rule. It describes the two resume refusals, whose messages this code writes, and 403 and 404, and nothing else. Co-Authored-By: Claude Opus 5.5 * Start the progress watermark afresh each time tracking starts again dispatchedThrough only advances while a progress-reporting subscription exists. When the last one ended, the key stayed, while QoS 0 and HTTP subscribers kept the broadcaster dispatching. The next such subscription then registered at that leftover key. MQTT saved it as the position of a new durable SUBSCRIBE, so a reconnect replayed history from before the SUBSCRIBE, or reset the session once retention passed the stale key. Tracking now starts with no watermark. A new subscription starts at the clock and binds at the first certified position, as it does when tracking first begins. Co-Authored-By: Claude Opus 5.5 * Keep each MQTT connection's will its own, and let packets behind a CONNECT wait out its takeover - A will row is keyed by client id and incarnation, so a connection that closes after a takeover publishes and deletes only its own will, never the will of the connection that replaced it. - The CONNECT's pending session is assigned before it waits on an older connection, so a packet sent right behind it waits too instead of closing the socket. - A closed connection stays a takeover's to wait for until its last save lands, found through a per-client-id map rather than a scan of every session. The wait gives up after 10 s with CONNACK "server unavailable", and a client that leaves while waiting has its session ended. - A QoS 0 SUBSCRIBE is saved before SUBACK like any other. An UNSUBSCRIBE whose save failed is reported failed again on a retry, and any failed save starts the checkpoint timer. - A collection resume checks the floor before each wait for a slow consumer, so a prune of history it already sent does not refuse it. Co-Authored-By: Claude Opus 5.5 * Keep a CONNECT that gave up waiting in line, and let an UNSUBSCRIBE retry wait for a save in flight - A CONNECT whose takeover wait times out keeps its place until the CONNECT before it finishes, so a later one can no longer connect past an earlier connection still loading the session. - An UNSUBSCRIBE retried while a save is in flight waits for that save, so it reports a failure instead of 0x11 for a removal that is not saved yet. - A session that has ended starts no checkpoint timer, so a SUBSCRIBE that finishes after the disconnect leaves none behind. - A takeover after the older connection's save failed resumes from the last save that landed; the design notes now say so, and a test pins it. Co-Authored-By: Claude Opus 5.5 * Re-size the unit:main step cap to 6 minutes, which main's slowest run had grown to within 6 seconds of main's last 8 pushes ran the step in 3m05-3m54 against its 4-minute cap, and legs that reach mocha's epilogue at ~4m08 are killed while they exit. The workflow's own rule sizes a cap at roughly 1.5x the slowest run that reached the epilogue, the derivation the LMDB step used. Co-Authored-By: Claude Opus 5.5 * Resume a durable session another cluster node saved from its position, unchecked, instead of resetting it hdb_durable_session replicates, but each node mints its own database generations, so a reconnect to another node failed every check and reset the session, where main resumed it unchecked from its position. A record now names the node that wrote it. On any other node its positions are unbound: they skip the pre-CONNACK check, replay unchecked as before, and bind again at the first certified position there. Co-Authored-By: Claude Opus 5.5 * Keep checking a renamed node's own durable positions, and save an older connection's positions before a takeover - An entry is bound here when its record names this node or its generation is this database's current one. A node whose name changed (a new certificate CN, an edited node.hostname) still holds the generation it bound, so its sessions stay checked against the floor rather than resuming unchecked, and so do rows written before records carried a node name. - A takeover on this thread now has the older connection take its positions and save them before it stops, so an acknowledgement whose checkpoint was still a turn away is not redelivered by the newer connection. Co-Authored-By: Claude Opus 5.5 * Fence a taken-over connection's writes once its final save settles A fresh older connection whose first QoS 1 SUBSCRIBE was still waiting on its resource when a newer connection took over could persist afterwards and, with no record yet, create one under its own incarnation, so the newer connection closed itself at its next save (and a clean start's deleted session came back). yieldTo now marks the session discarded once its final save settles, inside the writes promise the takeover waits for. Co-Authored-By: Claude Opus 5.5 * Seal an ended durable session against later saves, and let a CONNECT whose client left take nothing over - A session that has ended, by a disconnect or by yielding to a takeover, saves nothing after its final save. A socket close used to leave a SUBSCRIBE still waiting on its resource free to save once a reconnect had read and claimed the record, putting the old incarnation back over it, and yieldTo on an already-closed session fenced nothing. - A CONNECT whose client closed while it waited stops after each wait: queued behind another CONNECT, it no longer closes the session that CONNECT made live, and it never loads, claims or stores a will for a session nobody will use. Co-Authored-By: Claude Opus 5.5 * Resume a durable session best-effort when its database generation changed, resetting only on pruned history A position bound to another generation, whether another node's or this database's before a restore or copy, now resumes unchecked from where it is and binds again at the first certified position, instead of resetting the session. Only a 410, history that retention removed, still discards it. A live subscription that a restore ends saves positions and closes, so the client reconnects and resumes. Binding is now judged by the generation alone, so records no longer carry the node that wrote them. Co-Authored-By: Claude Opus 5.5 * Wait for the delivery to be consumed before the takeover test acknowledges it The consumer loop counts a delivery as consumed one turn after sending it, and until then the position stays before that delivery, which is correct for a message still in flight. On a fast runner the test's PUBACK and takeover landed inside that turn, so the saved position correctly stayed behind the delivery and the assertion failed. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 (cherry picked from commit ce32b18e4bd6e0fa26483d1e8eb70570e55d12b5)