A long-lived onSnapshot listener dies with Error: A backoff operation is already in progress. The error arrives at the user's error callback, so the listener is gone and the application has to resubscribe. On 2026-10-01 four listeners in one process died within 100ms of each other, on a Cloud Run instance that had just been woken by a request.
Environment: @google-cloud/firestore 7.11.6 via firebase-admin 13.10, Node 20, Cloud Functions 2nd gen on Cloud Run with cpu: 0.125, no always-on CPU, maxInstances: 1, and one request every five minutes. The stack below is from that version's build/src, but the code it points at is unchanged on main, so the line numbers in my reading are from handwritten/firestore/dev/src as it stands today.
The stack:
Error: A backoff operation is already in progress.
at ExponentialBackoff.backoffAndWait (/workspace/node_modules/@google-cloud/firestore/build/src/backoff.js:182:35)
at QueryWatch.initStream (/workspace/node_modules/@google-cloud/firestore/build/src/watch.js:290:14)
at QueryWatch.maybeReopenStream (/workspace/node_modules/@google-cloud/firestore/build/src/watch.js:244:18)
at Timeout.<anonymous> (/workspace/node_modules/@google-cloud/firestore/build/src/watch.js:267:18)
at listOnTimeout (node:internal/timers:581:17)
at processTimers (node:internal/timers:519:7)
What I think happens. resetIdleTimeout arms idleTimeoutHandle when a stream opens (watch.ts:428), and only two places clear it: resetIdleTimeout itself on the next data event (watch.ts:424), and shutdown (watch.ts:873). The stream's error and end handlers call maybeReopenStream without clearing it (watch.ts:517 and watch.ts:526), so the idle timer armed by a dead stream stays live. When it fires it ends a stream that has already gone and calls initStream a second time (watch.ts:439), while the backoff from the first call is still pending. backoffAndWait rejects on awaitingBackoffCompletion (backoff.ts:256), initStream's own .catch treats that as fatal and calls closeStream (watch.ts:532), and the listener is handed to the caller as a permanent failure. awaitingBackoffCompletion is cleared only by its own delay timer (backoff.ts:290), so any second initStream during a backoff reaches the same place. The stale idle timer is just the easiest way to get there.
A CPU-throttled instance makes it likely rather than rare. Between requests the instance gets no CPU, so the 120s idle timer and the backoff's own delay both come due during the freeze and then run in the same wake. That also explains four listeners failing at once: each Watch has its own backoff and its own idle timer, and they all came due together.
Is clearing idleTimeoutHandle in maybeReopenStream, or at the top of initStream, the right place to fix it?
A long-lived
onSnapshotlistener dies withError: A backoff operation is already in progress.The error arrives at the user's error callback, so the listener is gone and the application has to resubscribe. On 2026-10-01 four listeners in one process died within 100ms of each other, on a Cloud Run instance that had just been woken by a request.Environment:
@google-cloud/firestore7.11.6 viafirebase-admin13.10, Node 20, Cloud Functions 2nd gen on Cloud Run withcpu: 0.125, no always-on CPU,maxInstances: 1, and one request every five minutes. The stack below is from that version'sbuild/src, but the code it points at is unchanged onmain, so the line numbers in my reading are fromhandwritten/firestore/dev/srcas it stands today.The stack:
What I think happens.
resetIdleTimeoutarmsidleTimeoutHandlewhen a stream opens (watch.ts:428), and only two places clear it:resetIdleTimeoutitself on the next data event (watch.ts:424), andshutdown(watch.ts:873). The stream'serrorandendhandlers callmaybeReopenStreamwithout clearing it (watch.ts:517 and watch.ts:526), so the idle timer armed by a dead stream stays live. When it fires it ends a stream that has already gone and callsinitStreama second time (watch.ts:439), while the backoff from the first call is still pending.backoffAndWaitrejects onawaitingBackoffCompletion(backoff.ts:256),initStream's own.catchtreats that as fatal and callscloseStream(watch.ts:532), and the listener is handed to the caller as a permanent failure.awaitingBackoffCompletionis cleared only by its own delay timer (backoff.ts:290), so any secondinitStreamduring a backoff reaches the same place. The stale idle timer is just the easiest way to get there.A CPU-throttled instance makes it likely rather than rare. Between requests the instance gets no CPU, so the 120s idle timer and the backoff's own delay both come due during the freeze and then run in the same wake. That also explains four listeners failing at once: each
Watchhas its own backoff and its own idle timer, and they all came due together.Is clearing
idleTimeoutHandleinmaybeReopenStream, or at the top ofinitStream, the right place to fix it?