Repository navigation
fix(toolkit): keep source DB intact when DbMove copy fails - #6946
halibobo1205 wants to merge 5 commits into
Conversation
Prevent the prior flow from deleting source databases after logging and ignoring per-file copy errors. Preserve every source until all configured databases copy successfully. Roll back partial destinations, handle recursive database contents, and return a non-zero status on failure. Finalize symlinks only after the copy phase completes, report completion once, and cover rollback and direct retry behavior.
cc85dc5 to
a4045d5
Compare
|
|
||
| private boolean replaceSourceWithLink(Property p) { | ||
| try { | ||
| if (!FileUtils.deleteDir(p.original.toFile())) { |
There was a problem hiding this comment.
Great fix for the copy-failure case. One adjacent gap worth flagging while you're in here: FileUtils.deleteDir (plugins utils/FileUtils.java:73-84) still recurses with java.io.File, and isDirectory() follows symlinks — so a directory symlink gets traversed and its target's contents are deleted.
This composes with this PR: after db mv, the original path is a symlink to the moved DB (DbMove.java:176). If a later db lite merge deletes the archive dbs under that original path (DbLite.backupArchiveDbs calls FileUtils.deleteDir per db), it wipes the already-migrated DB on the new disk — two legitimate operations, chained data loss, no attacker needed.
Suggested fix: replace the recursion with Files.walkFileTree + NOFOLLOW_LINKS (delete the link itself, never follow). The copy phase already uses NOFOLLOW_LINKS (:154/:198), so this would make the delete side consistent. The same pattern also exists in common/.../FileUtil.java:114-126 and the toolkit copy in tron-docker's tools/toolkit — happy to split that into a separate PR if you prefer.
There was a problem hiding this comment.
Thanks for flagging this. The symlink traversal issue is valid, but it predates this PR: db mv already created symlinks, and this change does not modify FileUtils.deleteDir or the db lite merge workflow.
I’ll keep this PR focused on preventing source deletion after copy failures and reporting migration failures correctly. The deletion behavior and lite merge’s handling of relocated databases should be addressed in a separate PR, with regression coverage for the combined workflow.
| @@ -167,7 +244,7 @@ public Property(String name, Path original, Path destination) throws IOException | |||
| throw new IOException(original + " is symbolicLink!"); | |||
| } | |||
| this.destination = destination.toFile().getCanonicalFile().toPath(); | |||
There was a problem hiding this comment.
[MUST] Please reject destinations inside any selected source before copying. With configuration order B, A and A's destination under B's source, B is copied before A's destination exists. Finalizing B deletes A's copy; finalizing A then deletes its original, leaving a dangling link while returning 0. I reproduced this regression: the same configuration preserves A's data on the base. Cross-check the canonical paths with destination.startsWith(source) before creating any destination, return 2 for unsupported layouts, and add self-nesting and cross-database regression tests.
There was a problem hiding this comment.
Confirmed, and it is order-dependent on the base as well. Same layout, both configuration orders:
| order | base | before this fix |
|---|---|---|
| B, A (A's destination under B's source) | A kept | A lost, exit 0 |
| A, B | A lost, exit 0 | A kept |
A destination inside any selected source is now rejected up front with exit 2, regardless of order (9ec2584). Added self-nesting and cross-database tests covering both orders.
I'll update the design writeup in #6940 accordingly. It previously listed this layout as a non-goal, leaving the outcome to the operator and outside the guarantees; it is now rejected before anything is written.
| if (hasError.get()) { | ||
| return; | ||
| } | ||
| try { |
There was a problem hiding this comment.
[SHOULD] A runtime exception during a destination write, such as SecurityException under a SecurityManager, escapes this handler and bypasses rollback. I reproduced exit 1 with all sources intact but completed and partial destinations left behind; retry after removing the denial returns 2. Please handle expected filesystem runtime failures inside the workers and during destination creation/traversal, then roll back after all workers have finished. An outer finally alone does not ensure outstanding parallel copies have stopped.
There was a problem hiding this comment.
Fixed in e6daa90: the copy phase now handles runtime exceptions like I/O errors. Workers catch them, so the parallel forEach only returns after every worker has finished, and destination creation (Files.createDirectories) moved into the same handler; rollback runs afterwards. Same repro now: exit 1, no leftovers, and the unchanged retry returns 0.
I did not add a regression test for this path: raising a RuntimeException from the JDK file APIs here requires a SecurityManager, which is disabled by default since JDK 17 and removed in JDK 24. The shared failure path is covered by the existing I/O-failure tests.
| } | ||
|
|
||
| @Test | ||
| public void testSourceKeptWhenCopyFails() throws RocksDBException, IOException { |
There was a problem hiding this comment.
[SHOULD] This fails the first configured database, so it does not verify rollback after an earlier database has copied successfully. Please add a later-database failure case asserting unchanged source contents, removal of all created destinations, and successful retry. For cleanup failure, assert that all sources remain intact and leftover destinations are reported. For link-creation failure after source deletion, assert that the complete destination is retained and recovery instructions are emitted. Both failures should return non-zero and omit the success message.
| mixinStandardHelpOptions = true, | ||
| version = "db command 1.0", | ||
| description = "An rich command set that provides high-level operations for dbs.", | ||
| header = "All `db` tools operate directly on the database files.\n" |
There was a problem hiding this comment.
[NIT] Please use plain db in the help text, since the backticks are printed literally. Consider including the stop-the-node note in db mv -h as well; this parent-command header is not shown in the subcommand's help.
There was a problem hiding this comment.
Done in 5d35137. The note is now a shared constant (Db.STOP_NODE_HEADER) used as the header of both db and db mv, with plain db instead of backticks. db mv -h now starts with:
All db tools operate directly on the database files.
Before performing a database operation,
you must stop the currently running FullNode service.
Closes #6940
What does this PR do?
Makes
db mva fail-safe two-phase migration: validate first (exit 2, nothing written), then copy every database (recursive, link-aware, fail-fast), and only after all copies succeed replace sources with symlinks. Any copy failure keeps all sources, removes the created destinations, and exits 1 — the same command can simply be re-run. Finalization failures keep the complete copy and print recovery instructions.move db done./ exit 0 only on full success. Also documents the stop-the-node precondition inplugins/README.md.Why are these changes required?
The previous implementation ignored per-file copy errors, then deleted the source and symlinked it to an incomplete destination while still reporting success — any I/O fault (disk full, permissions, bad sector) meant silent, unrecoverable data loss (#6940).
This PR has been tested by:
DbMoveTestgrown to 15 cases (failure/rollback/retry, recursive copy, symlink and permission faults, exit codes).Follow up
None.
Extra details
Design writeup with sequence diagram and non-goals: see #6940.