Sitelet https://github.com/tronprotocol/java-tron/pull/6946
Skip to content

fix(toolkit): keep source DB intact when DbMove copy fails - #6946

Open
halibobo1205 wants to merge 5 commits into
tronprotocol:release_v4.8.3from
halibobo1205:feature/dbmove-data-safety
Open

halibobo1205 wants to merge 5 commits into
tronprotocol:release_v4.8.3from
halibobo1205:feature/dbmove-data-safety

Conversation

@halibobo1205

Copy link
Copy Markdown
Collaborator

Closes #6940

What does this PR do?

Makes db mv a fail-safe two-phase migration: validate first (exit 2, nothing written), then copy every database (recursive, link-aware, fail-fast), and only after all copies succeed replace sources with symlinks. Any copy failure keeps all sources, removes the created destinations, and exits 1 — the same command can simply be re-run. Finalization failures keep the complete copy and print recovery instructions. move db done. / exit 0 only on full success. Also documents the stop-the-node precondition in plugins/README.md.

Why are these changes required?

The previous implementation ignored per-file copy errors, then deleted the source and symlinked it to an incomplete destination while still reporting success — any I/O fault (disk full, permissions, bad sector) meant silent, unrecoverable data loss (#6940).

This PR has been tested by:

  • Unit Tests: DbMoveTest grown to 15 cases (failure/rollback/retry, recursive copy, symlink and permission faults, exit codes).
  • Manual Testing: simulated copy failures on real node data — sources kept, destinations rolled back, exit 1; retry completed the migration.

Follow up

None.

Extra details

Design writeup with sequence diagram and non-goals: see #6940.

Prevent the prior flow from deleting source databases after logging and ignoring per-file copy errors.

Preserve every source until all configured databases copy successfully. Roll back partial destinations, handle recursive database contents, and return a non-zero status on failure.

Finalize symlinks only after the copy phase completes, report completion once, and cover rollback and direct retry behavior.

private boolean replaceSourceWithLink(Property p) {
try {
if (!FileUtils.deleteDir(p.original.toFile())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great fix for the copy-failure case. One adjacent gap worth flagging while you're in here: FileUtils.deleteDir (plugins utils/FileUtils.java:73-84) still recurses with java.io.File, and isDirectory() follows symlinks — so a directory symlink gets traversed and its target's contents are deleted.

This composes with this PR: after db mv, the original path is a symlink to the moved DB (DbMove.java:176). If a later db lite merge deletes the archive dbs under that original path (DbLite.backupArchiveDbs calls FileUtils.deleteDir per db), it wipes the already-migrated DB on the new disk — two legitimate operations, chained data loss, no attacker needed.

Suggested fix: replace the recursion with Files.walkFileTree + NOFOLLOW_LINKS (delete the link itself, never follow). The copy phase already uses NOFOLLOW_LINKS (:154/:198), so this would make the delete side consistent. The same pattern also exists in common/.../FileUtil.java:114-126 and the toolkit copy in tron-docker's tools/toolkit — happy to split that into a separate PR if you prefer.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for flagging this. The symlink traversal issue is valid, but it predates this PR: db mv already created symlinks, and this change does not modify FileUtils.deleteDir or the db lite merge workflow.

I’ll keep this PR focused on preventing source deletion after copy failures and reporting migration failures correctly. The deletion behavior and lite merge’s handling of relocated databases should be addressed in a separate PR, with regression coverage for the combined workflow.

@@ -167,7 +244,7 @@ public Property(String name, Path original, Path destination) throws IOException
throw new IOException(original + " is symbolicLink!");
}
this.destination = destination.toFile().getCanonicalFile().toPath();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MUST] Please reject destinations inside any selected source before copying. With configuration order B, A and A's destination under B's source, B is copied before A's destination exists. Finalizing B deletes A's copy; finalizing A then deletes its original, leaving a dangling link while returning 0. I reproduced this regression: the same configuration preserves A's data on the base. Cross-check the canonical paths with destination.startsWith(source) before creating any destination, return 2 for unsupported layouts, and add self-nesting and cross-database regression tests.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, and it is order-dependent on the base as well. Same layout, both configuration orders:

order base before this fix
B, A (A's destination under B's source) A kept A lost, exit 0
A, B A lost, exit 0 A kept

A destination inside any selected source is now rejected up front with exit 2, regardless of order (9ec2584). Added self-nesting and cross-database tests covering both orders.

I'll update the design writeup in #6940 accordingly. It previously listed this layout as a non-goal, leaving the outcome to the operator and outside the guarantees; it is now rejected before anything is written.

if (hasError.get()) {
return;
}
try {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SHOULD] A runtime exception during a destination write, such as SecurityException under a SecurityManager, escapes this handler and bypasses rollback. I reproduced exit 1 with all sources intact but completed and partial destinations left behind; retry after removing the denial returns 2. Please handle expected filesystem runtime failures inside the workers and during destination creation/traversal, then roll back after all workers have finished. An outer finally alone does not ensure outstanding parallel copies have stopped.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e6daa90: the copy phase now handles runtime exceptions like I/O errors. Workers catch them, so the parallel forEach only returns after every worker has finished, and destination creation (Files.createDirectories) moved into the same handler; rollback runs afterwards. Same repro now: exit 1, no leftovers, and the unchanged retry returns 0.

I did not add a regression test for this path: raising a RuntimeException from the JDK file APIs here requires a SecurityManager, which is disabled by default since JDK 17 and removed in JDK 24. The shared failure path is covered by the existing I/O-failure tests.

}

@Test
public void testSourceKeptWhenCopyFails() throws RocksDBException, IOException {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SHOULD] This fails the first configured database, so it does not verify rollback after an earlier database has copied successfully. Please add a later-database failure case asserting unchanged source contents, removal of all created destinations, and successful retry. For cleanup failure, assert that all sources remain intact and leftover destinations are reported. For link-creation failure after source deletion, assert that the complete destination is retained and recovery instructions are emitted. Both failures should return non-zero and omit the success message.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in f3deb19.

mixinStandardHelpOptions = true,
version = "db command 1.0",
description = "An rich command set that provides high-level operations for dbs.",
header = "All `db` tools operate directly on the database files.\n"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NIT] Please use plain db in the help text, since the backticks are printed literally. Consider including the stop-the-node note in db mv -h as well; this parent-command header is not shown in the subcommand's help.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5d35137. The note is now a shared constant (Db.STOP_NODE_HEADER) used as the header of both db and db mv, with plain db instead of backticks. db mv -h now starts with:

All db tools operate directly on the database files.
Before performing a database operation,
you must stop the currently running FullNode service.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:DB Database

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

[Bug]Toolkit db mv may delete the source database after a copy failure, causing data loss

4 participants