Commit 2e09c92
mdbx: expose Env.Copy* with MDBX_CP_OVERWRITE and add Env.Defrag (#223)
* mdbx: enable Env.Copy* with new MDBX_CP_* flags and add Env.Defrag
- Uncomment Env.Copy/CopyFlag/CopyFD/CopyFDFlag and rewire them through
the unified mdbx_env_copy / mdbx_env_copy2fd entry points.
- Expose MDBX_CP_* flags (CopyDefaults, CopyForceDynamicSize,
CopyDontFlush, CopyThrottleMVCC and the new CopyOverwrite for
clobbering an existing target file).
- Bind mdbx_env_defrag via a thin cgo helper (mdbxgo_env_defrag), expose
DefragOptions / DefragResult and the MDBX_defrag_* stopping reasons.
- Replace the long-commented Copy tests with working ones plus dedicated
tests for the new CopyOverwrite flag and Env.Defrag.
* mdbx: make CopyFDFlag portable to Windows
On Windows mdbx_filehandle_t is HANDLE (void*), so the previous
C.mdbx_filehandle_t(fd) where fd is uintptr would not compile under
GOOS=windows. Route the call through a thin C helper that takes a
uintptr_t and performs the platform-specific cast in C, so the Go side
stays identical on every target. The CopyFD test now runs on Windows
as well.
* mdbx: drop redundant uint64_t cast in Env.Defrag
r.spent_time_dot16 is already typed C.uint64_t, so the explicit
conversion is flagged by unconvert.
* mdbx: skip TestEnv_Defrag on Windows
mdbx_env_defrag opens its own write transaction via txn_basal_start, and
on Windows it trips ERROR_LOCK_VIOLATION against the env's own LockFileEx
region when run on a handle that has just committed writes. The defrag
API binding itself is fine — only the test scenario hits the libmdbx
Windows locking interaction.
* mdbx: drop Windows skip from TestEnv_Defrag
Keep the failure visible in CI until libmdbx clarifies the expected
mdbx_env_defrag calling sequence on Windows / fixes the LockFileEx
region conflict that ERROR_LOCK_VIOLATIONs on a freshly written env.
* ci: re-trigger after GitHub Actions skipped the previous push
* ci: re-trigger workflows
* save
* mdbx: only compacting copies, defrag needs Exclusive, lint fixes
Copy: a non-compacting copy silently produces an EMPTY database. libmdbx's
copy_asis (v0.14.2) builds the destination meta-pages from a pristine model —
trees.main.root = P_INVALID, first_unallocated = NUM_METAS — and never writes
the source txn's roots or geometry into them, unlike copy_with_compacting.
The data pages are copied byte-for-byte, so the target opens without complaint
and reads as an empty db. That is what the "does not reproduce the committed
data" note in the tests was describing. Reject copies without CopyCompact
(ErrCopyNotCompacting) instead of exposing a backup API that loses data, and
let Copy/CopyFD compact on their own; TestEnv_Copy and TestEnv_CopyFD now
cover the flagless entry points that had no test.
Defrag: open the environment Exclusive in the test, like libmdbx's mdbx_defrag
tool does (MDBX_ENV_DEFAULTS|MDBX_EXCLUSIVE with an MDBX_ACCEDE fallback).
Cutting off trailing pages needs the whole-file lock; taking it per
transaction is what failed the win-2025 job with ERROR_LOCK_VIOLATION. The
requirement is now documented on Env.Defrag.
Lint (7 golangci-lint findings): intrange, fmtappendf, thelper, and the
"shrinked" misspelling — DefragResult.PagesShrinked is now PagesShrunk, with
the wrapper's C field renamed to match.
* mdbx: cast fd straight from uintptr_t in mdbxgo_env_copy2fd
Routing the file descriptor through a signed intptr_t on the way to
mdbx_filehandle_t is implementation-defined for a Windows HANDLE with the
high bit set (C11 6.3.1.3p3), and buys nothing on POSIX where the target
is a plain int. Cast from the unsigned uintptr_t directly, matching the
existing mdbxgo_tid_to_u64 idiom.
* mdbx: lock the OS thread in tests that begin a write txn
TestTxn_OpenDBI_zero and TestTxnEnvWarmup call Env.BeginTxn with write
flags without runtime.LockOSThread, which the API explicitly requires:
"BeginTxn does not call runtime.LockOSThread. Unless the Readonly flag is
passed goroutines must call runtime.LockOSThread before calling BeginTxn".
When the goroutine migrates between BeginTxn and the deferred Abort, the
abort runs on a foreign thread, mdbx_txn_abort fails with
MDBX_THREAD_MISMATCH, and Txn.abort discards that error. The write txn
stays alive, so mdbx_env_close returns MDBX_BUSY and the datafile is
never released. On POSIX the unlink still succeeds and nothing is
noticed; on Windows the t.TempDir cleanup fails with "The process cannot
access the file because it is being used by another process".
setup's cleanup swallowed the Close error, which is what kept this
invisible outside Windows. Report it instead, so a leaked write txn fails
the test that leaked it, on every platform.
* mdbx: drop the compacting-only copy guard, fixed upstream in v0.14.3
ErrCopyNotCompacting existed because libmdbx v0.14.2's copy_asis built
the destination meta-pages from a pristine model and never wrote the
source tree roots into them, so an as-is copy opened as an empty
database. v0.14.3 (vendored in f8b27ce) copies geometry, trees.gc,
trees.main, the canary and the txnid into the meta before writing it.
Restore the plain lmdb-go semantics: Copy and CopyFD do an as-is copy,
CopyFlag and CopyFDFlag pass the flags through untouched. Tests cover
both modes against both targets.
* mdbx: cut unreachable defrag constants and duplicate copy tests
DefragNoObstacles is the zero of an OR-mask, so StoppingReasons == 0
already says it. DefragDiscontinued and DefragAborted are only ever
raised by the progress callback, which Env.Defrag always passes as NULL,
so neither can appear in a result today.
TestEnv_CopyFlag_AsIs and TestEnv_CopyFDFlag_AsIs re-ran the path
TestEnv_Copy and TestEnv_CopyFD already cover: Copy is defined as
CopyFlag(path, CopyDefaults) and CopyDefaults is zero. TestEnv_Defrag's
nil-result check tested a value Defrag builds unconditionally, and its
PagesWhole assertion is implied by the PagesMoved one below it.
* mdbx: test DefragOptions.TimeLimit
Two things can go wrong with TimeLimit and neither shows up in
TestEnv_Defrag: the duration can fail to reach libmdbx at all, since it
travels as a count of 1/65536-second units rather than as a Duration, and
the bound can reach it but never stop anything.
Cover both. TimeLimit below TimeAtLeast is rejected by libmdbx, which it
can only do after the conversion, so that proves both durations arrive.
Then a millisecond limit against a database whose unbounded defrag takes
tens of milliseconds must come back with DefragTimeLimit in
StoppingReasons.
The limit stops defrag from scheduling further work rather than aborting
the batch in flight, so SpentTime can overshoot it and is not asserted.
Measured 23 pages moved over 2 cycles under the limit against 5080 over 4
without it, and the stopping reason is stable over 8 runs.
* mdbx: saturate NewDuration16dot16 instead of wrapping it
Duration16dot16 is unsigned and time.Duration is not, so a negative
duration wrapped straight through the conversion. Every libmdbx entry
point that takes one of these narrows it to 32 bits, and every one reads
0 as "no bound", which turned two ordinary mistakes into silent opposites
of what the caller asked for:
TimeLimit: deadline.Sub(time.Now()) // deadline already passed
wrapped to 18446744073709486077, whose low 32 bits are a bound of roughly
18h12m -- an unbounded defrag where the caller wanted none. And
TimeLimit: 24 * time.Hour
is 5662603224 ticks, past UINT32_MAX, so defrag_init's (uint32_t) cast
left 5h48m. A duration landing on a multiple of 2^32 ticks left the
detent equal to the start timestamp, stopping defrag after one cycle.
Sub-15.26us durations rounded to 0, i.e. no bound at all.
Clamp in the conversion rather than at the call sites: Env.Defrag is the
new caller, but Env.SetSyncPeriod and Txn.EnvWarmup pass the same value
through C.uint and had the same two bugs. Only a zero duration may now
produce 0; anything negative or below the unit becomes the smallest real
bound, and anything past the 32-bit ceiling saturates there.
* mdbx: handle the no-reader LaggardReader, and cut duplicated test fixtures
mdbx_env_defrag returns MDBX_LAGGARD_READER with no reader anywhere near
it: when defrag stalls on the same page four times over it does
rc = txn->dbs[FREE_DBI].items ? MDBX_LAGGARD_READER : MDBX_RESULT_TRUE;
so an empty GC gives "stopped early" and a non-empty one gives an errno,
for the same stall. That code escapes the MDBX_RESULT_TRUE normalization
at the end of mdbx_env_defrag, so it reaches Go as a real error. Both
defrag tests open Exclusive, which rules out an actual reader, yet both
called t.Fatalf on any error: whether they pass depends on GC layout,
page size and filesystem. Export the errno as LaggardReader, say what it
really means on Env.Defrag, and let the tests tolerate it.
DefragResult.PagesShrunk repeated upstream's promise that it can go
negative. It cannot: libmdbx computes it as before_defrag -
last_allocated, both pgno_t, so the unsigned 32-bit result zero-extends
and a "negative" shrink arrives as a value near 1<<32. Say so, since
`if res.PagesShrunk < 0` would never fire.
TestEnv_CopyFlag_Overwrite only asserted that the second copy failed, so
any unrelated breakage of CopyFlag passed as "refuses to clobber".
Assert os.ErrExist, which is what libmdbx's O_EXCL open produces and
which maps correctly on Windows too.
The three fixtures the new tests had inlined twice each -- write one
record, reopen and compare, write-then-delete for defrag -- become
seedCopyItem, verifyCopyItem and seedDefrag.
* mdbx: recover the sign of pages_shrinked instead of documenting its loss
The previous commit papered over this by telling callers that
PagesShrunk can never go negative. It can, and the information was
there: libmdbx computes it as an unsigned 32-bit subtraction of two
pgno_t and stores the wrapped result in a wider intptr_t, so a backwards
shrink arrives zero-extended near 1<<32. MAX_PAGENO is 0x7FFFffff, so
the true difference always fits in int32 and reinterpreting the low half
recovers it unambiguously. Do that in the shim and restore mdbx.h's own
description of the field.
Also drop the gratuitous rename. The shim mirrors all fifteen members of
MDBX_defrag_result_t under upstream's names; pages_shrinked was the only
one respelled, which costs the 1:1 grep trail against libmdbx for
nothing. The exported Go field stays PagesShrunk, since that is a Go API.
No test: defrag growing the file is what produces a negative value and I
could not provoke it, with or without reclaimable pages, at any size
tried. The positive path stays covered by TestEnv_Defrag.
* mdbx: drop the comment explaining that a field matches upstream
* mdbx: put the shim field back to pages_shrunk
287ac51 renamed it to upstream's pages_shrinked to keep the C side
greppable against libmdbx. That breaks the build: misspell is enabled in
.golangci.yml, cgo puts the member name into env.go, and the linter
rejects "shrinked" there. The original spelling was not gratuitous.
Keeps the sign recovery from 287ac51, which was the point of that commit.
The C statement still names the upstream field on its right-hand side, so
the grep trail survives where it actually runs.
* mdbx: cut the commentary back, and document the backlash clamp
Fold NewDuration16dot16's zero case into the switch it sits above, and
drop the repeated "0 = no bound" from every DefragOptions field now that
the struct comment says it once.
The rest is comment volume. Most of these explained upstream behaviour at
a length that dwarfed the code: 10 lines on a 10-line duration
conversion, 18 on Defrag, 6 on one cast. Kept the facts a reader cannot
get from the source in front of them -- why the fd cast avoids intptr_t,
why the low half of pages_shrinked recovers its sign, that LaggardReader
fires with no reader, that seedDefrag needs two transactions -- and
deleted the retellings.
One addition, not a cut: AcceptableBacklash now says libmdbx clamps it to
about 1018 pages at a 4 KiB page size. Measured on a 41 GiB chaindata,
values of 10, 20, 30, 40 and 41 GiB all behaved identically to autopilot,
and nothing in mdbx.h says why.
net -64 lines.
---------
Co-authored-by: Alexey Sharov <AskAlexSharov@gmail.com>1 parent 52e7938 commit 2e09c92
8 files changed
Lines changed: 474 additions & 141 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
4 | 7 | | |
5 | 8 | | |
6 | 9 | | |
| |||
9 | 12 | | |
10 | 13 | | |
11 | 14 | | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
12 | 19 | | |
13 | 20 | | |
14 | 21 | | |
15 | 22 | | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
16 | 27 | | |
17 | | - | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
18 | 38 | | |
19 | 39 | | |
20 | 40 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
36 | 37 | | |
37 | 38 | | |
38 | 39 | | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | | - | |
| 60 | + | |
61 | 61 | | |
62 | 62 | | |
63 | | - | |
| 63 | + | |
64 | 64 | | |
65 | | - | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
66 | 74 | | |
67 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
68 | 87 | | |
69 | 88 | | |
70 | 89 | | |
| |||
236 | 255 | | |
237 | 256 | | |
238 | 257 | | |
239 | | - | |
| 258 | + | |
240 | 259 | | |
241 | | - | |
242 | | - | |
243 | | - | |
244 | | - | |
245 | | - | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
246 | 264 | | |
247 | | - | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
248 | 268 | | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
254 | 274 | | |
255 | | - | |
| 275 | + | |
| 276 | + | |
256 | 277 | | |
257 | 278 | | |
258 | | - | |
259 | | - | |
260 | | - | |
261 | | - | |
262 | | - | |
263 | | - | |
264 | | - | |
265 | | - | |
266 | | - | |
267 | | - | |
268 | | - | |
269 | | - | |
270 | | - | |
271 | | - | |
272 | | - | |
273 | | - | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
274 | 292 | | |
275 | 293 | | |
276 | 294 | | |
| |||
686 | 704 | | |
687 | 705 | | |
688 | 706 | | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
0 commit comments