Documentation
¶
Overview ¶
Package lmdb provides bindings to the lmdb C API. The package bindings are fairly low level and are designed to provide a minimal interface that prevents misuse to a reasonable extent. When in doubt refer to the C documentation as a reference.
http://www.lmdb.tech/doc/ http://www.lmdb.tech/doc/starting.html http://www.lmdb.tech/doc/modules.html
Environment ¶
An LMDB environment holds named databases (key-value stores). An environment is represented as one file on the filesystem (though often a corresponding lock file exists).
LMDB recommends setting an environment's size as large as possible at the time of creation. On filesystems that support sparse files this should not adversely affect disk usage. Resizing an environment is possible but must be handled with care when concurrent access is involved.
Note that the package lmdb forces all Env objects to be opened with the NoTLS (MDB_NOTLS) flag. Without this flag LMDB would not be practically usable in Go (in the author's opinion). However, even for environments opened with this flag there are caveats regarding how transactions are used (see Caveats below).
Databases ¶
A database in an LMDB environment is an ordered key-value store that holds arbitrary binary data. Typically the keys are unique but duplicate keys may be allowed (DupSort), in which case the values for each duplicate key are ordered.
A single LMDB environment can have multiple named databases. But there is also a 'root' (unnamed) database that can be used to store data. Use caution storing data in the root database when named databases are in use. The root database serves as an index for named databases.
A database is referenced by an opaque handle known as its DBI which must be opened inside a transaction with the OpenDBI or OpenRoot methods. DBIs may be closed but it is not required. Typically, applications acquire handles for all their databases immediately after opening an environment and retain them for the lifetime of the process.
Transactions ¶
View (readonly) transactions in LMDB operate on a snapshot of the database at the time the transaction began. The number of simultaneously active view transactions is bounded and configured when the environment is initialized.
Update (read-write) transactions are serialized in LMDB. Attempts to create update transactions block until a lock may be obtained. Update transactions can create subtransactions which may be rolled back independently from their parent.
The lmdb package supplies managed and unmanaged transactions. Managed transactions do not require explicit calling of Abort/Commit and are provided through the Env methods Update, View, and RunTxn. The BeginTxn method on Env creates an unmanaged transaction but its use is not advised in most applications.
To provide ACID guarantees, a readonly transaction must acquire a "lock" in the LMDB environment to ensure that data it reads is consistent over the course of the transaction's lifetime, and that updates happening concurrently will not be seen. If a reader does not release its lock then stale data, which has been overwritten by later transactions, cannot be reclaimed by LMDB -- resulting in a rapid increase in file size.
Long-running read transactions may cause increase an applications storage requirements, depending on the application write workload. But, typically the complete failure of an application to terminate a read transactions will result in continual increase file size to the point where the storage volume becomes full or a quota has been reached.
There are steps an application may take to greatly reduce the possibility of unterminated read transactions. The first safety measure is to avoid the use of Env.BeginTxn, which creates unmanaged transactions, and always use Env.View or Env.Update to create managed transactions that are (mostly) guaranteed to terminate. If Env.BeginTxn must be used try to defer a call to the Txn's Abort method (this is useful even for update transactions).
txn, err := env.BeginTxn(nil, 0)
if err != nil {
// ...
}
defer txn.Abort() // Safe even if txn.Commit() is called later.
Because application crashes and signals from the operation system may cause unexpected termination of a readonly transaction before Txn.Abort may be called it is also important that applications clear any readers held for dead OS processes when they start.
numStale, err := env.ReaderCheck()
if err != nil {
// ...
}
if numStale > 0 {
log.Printf("Released locks for %d dead readers", numStale)
}
If an application gets accessed by multiple programs concurrently it is also a good idea to periodically call Env.ReaderCheck during application execution. However, note that Env.ReaderCheck cannot find readers opened by the application itself which have since leaked. This package installs no Txn finalizers: a leaked read-only Txn keeps its reader slot and pins its MVCC snapshot for the life of the Env, and a leaked write Txn holds the exclusive writer lock, blocking all further writes. Every transaction must therefore be terminated (Env.View and Env.Update do this automatically).
Caveats ¶
Write transactions (those created without the Readonly flag) must be created in a goroutine that has been locked to its thread by calling the function runtime.LockOSThread. Furthermore, all methods on such transactions must be called from the goroutine which created them. This is a fundamental limitation of LMDB even when using the NoTLS flag (which the package always uses). The Env.Update method assists the programmer by calling runtime.LockOSThread automatically but it cannot sufficiently abstract write transactions to make them completely safe in Go.
A goroutine must never create a write transaction if the application programmer cannot determine whether the goroutine is locked to an OS thread. This is a consequence of goroutine restrictions on write transactions and limitations in the runtime's thread locking implementation. In such situations updates desired by the goroutine in question must be proxied by a goroutine with a known state (i.e. "locked" or "unlocked"). See the included examples for more details about dealing with such situations.
Index ¶
- Constants
- Variables
- func BuildOptions() string
- func CursorToPool(c *Cursor)
- func DistributeCursors(first, last *Cursor, cursors []*Cursor, deepness uint) (allSet bool, err error)
- func GetSysRamInfo() (pageSize, totalPages, availablePages int, err error)
- func IsErrno(err error, errno Errno) bool
- func IsErrnoFn(err error, fn func(error) bool) bool
- func IsErrnoSys(err error, errno syscall.Errno) bool
- func IsKeyExists(err error) bool
- func IsMapFull(err error) bool
- func IsNoData(err error) bool
- func IsNotExist(err error) bool
- func IsNotFound(err error) bool
- func Version() string
- type Cmp
- type CmpFunc
- type CommitLatency
- type CommitLatencyGC
- type Cursor
- func (c *Cursor) Bind(txn *Txn, db DBI) error
- func (c *Cursor) Close()
- func (c *Cursor) Count() (uint64, error)
- func (c *Cursor) DBI() DBI
- func (c *Cursor) Del(flags uint) error
- func (c *Cursor) DeleteRange(end *Cursor, endIncluding bool) (numberAffected uint64, err error)
- func (c *Cursor) Distance(last *Cursor, deepness uint) (int, error)
- func (c *Cursor) EstimateDistance(last *Cursor) (int, error)
- func (c *Cursor) EstimateMove(key, data []byte, op uint) (int, error)
- func (c *Cursor) Get(setkey, setval []byte, op uint) (key, val []byte, err error)
- func (c *Cursor) IsClosed() bool
- func (c *Cursor) Open(txn *Txn, db DBI) error
- func (c *Cursor) Put(key, val []byte, flags uint) error
- func (c *Cursor) PutCurrent(key, val []byte) error
- func (c *Cursor) PutMulti(key []byte, page []byte, stride int, flags uint) error
- func (c *Cursor) PutReserve(key []byte, n int, flags uint) ([]byte, error)
- func (c *Cursor) RangeDel(mode uint) (numberAffected uint64, err error)
- func (c *Cursor) Renew(txn *Txn) error
- func (c *Cursor) Scroll(amount int, deepness uint) error
- func (c *Cursor) Txn() *Txn
- func (c *Cursor) Unbind() error
- type DBI
- type DefragOptions
- type DefragResult
- type Duration16dot16
- type EnfInfoPageOps
- type Env
- func (env *Env) BeginTxn(parent *Txn, flags uint) (*Txn, error)
- func (env *Env) CHandle() unsafe.Pointer
- func (env *Env) Close() error
- func (env *Env) CloseDBI(db DBI)
- func (env *Env) Copy(path string) error
- func (env *Env) CopyFD(fd uintptr) error
- func (env *Env) CopyFDFlag(fd uintptr, flags uint) error
- func (env *Env) CopyFlag(path string, flags uint) error
- func (env *Env) Defrag(opts DefragOptions) (*DefragResult, error)
- func (env *Env) FD() (uintptr, error)
- func (env *Env) Flags() (uint, error)
- func (env *Env) GetOption(option uint) (uint64, error)
- func (env *Env) GetSyncBytes() (uint, error)
- func (env *Env) GetSyncPeriod() (time.Duration, error)
- func (env *Env) Info(txn *Txn) (*EnvInfo, error)
- func (env *Env) Label() Label
- func (env *Env) MaxKeySize() int
- func (env *Env) Open(path string, flags uint, mode os.FileMode) error
- func (env *Env) Path() (string, error)
- func (env *Env) ReaderCheck() (int, error)
- func (env *Env) ReaderList(fn func(ReaderInfo) error) error
- func (env *Env) ReaderStats() (ReaderStats, error)
- func (env *Env) Readers() ([]ReaderInfo, error)
- func (env *Env) RunTxn(flags uint, fn TxnOp) error
- func (env *Env) SetDebug(logLvl LogLvl, dbg int, logger C.MDBX_debug_func) error
- func (env *Env) SetFlags(flags uint) error
- func (env *Env) SetGeometry(sizeLower int, sizeNow int, sizeUpper int, growthStep int, shrinkThreshold int, ...) error
- func (env *Env) SetOption(option uint, value uint64) error
- func (env *Env) SetStrictThreadMode(mode bool)
- func (env *Env) SetSyncBytes(threshold uint) error
- func (env *Env) SetSyncPeriod(value time.Duration) error
- func (env *Env) Stat() (*Stat, error)
- func (env *Env) Sync(force bool, nonblock bool) error
- func (env *Env) SyncForce() error
- func (env *Env) SyncPoll() error
- func (env *Env) UnsetFlags(flags uint) error
- func (env *Env) Update(fn TxnOp) error
- func (env *Env) UpdateLocked(fn TxnOp) error
- func (env *Env) View(fn TxnOp) error
- type EnvInfo
- type EnvInfoGeo
- type Errno
- type GCInfo
- type Label
- type LogLvl
- type Multi
- type OpError
- type ReaderInfo
- type ReaderStats
- type Stat
- type TxInfo
- type Txn
- func (txn *Txn) Abort()
- func (txn *Txn) Amend(flags uint) (writeTxn *Txn, snapshotTooOld bool, err error)
- func (txn *Txn) CHandle() unsafe.Pointer
- func (txn *Txn) Checkpoint(weakeningDurability uint) (lat CommitLatency, noChanges bool, err error)
- func (txn *Txn) Clone() (*Txn, error)
- func (txn *Txn) CloneInto(target *Txn) error
- func (txn *Txn) Cmp(dbi DBI, a []byte, b []byte) int
- func (txn *Txn) Commit() (CommitLatency, error)
- func (txn *Txn) CommitEmbarkRead() (lat CommitLatency, noChanges bool, err error)
- func (txn *Txn) CreateDBI(name string) (DBI, error)
- func (txn *Txn) DCmp(dbi DBI, a []byte, b []byte) int
- func (txn *Txn) Del(dbi DBI, key, val []byte) error
- func (txn *Txn) Drop(dbi DBI, del bool) error
- func (txn *Txn) EnvWarmup(flags uint, timeout time.Duration) error
- func (txn *Txn) EstimateRange(dbi DBI, beginKey, beginData, endKey, endData []byte) (int, error)
- func (txn *Txn) Flags(dbi DBI) (uint, error)
- func (txn *Txn) GCInfo() (*GCInfo, error)
- func (txn *Txn) Get(dbi DBI, key []byte) ([]byte, error)
- func (txn *Txn) ID() uint64
- func (txn *Txn) Info(scanRlt bool) (*TxInfo, error)
- func (txn *Txn) ListDBI() (res []string, err error)
- func (txn *Txn) OpenCursor(dbi DBI) (*Cursor, error)
- func (txn *Txn) OpenDBI(name string, flags uint, cmp, dcmp CmpFunc) (DBI, error)deprecated
- func (txn *Txn) OpenDBISimple(name string, flags uint) (DBI, error)
- func (txn *Txn) OpenRoot(flags uint) (DBI, error)
- func (txn *Txn) Park(autounpark bool) error
- func (txn *Txn) Put(dbi DBI, key, val []byte, flags uint) error
- func (txn *Txn) PutReserve(dbi DBI, key []byte, n int, flags uint) ([]byte, error)
- func (txn *Txn) Refresh() (atTip bool, err error)
- func (txn *Txn) ReleaseAllCursors(unbind bool) error
- func (txn *Txn) Renew() error
- func (txn *Txn) Reset() error
- func (txn *Txn) Rollback() error
- func (txn *Txn) RunOp(fn TxnOp, terminate bool) error
- func (txn *Txn) Sequence(dbi DBI, increment uint64) (uint64, error)
- func (txn *Txn) StatDBI(dbi DBI) (*Stat, error)
- func (txn *Txn) Sub(fn TxnOp) error
- func (txn *Txn) Unpark(restartIfOusted bool) (restarted bool, err error)
- type TxnOp
- Bugs
Constants ¶
const ( First = C.MDBX_FIRST // The first item. FirstDup = C.MDBX_FIRST_DUP // The first value of current key (DupSort). GetBoth = C.MDBX_GET_BOTH // Get the key as well as the value (DupSort). GetBothRange = C.MDBX_GET_BOTH_RANGE // Get the key and the nearsest value (DupSort). GetCurrent = C.MDBX_GET_CURRENT // Get the key and value at the current position. GetMultiple = C.MDBX_GET_MULTIPLE // Get up to a page dup values for key at current position (DupFixed). Last = C.MDBX_LAST // Last item. LastDup = C.MDBX_LAST_DUP // Position at last value of current key (DupSort). Next = C.MDBX_NEXT // Next value. NextDup = C.MDBX_NEXT_DUP // Next value of the current key (DupSort). NextMultiple = C.MDBX_NEXT_MULTIPLE // Get key and up to a page of values from the next cursor position (DupFixed). NextNoDup = C.MDBX_NEXT_NODUP // The first value of the next key (DupSort). Prev = C.MDBX_PREV // The previous item. PrevDup = C.MDBX_PREV_DUP // The previous item of the current key (DupSort). PrevNoDup = C.MDBX_PREV_NODUP // The last data item of the previous key (DupSort). PrevMultiple = C.MDBX_PREV_MULTIPLE // Set = C.MDBX_SET // The specified key. SetKey = C.MDBX_SET_KEY // Get key and data at the specified key. SetRange = C.MDBX_SET_RANGE // The first key no less than the specified key. SetLowerBound = C.MDBX_SET_LOWERBOUND // The first key/value pair no less than the specified key/value pair. SetUpperBound = C.MDBX_SET_UPPERBOUND // The first key/value pair greater than the specified key/value pair. KeyLesserThan = C.MDBX_TO_KEY_LESSER_THAN KeyLesserOrEqual = C.MDBX_TO_KEY_LESSER_OR_EQUAL KeyEqual = C.MDBX_TO_KEY_EQUAL KeyGreaterOrEqual = C.MDBX_TO_KEY_GREATER_OR_EQUAL KeyGreaterThan = C.MDBX_TO_KEY_GREATER_THAN ExactKeyValueLesserThan = C.MDBX_TO_EXACT_KEY_VALUE_LESSER_THAN ExactKeyValueLesserOrEqual = C.MDBX_TO_EXACT_KEY_VALUE_LESSER_OR_EQUAL ExactKeyValueEqual = C.MDBX_TO_EXACT_KEY_VALUE_EQUAL ExactKeyValueGreaterOrEqual = C.MDBX_TO_EXACT_KEY_VALUE_GREATER_OR_EQUAL ExactKeyValueGreaterThan = C.MDBX_TO_EXACT_KEY_VALUE_GREATER_THAN PairLesserThan = C.MDBX_TO_PAIR_LESSER_THAN PairLesserOrEqual = C.MDBX_TO_PAIR_LESSER_OR_EQUAL PairEqual = C.MDBX_TO_PAIR_EQUAL PairGreaterOrEqual = C.MDBX_TO_PAIR_GREATER_OR_EQUAL PairGreaterThan = C.MDBX_TO_PAIR_GREATER_THAN LesserThan = KeyLesserThan // Deprecated: use KeyLesserThan. )
const ( // Flags for Txn.Put and Cursor.Put. // // See mdb_put and mdb_cursor_put. Upsert = C.MDBX_UPSERT // Replace the item at the current key position (Cursor only) Current = C.MDBX_CURRENT // Replace the item at the current key position (Cursor only) NoDupData = C.MDBX_NODUPDATA // Store the key-value pair only if key is not present (DupSort). NoOverwrite = C.MDBX_NOOVERWRITE // Store a new key-value pair only if key is not present. Append = C.MDBX_APPEND // Append an item to the database. AppendDup = C.MDBX_APPENDDUP // Append an item to the database (DupSort). AllDups = C.MDBX_ALLDUPS )
The MDB_MULTIPLE and MDB_RESERVE flags are special and do not fit the calling pattern of other calls to Put. They are not exported because they require special methods, PutMultiple and PutReserve in which the flag is implied and does not need to be passed.
const ( // Flags for Cursor.RangeDel // // See mdbx_cursor_bunch_delete. DeleteCurrentValue = C.MDBX_DELETE_CURRENT_VALUE DeleteCurrentMultiValBeforeExcluding = C.MDBX_DELETE_CURRENT_MULTIVAL_BEFORE_EXCLUDING DeleteCurrentMultiValBeforeIncluding = C.MDBX_DELETE_CURRENT_MULTIVAL_BEFORE_INCLUDING DeleteCurrentMultiValAfterIncluding = C.MDBX_DELETE_CURRENT_MULTIVAL_AFTER_INCLUDING DeleteCurrentMultiValAfterExcluding = C.MDBX_DELETE_CURRENT_MULTIVAL_AFTER_EXCLUDING DeleteCurrentValueMultiValAll = C.MDBX_DELETE_CURRENT_MULTIVAL_ALL DeleteBeforeExcluding = C.MDBX_DELETE_BEFORE_EXCLUDING DeleteBeforeIncluding = C.MDBX_DELETE_BEFORE_INCLUDING DeleteAfterIncluding = C.MDBX_DELETE_AFTER_INCLUDING DeleteAfterExcluding = C.MDBX_DELETE_AFTER_EXCLUDING DeleteWhole = C.MDBX_DELETE_WHOLE )
const ( EnvDefaults = C.MDBX_ENV_DEFAULTS LifoReclaim = C.MDBX_LIFORECLAIM // FixedMap = C.MDBX_FIXEDMAP // Danger zone. Map memory at a fixed address. NoSubdir = C.MDBX_NOSUBDIR // Argument to Open is a file, not a directory. Accede = C.MDBX_ACCEDE Readonly = C.MDBX_RDONLY // Used in several functions to denote an object as readonly. WriteMap = C.MDBX_WRITEMAP // Use a writable memory map. NoMetaSync = C.MDBX_NOMETASYNC // Don't fsync metapage after commit. UtterlyNoSync = C.MDBX_UTTERLY_NOSYNC SafeNoSync = C.MDBX_SAFE_NOSYNC Durable = C.MDBX_SYNC_DURABLE // Deprecated: use NoStickyThreads instead. libmdbx removed the MDBX_NOTLS // alias (superseded by MDBX_NOSTICKYTHREADS since 0.13), so this now maps // to the same flag value and keeps existing callers compiling. NoTLS = C.MDBX_NOSTICKYTHREADS // Danger zone. When unset reader locktable slots are tied to their thread. NoStickyThreads = C.MDBX_NOSTICKYTHREADS // Danger zone. Like MDBX_NOTLS. But also allow move RwTx between threads. Still require to call Begin/Rollback in same thread. // NoLock = C.MDBX_NOLOCK // Danger zone. MDBX does not use any locks. NoReadahead = C.MDBX_NORDAHEAD // Disable readahead. Requires OS support. NoMemInit = C.MDBX_NOMEMINIT // Disable MDBX memory initialization. Exclusive = C.MDBX_EXCLUSIVE // Open the environment in exclusive/monopolistic mode. )
const ( MinPageSize = C.MDBX_MIN_PAGESIZE MaxPageSize = C.MDBX_MAX_PAGESIZE MaxDbi = C.MDBX_MAX_DBI )
const ( CopyDefaults = C.MDBX_CP_DEFAULTS // Perform copy as-is, without compaction CopyCompact = C.MDBX_CP_COMPACT // Perform compaction while copying: omit free pages and renumber CopyForceDynamicSize = C.MDBX_CP_FORCE_DYNAMIC_SIZE // Force resizable copy (dynamic size instead of fixed) CopyDontFlush = C.MDBX_CP_DONT_FLUSH // Don't explicitly flush the written data to output media CopyThrottleMVCC = C.MDBX_CP_THROTTLE_MVCC // Use read transaction parking during copying MVCC-snapshot CopyOverwrite = C.MDBX_CP_OVERWRITE // Silently overwrite the target file if it exists )
These flags are exclusively used in the Env.CopyFlag and Env.CopyFDFlag methods.
const ( DefragStepSize = C.MDBX_defrag_step_size // Step transaction size limit reached DefragLargeChunk = C.MDBX_defrag_large_chunk // Preliminary movement is necessary DefragLaggardReader = C.MDBX_defrag_laggard_reader // A reader is preventing further defragmentation DefragEnoughThreshold = C.MDBX_defrag_enough_threshold // User-set goal achieved DefragTimeLimit = C.MDBX_defrag_time_limit // Specified time limit reached DefragError = C.MDBX_defrag_error // An error occurred during defragmentation )
DefragResult.StoppingReasons is an OR'ed mask of these; zero means no obstacles. MDBX_defrag_discontinued and MDBX_defrag_aborted are omitted, being reachable only through the progress callback Env.Defrag passes as NULL.
See MDBX_defrag_stopping_reasons_t.
const ( LogLvlFatal = C.MDBX_LOG_FATAL LogLvlError = C.MDBX_LOG_ERROR LogLvlWarn = C.MDBX_LOG_WARN LogLvlNotice = C.MDBX_LOG_NOTICE LogLvlVerbose = C.MDBX_LOG_VERBOSE LogLvlDebug = C.MDBX_LOG_DEBUG LogLvlTrace = C.MDBX_LOG_TRACE LogLvlExtra = C.MDBX_LOG_EXTRA LogLvlDoNotChange = C.MDBX_LOG_DONTCHANGE )
const ( DbgAssert = C.MDBX_DBG_ASSERT DbgAudit = C.MDBX_DBG_AUDIT DbgJitter = C.MDBX_DBG_JITTER DbgDump = C.MDBX_DBG_DUMP DbgLegacyMultiOpen = C.MDBX_DBG_LEGACY_MULTIOPEN DbgLegacyTxOverlap = C.MDBX_DBG_LEGACY_OVERLAP DbgDoNotChange = C.MDBX_DBG_DONTCHANGE )
const ( OptMaxDB = C.MDBX_opt_max_db OptMaxReaders = C.MDBX_opt_max_readers OptSyncBytes = C.MDBX_opt_sync_bytes OptSyncPeriod = C.MDBX_opt_sync_period OptRpAugmentLimit = C.MDBX_opt_rp_augment_limit OptLooseLimit = C.MDBX_opt_loose_limit // OptDpReserveLimit limits dirty-page instances kept in libmdbx's reserve // pool between write transactions (MDBX_opt_dp_reserve_limit). OptDpReserveLimit = C.MDBX_opt_dp_reserve_limit // Deprecated: misnamed alias of OptDpReserveLimit. OptDpReverseLimit = C.MDBX_opt_dp_reserve_limit OptTxnDpLimit = C.MDBX_opt_txn_dp_limit OptTxnDpInitial = C.MDBX_opt_txn_dp_initial OptSpillMaxDenominator = C.MDBX_opt_spill_max_denominator OptSpillMinDenominator = C.MDBX_opt_spill_min_denominator OptSpillParent4ChildDenominator = C.MDBX_opt_spill_parent4child_denominator OptMergeThreshold16dot16Percent = C.MDBX_opt_merge_threshold OptPreferWafInsteadofBalance = C.MDBX_opt_prefer_waf_insteadof_balance OptGCTimeLimit = C.MDBX_opt_gc_time_limit // OptPrefaultWriteEnable controls the prefault-write optimization (mincore() + pwrite() // of each not-in-core page before it is touched via the writemap). It only pays off when // the database is much larger than RAM; when the working set fits in RAM it is pure // overhead. Set to 0 to disable, 1 to force-enable, or use the env default when unset. OptPrefaultWriteEnable = C.MDBX_opt_prefault_write_enable // OptPresyncThreshold sets how many bytes of not-yet-synced data may accumulate before // SyncPoll/Sync performs a preliminary flush without holding the transaction lock // (MDBX_opt_presync_threshold, new in libmdbx 0.14.3). Pushing the bulk of the data out // early shortens the locked final stage of the sync. A too large threshold lengthens the // latency spikes it is meant to smooth; a too small one multiplies sync calls and their // overhead. Value in bytes: minimum 1, maximum 2 GiB, default 256 KiB. OptPresyncThreshold = C.MDBX_opt_presync_threshold )
const ( ReverseKey = C.MDBX_REVERSEKEY // Use reverse string keys. DupSort = C.MDBX_DUPSORT // Use sorted duplicates. DupFixed = C.MDBX_DUPFIXED // Duplicate items have a fixed size (DupSort). ReverseDup = C.MDBX_REVERSEDUP // Reverse duplicate values (DupSort). Create = C.MDBX_CREATE // Createt DB if not already existing. DBAccede = C.MDBX_DB_ACCEDE // Use sorted duplicates. )
This flags are used exclusively for Txn.OpenDBISimple and Txn.OpenRoot. The Create flag must always be supplied when opening a non-root DBI for the first time.
BUG(bmatsuo): MDBX_INTEGERKEY and MDBX_INTEGERDUP aren't usable. I'm not sure they would be faster with the cgo bridge. They need to be tested and benchmarked.
const ( TxRW = C.MDBX_TXN_READWRITE TxRO = C.MDBX_TXN_RDONLY TxPrepareRO = C.MDBX_TXN_RDONLY_PREPARE TxTry = C.MDBX_TXN_TRY TxNoMetaSync = C.MDBX_TXN_NOMETASYNC TxNoSync = C.MDBX_TXN_NOSYNC )
const ( WarmupDefault = C.MDBX_warmup_default WarmupForce = C.MDBX_warmup_force WarmupOomSafe = C.MDBX_warmup_oomsafe WarmupLock = C.MDBX_warmup_lock WarmupTouchLimit = C.MDBX_warmup_touchlimit WarmupRelease = C.MDBX_warmup_release )
const (
AllowTxOverlap = C.MDBX_DBG_LEGACY_OVERLAP
)
const Major = C.MDBX_VERSION_MAJOR
const Minor = C.MDBX_VERSION_MINOR
Variables ¶
var CorruptErrorBacktraceRecommendations = "Otherwise - please create issue in Application repo." // with backtrace or coredump. To create coredump set compile option 'MDBX_FORCE_ASSERTIONS=1' and env variable 'GOTRACEBACK=crash'."
var CorruptErrorHardwareRecommendations = "" /* 403-byte string literal not displayed */
App can re-define this messages from init() func
var CorruptErrorMessage = CorruptErrorHardwareRecommendations + " " + CorruptErrorBacktraceRecommendations + " " + CorruptErrorRecoveryRecommendations
var CorruptErrorRecoveryRecommendations = "" /* 191-byte string literal not displayed */
var ErrNoData = errors.New("cursor is not positioned to data")
ErrNoData is returned when a cursor operation reads the current position of a cursor that is not positioned to any data (a "hollow" cursor), e.g. MDBX_GET_CURRENT on a cursor that was never moved. Distinct from ErrNotFound, which reports that a searched-for key/value does not exist.
var ErrNotFound = errors.New("key not found")
var (
LoggerDoNotChange = C.MDBX_LOGGER_DONTCHANGE
)
var MapFullErrorMessage = "The allocated database storage size limit has been reached."
var ReaderTIDTxnOusted = readerTIDTxnOusted
ReaderTIDTxnOusted is the raw thread marker used by libmdbx for ousted read transactions. Prefer ReaderInfo.Ousted for application logic.
var ReaderTIDTxnParked = readerTIDTxnParked
ReaderTIDTxnParked is the raw thread marker used by libmdbx for parked read transactions. Prefer ReaderInfo.Parked for application logic.
Functions ¶
func BuildOptions ¶ added in v0.39.16
func BuildOptions() string
BuildOptions returns the build-time configuration options used when compiling libmdbx. This includes settings like MDBX_USE_FALLOCATE, MDBX_DEBUG, and other compile-time flags.
Example output: " MDBX_DEBUG=0 ... MDBX_USE_FALLOCATE=1 ..."
func CursorToPool ¶
func CursorToPool(c *Cursor)
CursorToPool returns c for reuse. Closed cursors are dropped (their handle is gone; Bind/Renew would always fail). The cursor's Go txn reference is cleared so a pooled cursor cannot operate on — or keep alive — an ended transaction; callers must Bind/Renew before use. Note cursors evicted from the pool by GC leak their C allocation — Close cursors you do not re-pool.
func DistributeCursors ¶ added in v0.40.2
func DistributeCursors(first, last *Cursor, cursors []*Cursor, deepness uint) (allSet bool, err error)
DistributeCursors positions each cursor in cursors at an evenly-spaced position across the range delimited by first and last, so that consecutive cursors delimit approximately equally-sized sub-ranges. This is the intended primitive for splitting a key range into balanced chunks for parallel workers (e.g. concurrent range deletion or warm-up).
A nil first means the beginning of the table and a nil last means the end; at least one of them must be non-nil and positioned. Every entry in cursors must be a non-nil open cursor bound to the same table and transaction (a nil entry returns an error); these are the cursors that get positioned.
deepness selects the B-tree level at which the distribution is computed: for the positions to match a number of keys/values it must be at least the B-tree height (plus the nested height for DupSort tables); when in doubt pass a deliberately large value such as 42.
allSet reports whether every cursor was positioned. It is false (with a nil error) when the range held fewer positions than len(cursors); the surplus cursors are left unset (at EOF).
See mdbx_cursor_distribute.
func GetSysRamInfo ¶ added in v0.39.1
func IsErrnoFn ¶
IsErrnoFn calls fn on the error underlying err and returns the result. If err is an *OpError then err.Errno is passed to fn. Otherwise err is passed directly to fn.
func IsErrnoSys ¶
IsErrnoSys returns true if err's errno is the given errno.
func IsKeyExists ¶
func IsNoData ¶ added in v0.40.3
IsNoData returns true if a cursor read (e.g. Cursor.Get with MDBX_GET_CURRENT) hit a cursor that is not positioned to any data. See ErrNoData.
func IsNotExist ¶
IsNotExist returns true the path passed to the Env.Open method does not exist.
func IsNotFound ¶
IsNotFound returns true if the key requested in Txn.Get or Cursor.Get does not exist or if the Cursor reached the end of the database without locating a value (EOF).
Types ¶
type CmpFunc ¶
type CmpFunc *C.MDBX_cmp_func
type CommitLatency ¶
type CommitLatencyGC ¶ added in v0.27.18
type CommitLatencyGC struct {
/** \brief The time "by the wall clock" spent reading and searching inside the GC for the user's data. */
WorkRtime time.Duration
/** \brief The number of search iterations inside GC when allocating pages for the sake of user's data. */
WorkRsteps uint32
/** \brief The number of requests to allocate page sequences for the sake of user's data. */
WorkRxpages uint32
WorkMajflt uint32
SelfMajflt uint32
WorkCounter uint32
SelfCounter uint32
/** \brief The time "by the wall clock" spent reading and searching inside the GC
* for the purposes of maintaining and updating the GC itself. */
SelfRtime time.Duration
SelfXtime time.Duration
WorkXtime time.Duration
/** \brief The number of search iterations inside the GC when allocating pages for the purposes
* of maintaining and updating the GC itself. */
SelfRsteps uint32
/** \brief The number of page sequences allocation requests for the GC itself. */
SelfXpages uint32
/** \brief The number of GC update iterations is greater than 1 if there were repeats/restarts. */
Wloops uint32
/** \brief The number of iterations of merging GC items. */
Coalescences uint32
Wipes uint32
Flushes uint32
Kicks uint32
/** \brief The maximum observed difference between the latest and oldest reader MVCC snapshots. */
MaxReaderLag uint32
/** \brief The maximum observed number of pages withheld from reclamation due to readers holding old MVCC snapshots. */
MaxRetainedPages uint32
}
type Cursor ¶
type Cursor struct {
// contains filtered or unexported fields
}
Cursor operates on data inside a transaction and holds a position in the database.
See MDB_cursor.
func CreateCursor ¶
func CreateCursor() *Cursor
func CursorFromPool ¶
func CursorFromPool() *Cursor
func (*Cursor) Bind ¶
Bind Using of the `mdbx_cursor_bind()` is equivalent to calling mdbx_cursor_renew() but with specifying an arbitrary dbi handle.
func (*Cursor) Close ¶
func (c *Cursor) Close()
Close the cursor handle and free its underlying libmdbx cursor. Unlike LMDB, libmdbx never frees cursors at transaction end, so Close is required for every cursor; it may be called before or after the txn ends (a live write-txn cursor must be closed from the txn's own thread). No-op if already closed. Do not Close after ReleaseAllCursors(false) — see there.
See mdbx_cursor_close.
func (*Cursor) Count ¶
Count returns the number of duplicates for the current key.
See mdb_cursor_count.
func (*Cursor) DBI ¶
DBI returns the cursor's database handle. If c has been closed than an invalid DBI is returned.
func (*Cursor) Del ¶
Del deletes the item referred to by the cursor from the database.
See mdb_cursor_del.
func (*Cursor) DeleteRange ¶ added in v0.40.2
DeleteRange performs a mass deletion of the items between the position of the receiver cursor (the beginning of the range) and the position of end (the end of the range). It is much faster than deleting items one by one because whole pages and branches are cut out of the B+ tree.
Both cursors must already be positioned (see Cursor.Get) and bound to the same table and write transaction. A nil end deletes up to the last item. endIncluding controls whether the item at end's position is itself deleted.
It returns the number of deleted items.
See mdbx_cursor_delete_range.
func (*Cursor) Distance ¶ added in v0.40.2
Distance calculates the number of elements between the position of the receiver cursor and the position of last. Both cursors must be positioned and bound to the same table and transaction; a nil last means the end of the table.
deepness limits the B-tree level at which the difference is measured: 0 is the root (fast but rough), larger values descend toward the leaves (slower but exact). For an exact count deepness must be at least the B-tree height (plus the nested height for DupSort tables); when in doubt pass a deliberately large value such as 42.
See mdbx_cursor_distance.
func (*Cursor) EstimateDistance ¶ added in v0.40.2
EstimateDistance estimates the number of elements between the position of the receiver cursor and the position of last. Both cursors must be non-nil, positioned, and initialized for the same table and transaction. Unlike Distance, this estimate has no end-of-table sentinel: a nil last is not a valid argument. The result is a rough estimate suitable for building/optimizing query plans, not an exact count.
See mdbx_estimate_distance.
func (*Cursor) EstimateMove ¶ added in v0.40.2
EstimateMove estimates the distance, as a number of elements, that the cursor would move if the given key/data + op were applied via Get. The cursor's own position and state are left unchanged. The result is a rough estimate suitable for building/optimizing query plans, not an exact count.
key/data are interpreted the same way as in Get for the given op; pass nil where the op does not require them.
See mdbx_estimate_move.
func (*Cursor) Get ¶
Get retrieves items from the database. Returned key/val are zero-copy views into the memory-mapped file (except key for op Set, see below): read-only and valid only until the next update operation in a write txn or until the transaction ends. Copy them if they must live longer.
The Set op returns a key sharing memory with setkey (the caller's own buffer, not the database file); it stays valid after the txn ends. For FirstDup and LastDup only val is returned; key is nil.
setkey/setval are forwarded to mdbx_cursor_get as the op requires; ops that take no input value ignore setval.
See mdbx_cursor_get.
func (*Cursor) IsClosed ¶ added in v0.41.1
IsClosed reports whether the cursor holds no libmdbx cursor, i.e. it is nil, was never opened, or has already been closed. Nil-safe so it fully replaces the nil-*Cursor checks callers used before Open existed.
func (*Cursor) Open ¶ added in v0.41.1
Open binds an unopened Cursor to the table in place. Unlike Txn.OpenCursor it does not allocate the Cursor, so callers may embed Cursor by value. Close is still required.
func (*Cursor) PutCurrent ¶ added in v0.39.16
PutCurrent replaces the data of the item at the current cursor position. For DupSort databases, this replaces the current duplicate entry in-place, avoiding a separate Del+Put round-trip (saves one CGo call per update). The cursor must be positioned (e.g. via Get with GetBothRange) before calling.
Equivalent to Put(key, val, Current).
func (*Cursor) PutMulti ¶
PutMulti stores a set of contiguous items with stride size under key. PutMulti panics if len(page) is not a multiple of stride. The cursor's database must be DupFixed and DupSort.
See mdb_cursor_put.
func (*Cursor) PutReserve ¶
PutReserve returns a []byte of length n that can be written to, potentially avoiding a memcopy. The returned byte slice is only valid in txn's thread, before it has terminated.
func (*Cursor) RangeDel ¶ added in v0.39.18
RangeDel deletes a range of items referred to by the cursor from the database.
It returns the number of affected (deleted) items. Modes: see mdbx_cursor_bunch_delete.
func (*Cursor) Scroll ¶ added in v0.40.2
Scroll moves the cursor by amount logical positions at the given B-tree level. A positive amount moves forward (toward the end of the table), a negative one moves backward. The cursor must already be positioned.
deepness selects the B-tree level the steps are taken at: for the movement to match a number of keys/values it must be at least the B-tree height (plus the nested height for DupSort tables); when in doubt pass a deliberately large value such as 42.
Scroll returns an error for which IsNotFound reports true if the end of the data is reached before the cursor moved by the full amount.
See mdbx_cursor_scroll.
type DefragOptions ¶ added in v0.44.0
type DefragOptions struct {
DefragAtLeast uint64 // shrink by at least this many pages; must be <= DefragEnough
TimeAtLeast time.Duration // keep going at least this long; must be <= TimeLimit
DefragEnough uint64 // stop once shrunk by this many pages
TimeLimit time.Duration // stop after this long
// AcceptableBacklash stops defrag once a further cycle would gain no more
// than this many pages. -1 selects autopilot. libmdbx silently clamps it
// to one GC overflow page of page numbers, ~1018 at a 4KiB page size, so
// larger values all behave alike.
AcceptableBacklash int64
PreferredBatch int64 // preferred max pages moved per cycle
}
DefragOptions controls Env.Defrag. Counts are pages, not bytes. Zero means "no bound" for every field.
See mdbx_env_defrag.
type DefragResult ¶ added in v0.44.0
type DefragResult struct {
PagesShrunk int64 // Pages the file shrank by; negative if it could not shrink.
PagesMoved uint64 // Total pages moved during defragmentation.
PagesScheduled uint64 // Pages scheduled to move at the next stage of the current cycle.
PagesRetained uint64 // Pages held by other processes via MVCC-snapshots.
PagesLeft uint64 // Estimated remaining defragmentable pages.
PagesWhole uint64 // Total number of pages in the database.
ObstructedPgNo uint64 // Page where defragmentation stumbled.
ObstructedSpan uint64 // Length of the large/overflow-page span where it stumbled.
ObstructedTxnID uint64 // Earliest MVCC-snapshot txnid preventing defragmentation.
ObstructorTID uint64 // Native TID of one of the blocking readers.
ObstructorPID int64 // Native PID of one of the blocking readers.
CycleProgress uint // Rough estimate of current cycle progress in permilles (1000 = 100%).
Cycles uint // Number of defragmentation cycles performed.
StoppingReasons uint // OR'ed mask of Defrag* stopping reasons.
SpentTime time.Duration
}
DefragResult holds the metrics returned by Env.Defrag.
See MDBX_defrag_result_t.
type Duration16dot16 ¶
type Duration16dot16 uint64
func NewDuration16dot16 ¶
func NewDuration16dot16(duration time.Duration) Duration16dot16
NewDuration16dot16 converts duration to libmdbx's 1/65536-second units, saturating rather than wrapping. libmdbx reads 0 as "no bound", so only a zero duration may produce it: negative and sub-unit durations become the smallest bound instead.
func (Duration16dot16) ToDuration ¶
func (d Duration16dot16) ToDuration() time.Duration
type EnfInfoPageOps ¶
type EnfInfoPageOps struct {
Newly uint64 /**< Quantity of a new pages added */
Cow uint64 /**< Quantity of pages copied for update */
Clone uint64 /**< Quantity of parent's dirty pages clones for nested transactions */
Split uint64 /**< Page splits */
Merge uint64 /**< Page merges */
Spill uint64 /**< Quantity of spilled dirty pages */
Unspill uint64 /**< Quantity of unspilled/reloaded pages */
Wops uint64 /**< Number of explicit write operations (not a pages) to a disk */
Minicore uint64 /**< Number of mincore() calls */
Prefault uint64 /**< Number of prefault write operations (not a pages) */
Msync uint64 /**< Number of explicit write operations (not a pages) to a disk */
Fsync uint64 /**< Number of explicit write operations (not a pages) to a disk */
}
type Env ¶
type Env struct {
// contains filtered or unexported fields
}
Env is opaque structure for a database environment. A DB environment supports multiple databases, all residing in the same shared-memory map.
See MDBX_env.
func (*Env) BeginTxn ¶
BeginTxn is an unsafe, low-level method to initialize a new transaction on env. The Txn returned by BeginTxn is unmanaged and must be terminated by calling either its Abort or Commit methods to ensure that its resources are released.
BeginTxn does not call runtime.LockOSThread. Unless the Readonly flag is passed goroutines must call runtime.LockOSThread before calling BeginTxn and the returned Txn must not have its methods called from another goroutine. Failure to meet these restrictions can have undefined results that may include deadlocking your application.
Instead of calling BeginTxn users should prefer calling the View and Update methods, which assist in management of Txn objects and provide OS thread locking required for write transactions.
Unterminated transactions can adversly effect database performance and cause the database to grow until the map is full.
See mdbx_txn_begin.
func (*Env) Close ¶
Close shuts down the environment and releases the memory map. On MDBX_BUSY (a write transaction is running in another thread) libmdbx keeps the handle alive: Close returns the error and env stays open so the caller can retry. On any other failure libmdbx has already destroyed the handle ("If any other error code was returned then given MDBX_env instance has been destroyed and released", mdbx.h), so env is marked closed and only the error is reported. Nil if already closed.
See mdbx_env_close.
func (*Env) CloseDBI ¶
CloseDBI closes the database handle, db. Normally calling CloseDBI explicitly is not necessary.
It is the caller's responsibility to serialize calls to CloseDBI.
See mdbx_dbi_close.
func (*Env) Copy ¶ added in v0.44.0
Copy copies the data in env as-is to an environment at path. The target path must not already exist; pass CopyOverwrite via CopyFlag to overwrite.
See mdbx_env_copy.
func (*Env) CopyFD ¶ added in v0.44.0
CopyFD copies env as-is to the file descriptor fd.
See mdbx_env_copy2fd.
func (*Env) CopyFDFlag ¶ added in v0.44.0
CopyFDFlag copies env to the file descriptor fd, with options. On Windows fd must be a native HANDLE value (as returned by os.File.Fd); on POSIX it is a regular int file descriptor.
See mdbx_env_copy2fd.
func (*Env) CopyFlag ¶ added in v0.44.0
CopyFlag copies the data in env to an environment at path, with options.
See mdbx_env_copy.
func (*Env) Defrag ¶ added in v0.44.0
func (env *Env) Defrag(opts DefragOptions) (*DefragResult, error)
Defrag defragments the database in place: pages near the end of the file are moved into free pages nearer the beginning, then the trailing free pages are cut off. It is ACID and runs in several committed cycles.
Open the environment with Exclusive: cutting the tail needs the whole-file lock, and without it Windows fails the shrink with ERROR_LOCK_VIOLATION.
The result is non-nil even on error. Reaching the requested goals only partly is not an error; read result.StoppingReasons for the reason. err can be LaggardReader, which also means "stopped early" rather than "failed".
See mdbx_env_defrag.
func (*Env) FD ¶ added in v0.39.16
FD returns the open file descriptor (or Windows file handle) for the given environment. An error is returned if the environment has not been successfully Opened (where C API just retruns an invalid handle).
See mdbx_env_get_fd.
func (*Env) GetSyncBytes ¶ added in v0.38.6
func (*Env) Info ¶
Info returns information about the environment.
txn may be nil: mdbx_env_info_ex accepts a NULL transaction and reports the last committed snapshot. Passing a txn pins the report to its snapshot. On an Env that was created but not yet opened, the C API reports success with only a subset of fields populated (geometry, page sizes, and other pre-open state; earlier versions failed because they could not begin the temporary read txn).
See mdbx_env_info_ex.
func (*Env) MaxKeySize ¶
MaxKeySize returns the maximum allowed length for a key.
See mdbx_env_get_maxkeysize.
func (*Env) Open ¶
Open an environment handle. If this function fails Close() must be called to discard the Env handle. Open always ORs in NoStickyThreads (formerly NoTLS), so transactions are not tied to the OS thread that created them.
WARNING: because NoStickyThreads is always set, as of libmdbx 0.14.x the env functions that need the writer lock but take no txn — SetFlags, SetOption, SetGeometry, Sync/SyncForce/SyncPoll, Stat/Info(nil) and Close — acquire that lock when the calling goroutine does not own the in-flight write transaction. Calling any of them from a goroutine that the write transaction is itself waiting on will deadlock. Perform such calls from the writer's own goroutine, or while no write transaction is running.
See mdbx_env_open.
func (*Env) Path ¶
Path returns the path argument passed to Open. Path returns a non-nil error if env.Open() was not previously called.
See mdbx_env_get_path.
func (*Env) ReaderCheck ¶
ReaderCheck clears stale entries from the reader lock table and returns the number of entries cleared.
See mdbx_reader_check()
func (*Env) ReaderList ¶ added in v0.39.18
func (env *Env) ReaderList(fn func(ReaderInfo) error) error
ReaderList enumerates the MDBX reader lock table as structured records.
BytesRetained is the approximate amount of data prevented from reuse by the reader's MVCC snapshot.
func (*Env) ReaderStats ¶ added in v0.39.18
func (env *Env) ReaderStats() (ReaderStats, error)
ReaderStats returns aggregate MDBX reader metrics.
func (*Env) Readers ¶ added in v0.39.18
func (env *Env) Readers() ([]ReaderInfo, error)
Readers returns a snapshot of all entries in the MDBX reader lock table.
func (*Env) RunTxn ¶
RunTxn creates a new Txn and calls fn with it as an argument. Run commits the transaction if fn returns nil otherwise the transaction is aborted. Because RunTxn terminates the transaction goroutines should not retain references to it or its data after fn returns.
RunTxn does not call runtime.LockOSThread. Unless the Readonly flag is passed the calling goroutine should ensure it is locked to its thread and any goroutines started by fn must not call methods on the Txn object it is passed.
See mdbx_txn_begin.
func (*Env) SetGeometry ¶
func (*Env) SetStrictThreadMode ¶ added in v0.39.4
SetStrictThreadMode in this mode mdbx panics when tx opening and closing are happening in different threads
func (*Env) SetSyncBytes ¶ added in v0.38.6
func (*Env) Sync ¶
Sync flushes buffers to disk. If force is true a synchronous flush occurs and ignores any NoMetaSync/SafeNoSync/UtterlyNoSync flag on the environment.
See mdbx_env_sync.
func (*Env) SyncForce ¶ added in v0.40.3
SyncForce forces a synchronous flush of the data buffers to disk, ignoring any NoMetaSync/SafeNoSync/UtterlyNoSync flag on the environment. It blocks if a write transaction is running on another thread.
It is the shortcut to calling Sync(force=true, nonblock=false).
See mdbx_env_sync.
func (*Env) SyncPoll ¶ added in v0.40.3
SyncPoll runs the lazy/asynchronous sync in polling mode: it checks the thresholds set by SetSyncBytes and/or SetSyncPeriod and flushes unsynced data to disk only when at least one threshold is reached. It does not wait if a write transaction is running on another thread (returns MDBX_BUSY instead).
A nil error is returned both when a flush happened and when there was nothing pending to flush (MDBX_RESULT_TRUE).
It is the shortcut to calling Sync(force=false, nonblock=true).
See mdbx_env_sync_poll.
func (*Env) Update ¶
Update calls fn with a writable transaction. Update commits the transaction if fn returns a nil error otherwise Update aborts the transaction and returns the error.
Update calls runtime.LockOSThread to lock the calling goroutine to its thread and until fn returns and the transaction has been terminated, at which point runtime.UnlockOSThread is called. If the calling goroutine is already known to be locked to a thread, use UpdateLocked instead to avoid premature unlocking of the goroutine.
Neither Update nor UpdateLocked cannot be called safely from a goroutine where it isn't known if runtime.LockOSThread has been called. In such situations writes must either be done in a newly created goroutine which can be safely locked, or through a worker goroutine that accepts updates to apply and delivers transaction results using channels. See the package documentation and examples for more details.
Goroutines created by the operation fn must not use methods on the Txn object that fn is passed. Doing so would have undefined and unpredictable results for your program (likely including data loss, deadlock, etc).
Any call to Commit, Abort, Reset or Renew on a Txn created by Update will panic.
func (*Env) UpdateLocked ¶
UpdateLocked behaves like Update but does not lock the calling goroutine to its thread. UpdateLocked should be used if the calling goroutine is already locked to its thread for another purpose.
Neither Update nor UpdateLocked cannot be called safely from a goroutine where it isn't known if runtime.LockOSThread has been called. In such situations writes must either be done in a newly created goroutine which can be safely locked, or through a worker goroutine that accepts updates to apply and delivers transaction results using channels. See the package documentation and examples for more details.
Goroutines created by the operation fn must not use methods on the Txn object that fn is passed. Doing so would have undefined and unpredictable results for your program (likely including data loss, deadlock, etc).
Any call to Commit, Abort, Reset or Renew on a Txn created by UpdateLocked will panic.
func (*Env) View ¶
View creates a readonly transaction with a consistent view of the environment and passes it to fn. View terminates its transaction after fn returns. Any error encountered by View is returned.
Unlike with Update transactions, goroutines created by fn are free to call methods on the Txn passed to fn provided they are synchronized in their accesses (e.g. using a mutex or channel).
Any call to Commit, Abort, Reset or Renew on a Txn created by View will panic.
type EnvInfo ¶
type EnvInfo struct {
MapSize int64 // Size of the data memory map
LastPNO int64 // ID of the last used page
Geo EnvInfoGeo
/** Statistics of page operations.
* \details Overall statistics of page operations of all (running, completed
* and aborted) transactions in the current multi-process session (since the
* first process opened the database). */
PageOps EnfInfoPageOps
LastTxnID int64 // ID of the last committed transaction. keep for backward compatibility - use RecentTxnID
RecentTxnID uint64 // ID of the last committed transaction
LatterReaderTxnID uint64 // ID of the last reader transaction
MaxReaders uint // maximum number of threads for the environment
NumReaders uint // maximum number of threads used in the environment
PageSize uint //
SystemPageSize uint //
MiLastPgNo uint64 //
AutoSyncThreshold uint //
UnsyncedBytes uint // how many bytes have been committed but not flushed yet to disk
SinceSync time.Duration //
AutosyncPeriod time.Duration //
SinceReaderCheck time.Duration //
Flags uint //
}
EnvInfo contains information an environment.
See MDBX_envinfo.
func PreOpenSnapInfo ¶ added in v0.39.0
type EnvInfoGeo ¶
type Errno ¶
Errno is an error type that represents the (unique) errno values defined by LMDB. Other errno values (such as EINVAL) are represented with type syscall.Errno. On Windows, LMDB return codes are translated into portable syscall.Errno constants (e.g. syscall.EINVAL, syscall.EACCES, etc.).
Most often helper functions such as IsNotFound may be used instead of dealing with Errno values directly.
lmdb.IsNotFound(err) lmdb.IsErrno(err, lmdb.TxnFull) lmdb.IsErrnoSys(err, syscall.EINVAL) lmdb.IsErrnoFn(err, os.IsPermission)
const ( KeyExist Errno = C.MDBX_KEYEXIST NotFound Errno = C.MDBX_NOTFOUND PageNotFound Errno = C.MDBX_PAGE_NOTFOUND Corrupted Errno = C.MDBX_CORRUPTED Panic Errno = C.MDBX_PANIC VersionMismatch Errno = C.MDBX_VERSION_MISMATCH Invalid Errno = C.MDBX_INVALID MapFull Errno = C.MDBX_MAP_FULL DBsFull Errno = C.MDBX_DBS_FULL ReadersFull Errno = C.MDBX_READERS_FULL TxnFull Errno = C.MDBX_TXN_FULL CursorFull Errno = C.MDBX_CURSOR_FULL PageFull Errno = C.MDBX_PAGE_FULL Incompatible Errno = C.MDBX_INCOMPATIBLE BadRSlot Errno = C.MDBX_BAD_RSLOT BadTxn Errno = C.MDBX_BAD_TXN BadValSize Errno = C.MDBX_BAD_VALSIZE BadDBI Errno = C.MDBX_BAD_DBI Perm Errno = C.MDBX_EPERM // Ousted reports that a parked reader was ousted by a writer to // recycle old MVCC snapshots (returned e.g. by Txn.Unpark with // restartIfOusted=false, or by reads in a parked-and-ousted txn). Ousted Errno = C.MDBX_OUSTED // LaggardReader means defrag stopped early. Upstream's name misleads: it // also fires with no reader, when defrag stalls on one page and the GC is // not empty. LaggardReader Errno = C.MDBX_LAGGARD_READER )
The most common error codes do not need to be handled explicitly. Errors can be checked through helper functions IsNotFound, IsMapFull, etc, Otherwise they should be checked using the IsErrno function instead of direct comparison because they will typically be wrapped with an OpError.
type GCInfo ¶ added in v0.39.18
type GCInfo struct {
PagesAllocated uint64
PagesBacked uint64
PagesTotal uint64
PagesGC uint64
PagesReclaimable uint64
PagesRetained uint64
MaxReaderLag uint64
MaxRetainedPages uint64
}
GCInfo describes MDBX garbage collection and page usage for a transaction.
type Label ¶ added in v0.39.0
type Label string
Label - will be added to error messages. For better understanding - which DB has problem.
const Default Label = "default"
type LogLvl ¶
type LogLvl = C.MDBX_log_level_t
type Multi ¶
type Multi struct {
// contains filtered or unexported fields
}
Multi is a wrapper for a contiguous page of sorted, fixed-length values passed to Cursor.PutMulti or retrieved using Cursor.Get with the GetMultiple/NextMultiple flag.
Multi values are only useful in databases opened with DupSort|DupFixed.
func WrapMulti ¶
WrapMulti converts a page of contiguous values with stride size into a Multi. WrapMulti panics if len(page) is not a multiple of stride.
_, val, _ := cursor.Get(nil, nil, lmdb.FirstDup) _, page, _ := cursor.Get(nil, nil, lmdb.GetMultiple) multi := lmdb.WrapMulti(page, len(val))
See mdb_cursor_get and MDB_GET_MULTIPLE.
func (*Multi) Size ¶
Size returns the total size of the Multi data and is equal to
m.Len()*m.Stride()
type OpError ¶
OpError is an error returned by the C API. Not all errors returned by lmdb-go have type OpError but typically they do. The Errno field will either have type Errno or syscall.Errno.
type ReaderInfo ¶ added in v0.39.18
type ReaderInfo struct {
Num int
Slot int
PID int
TID uint64
TxID uint64
Lag uint64
BytesUsed uint64
BytesRetained uint64
Parked bool
Ousted bool
}
ReaderInfo describes one entry in the MDBX reader lock table.
type ReaderStats ¶ added in v0.39.18
type ReaderStats struct {
Count uint64
OldestTxID uint64
MaxLag uint64
MaxBytesUsed uint64
MaxBytesRetained uint64
SumBytesRetained uint64
Parked uint64
Ousted uint64
}
ReaderStats summarizes MDBX reader lock table state for metrics and monitoring. Collecting it may require scanning the reader lock table.
type Stat ¶
type Stat struct {
PSize uint // Size of a database page. This is currently the same for all databases.
Depth uint // Depth (height) of the B-tree
BranchPages uint64 // Number of internal (non-leaf) pages
LeafPages uint64 // Number of leaf pages
OverflowPages uint64 // Number of overflow pages
Entries uint64 // Number of data items
LastTxId uint64 // Transaction ID of committed last modification
}
Stat contains database status information.
See MDBX_stat.
type TxInfo ¶
type TxInfo struct {
Id uint64 // The ID of the transaction. For a READ-ONLY transaction, this corresponds to the snapshot being read
/** For READ-ONLY transaction: the lag from a recent MVCC-snapshot, i.e. the
number of committed transaction since read transaction started. For WRITE
transaction (provided if `scan_rlt=true`): the lag of the oldest reader
from current transaction (i.e. at least 1 if any reader running). */
ReadLag uint64
/** Used space by this transaction, i.e. corresponding to the last used
* database page. */
SpaceUsed uint64
/** Current size of database file. */
SpaceLimitSoft uint64
/** Upper bound for size the database file, i.e. the value `size_upper`
argument of the appropriate call of \ref mdbx_env_set_geometry(). */
SpaceLimitHard uint64
/** For READ-ONLY transaction: The total size of the database pages that were
retired by committed write transactions after the reader's MVCC-snapshot,
i.e. the space which would be freed after the Reader releases the
MVCC-snapshot for reuse by completion read transaction.
For WRITE transaction: The summarized size of the database pages that were
retired for now due Copy-On-Write during this transaction. */
SpaceRetired uint64
/** For READ-ONLY transaction: the space available for writer(s) and that
must be exhausted for reason to call the Handle-Slow-Readers callback for
this read transaction. For WRITE transaction: the space inside transaction
that left to `MDBX_TXN_FULL` error. */
SpaceLeftover uint64
/** For READ-ONLY transaction (provided if `scan_rlt=true`): The space that
actually become available for reuse when only this transaction will be
finished.
For WRITE transaction: The summarized size of the dirty database
pages that generated during this transaction. */
SpaceDirty uint64
Spill uint64
Unspill uint64
}
type Txn ¶
type Txn struct {
// Pooled may be set to true while a Txn is stored in a sync.Pool. Kept
// for lmdb-go/mdbxpool compatibility; this package has no Txn finalizer,
// so the flag has no effect.
Pooled bool
// contains filtered or unexported fields
}
Txn is a database transaction in an environment.
WARNING: A writable Txn is not threadsafe and may only be used in the goroutine that created it.
See MDBX_txn.
func (*Txn) Abort ¶
func (txn *Txn) Abort()
Abort discards pending writes in the transaction. A Txn cannot be used again after Abort is called.
See mdbx_txn_abort.
func (*Txn) Amend ¶ added in v0.40.2
Amend promotes a read-only transaction into a write transaction that modifies data relative to the read transaction's MVCC-snapshot. This only succeeds if no other write transaction has committed since the read transaction started; otherwise snapshotTooOld is true and no actions have been performed.
If TxPrepareRO is included in flags, the original read transaction handle is preserved (in reset state) and may be reused via Renew. Otherwise the original read transaction handle is consumed and must not be used again.
On success a new *Txn representing the write transaction is returned.
WARNING: Cursors opened against the original read transaction are not transferred to the new write transaction. Close them before calling Amend (or, with TxPrepareRO, after the call but before Renew).
See mdbx_txn_amend.
func (*Txn) Checkpoint ¶ added in v0.40.2
func (txn *Txn) Checkpoint(weakeningDurability uint) (lat CommitLatency, noChanges bool, err error)
Checkpoint commits the operations of the write transaction and immediately starts a new write transaction reusing the same handle, without releasing any locks. This is useful for breaking up long write pipelines into smaller committed chunks while preventing other writers from interleaving.
weakeningDurability may be used to relax sync/durability for the committed changes (e.g. TxNoSync or TxNoMetaSync). Pass 0 for default durability.
noChanges is true when the transaction contained no changes (MDBX_RESULT_TRUE); in that case the txn handle is left untouched and remains usable.
WARNING: A successful Checkpoint internally tears down and restarts the underlying libmdbx transaction. All cursors opened against this Txn become unusable after Checkpoint returns — libmdbx does not currently support cursor transfer across the restart. Close them (before or after the call; Close stays safe and required) to free their allocations.
On a non-RESULT_TRUE error the libmdbx commit path terminates the handle internally; the wrapper clears it so a deferred Abort/Commit is a no-op.
See mdbx_txn_checkpoint.
func (*Txn) Clone ¶ added in v0.40.2
Clone creates a new read-only transaction that observes the same MVCC-snapshot as the origin transaction. Cloning a write transaction is also allowed and yields a read-only view of the pre-write snapshot (without uncommitted changes).
The origin transaction must not be used concurrently while Clone runs. It is the caller's responsibility to keep the origin alive on its own thread until this function returns.
See mdbx_txn_clone.
func (*Txn) CloneInto ¶ added in v0.40.2
CloneInto re-uses an existing (reset) read-only transaction handle as the destination for a clone of the origin transaction. The target *Txn must be a reset read-only transaction obtained from the same Env. On success the target is renewed against the origin's snapshot.
Passing a write (non-readonly) target is rejected here rather than at the libmdbx layer: the C bailout for a write target would invoke txn_ro_reset() on the write handle, corrupting its state. See mdbx_txn_clone in mdbx.c.
See mdbx_txn_clone.
func (*Txn) Cmp ¶
Cmp - this func follow bytes.Compare return style: The result will be 0 if a==b, -1 if a < b, and +1 if a > b.
func (*Txn) Commit ¶
func (txn *Txn) Commit() (CommitLatency, error)
Commit persists all transaction operations to the database. A Txn cannot be used again after Commit is called.
See mdbx_txn_commit.
func (*Txn) CommitEmbarkRead ¶ added in v0.40.2
func (txn *Txn) CommitEmbarkRead() (lat CommitLatency, noChanges bool, err error)
CommitEmbarkRead commits the write transaction and atomically starts a new read-only transaction observing the just-committed snapshot, all without releasing the writer locks between the two operations. This guarantees that no other writer can interleave a commit between the write commit and the new reader.
On success the same Txn handle is reused and transitions to read-only mode.
noChanges is true when libmdbx returned MDBX_RESULT_TRUE (no dirty pages to commit). On that path libmdbx terminates the write handle without allocating a read replacement, so the Txn is no longer usable; the wrapper clears it. The noChanges path is only reachable in builds with MDBX_NOSUCCESS_PURE_COMMIT enabled — under the default build a no-op commit succeeds normally.
WARNING: All cursors opened against this Txn become unusable after CommitEmbarkRead returns. Close them (before or after the call) to free their allocations.
See mdbx_txn_commit_embark_read.
func (*Txn) CreateDBI ¶
CreateDBI is a shorthand for OpenDBISimple that passed the flag lmdb.Create.
func (*Txn) DCmp ¶
DCmp - this func follow bytes.Compare return style: The result will be 0 if a==b, -1 if a < b, and +1 if a > b.
func (*Txn) Del ¶
Del deletes an item from database dbi. Del ignores val unless dbi has the DupSort flag.
See mdbx_del.
func (*Txn) Drop ¶
Drop empties the database if del is false. Drop deletes and closes the database if del is true.
See mdbx_drop.
func (*Txn) EstimateRange ¶ added in v0.40.2
EstimateRange estimates, as a number of elements, the size of the key range that starts at beginKey and ends at endKey. The result is a rough estimate based on the b-tree structure, intended for building and optimizing query plans (e.g. splitting a key range into evenly-sized chunks for parallel workers); it is not an exact count.
A nil beginKey means the range starts at the first item of dbi; a nil endKey means it ends at the last item. beginData/endData are only meaningful for DupSort databases (pass nil otherwise) and may only be supplied together with the corresponding key.
See mdbx_estimate_range.
func (*Txn) GCInfo ¶ added in v0.39.18
GCInfo returns garbage collection and page usage information for txn.
func (*Txn) Get ¶
Get retrieves items from database dbi. The returned slice is a zero-copy view into the memory-mapped file: read-only and valid only until the next update operation in a write txn or until the transaction ends. Copy it if it must live longer.
See mdbx_get.
func (*Txn) ID ¶
ID returns the identifier for txn. A view transaction identifier corresponds to the Env snapshot being viewed and may be shared with other view transactions.
See mdbx_txn_id.
func (*Txn) Info ¶
scan_rlt The boolean flag controls the scan of the read lock
table to provide complete information. Such scan is relatively expensive and you can avoid it if corresponding fields are not needed. See description of \ref MDBX_txn_info.
func (*Txn) OpenCursor ¶
OpenCursor allocates and initializes a Cursor to database dbi.
See mdbx_cursor_open.
func (*Txn) OpenDBI
deprecated
OpenDBI opens a named database in the environment. An error is returned if name is empty. The DBI returned by OpenDBI can be used in other transactions but not before Txn has terminated.
OpenDBI can only be called after env.SetMaxDBs() has been called to set the maximum number of named databases.
The C API uses null terminated strings for database names. A consequence is that names cannot contain null bytes themselves. OpenDBI does not check for null bytes in the name argument.
See mdbx_dbi_open.
Deprecated: use OpenDBISimple instead
func (*Txn) OpenDBISimple ¶
OpenDBISimple opens a named database in the environment. An error is returned if name is empty. The DBI returned by OpenDBISimple can be used in other transactions but not before Txn has terminated.
OpenDBISimple can only be called after env.SetMaxDBs() has been called to set the maximum number of named databases.
The C API uses null terminated strings for database names. A consequence is that names cannot contain null bytes themselves. OpenDBISimple does not check for null bytes in the name argument.
See mdbx_dbi_open.
func (*Txn) OpenRoot ¶
OpenRoot opens the root database. OpenRoot behaves similarly to OpenDBISimple but does not require env.SetMaxDBs() to be called beforehand. And, OpenRoot can be called without flags in a View transaction.
func (*Txn) Park ¶ added in v0.39.3
Park puts a read-only transaction in the "parked" state so it does not block recycling of old MVCC snapshots. Data pointers obtained before parking must not be dereferenced until unparked. Write transactions cannot be parked; libmdbx reports MDBX_TXN_INVALID for them.
A parked reader may be ousted by a writer at any point, so while parked ID() queries libmdbx on every call instead of using its cached value. With autounpark, a read that finds the txn ousted fails with an Ousted error and leaves the handle reset — reusable via Renew, like the Unpark(false) outcome.
Park returns an error if the Env has already been closed (older versions were a silent no-op in that case).
See mdbx_txn_park.
func (*Txn) PutReserve ¶
PutReserve returns a []byte of length n that can be written to, potentially avoiding a memcopy. The returned byte slice is only valid in txn's thread, before it has terminated.
func (*Txn) Refresh ¶ added in v0.40.2
Refresh advances a read-only transaction to the most recent MVCC-snapshot without aborting and renewing the reader slot. It is cheaper than Reset+Renew because it keeps the reader registration alive.
The returned `atTip` bool is true when the transaction was already viewing the latest snapshot and no work was performed.
See mdbx_txn_refresh.
func (*Txn) ReleaseAllCursors ¶ added in v0.39.3
ReleaseAllCursors unbinds (unbind=true) or closes (unbind=false) all cursors of the transaction.
WARNING: with unbind=false libmdbx frees the C cursors; every Go Cursor bound to this Txn is left holding a dangling handle and must be discarded WITHOUT calling Close (Close would touch freed memory).
func (*Txn) Renew ¶
Renew reuses a transaction that was previously reset by calling txn.Reset(). Renew panics if txn is managed by Update, View, etc.
See mdbx_txn_renew.
func (*Txn) Reset ¶
Reset aborts the transaction clears internal state so the transaction may be reused by calling Renew. If txn is not going to be reused txn.Abort() must be called to release its slot in the lock table and free its memory. Reset panics if txn is managed by Update, View, etc.
Reset returns the error reported by mdbx_txn_reset, e.g. EINVAL on POSIX (check with IsErrnoSys(err, syscall.EINVAL)) when txn is not read-only.
See mdbx_txn_reset.
func (*Txn) Rollback ¶ added in v0.40.2
Rollback aborts all uncommitted changes in a write transaction and immediately restarts it on a fresh snapshot, keeping the writer locks held. Unlike Abort followed by BeginTxn, no other writer can grab the lock in between.
The Txn handle remains valid after a successful Rollback and may continue to be used. On error the wrapper does NOT clear the handle — libmdbx's pre-validation errors (BAD_TXN, EINVAL, THREAD_MISMATCH) leave the transaction intact, and the caller is expected to call Abort() to release it. Treating MDBX_RESULT_TRUE as a silent success is defensive: the current libmdbx code does not return it for rollback, but the public docs list it as a possible return.
WARNING: All cursors opened against this Txn become unusable after a successful Rollback (libmdbx deactivates them via txn_done_cursors before the restart). Close them (before or after the call) to free their allocations.
See mdbx_txn_rollback.
func (*Txn) RunOp ¶
RunOp executes fn with txn as an argument. During the execution of fn no goroutine may call the Commit, Abort, Reset, and Renew methods on txn. RunOp returns the result of fn without any further action. RunOp will not abort txn if fn returns an error, unless terminate is true. If terminate is true then RunOp will attempt to commit txn if fn is successful, otherwise RunOp will abort txn before returning any failure encountered.
RunOp primarily exists to allow applications and other packages to provide variants of the managed transactions provided by lmdb (i.e. View, Update, etc). For example, the lmdbpool package uses RunOp to provide an Txn-friendly sync.Pool and a function analogous to Env.View that uses transactions from that pool.
func (*Txn) Sub ¶
Sub executes fn in a subtransaction. Sub commits the subtransaction iff a nil error is returned by fn and otherwise aborts it. Sub returns any error it encounters.
Sub may only be called on an Update Txn (one created without the Readonly flag). Calling Sub on a View transaction will return an error. Sub assumes the calling goroutine is locked to an OS thread and will not call runtime.LockOSThread.
Any call to Abort, Commit, Renew, or Reset on a Txn created by Sub will panic.
func (*Txn) Unpark ¶ added in v0.39.3
Unpark restores a transaction parked with Park.
restarted is true when the txn was ousted and restarted on the most recent MVCC snapshot (requires restartIfOusted=true): earlier pointers are invalid and the cached ID is refreshed. If ousted with restartIfOusted=false, Unpark returns an Ousted error (IsErrno(err, Ousted)) and the handle stays reusable via Renew/Abort.
Unpark returns an error if the Env has already been closed (older versions were a silent no-op in that case).
See mdbx_txn_unpark.
type TxnOp ¶
TxnOp is an operation applied to a managed transaction. The Txn passed to a TxnOp is managed and the operation must not call Commit, Abort, Renew, or Reset on it.
IMPORTANT: TxnOps that write to the database (those passed to Env.Update or Txn.Sub) must not use the Txn in another goroutine (passing it directly or otherwise through closure). Doing so has undefined results.
Notes ¶
Bugs ¶
MDBX_INTEGERKEY and MDBX_INTEGERDUP aren't usable. I'm not sure they would be faster with the cgo bridge. They need to be tested and benchmarked.