Sitelet https://github.com/apache/paimon/pull/10363
Skip to content

[python] Preserve full precision in decimal global index keys - #10363

Open
TheR1sing3un wants to merge 1 commit into
apache:masterfrom
TheR1sing3un:fix/python-decimal-index-precision
Open

TheR1sing3un wants to merge 1 commit into
apache:masterfrom
TheR1sing3un:fix/python-decimal-index-precision

Conversation

@TheR1sing3un

Copy link
Copy Markdown
Member

Purpose

DECIMAL global-index keys currently depend on Python's ambient decimal context. A valid DECIMAL(38,2) value such as 123456789012345678901234567890123456.78 fails to serialize at the default precision of 28, and decoding its existing unscaled bytes silently rounds away digits. This can break BTree/bitmap equality and range lookups.

Use a dedicated context with sufficient precision for scale rounding and reconstruct decoded decimals exactly from their coefficient and scale. Preserve HALF_UP rounding and the existing compact/non-compact byte formats without changing the caller's context.

Tests

  • New regression coverage for compact and 38-digit decimals, negative values, scale 38, zero, exact decoding, rounding with caller traps enabled, and persisted BTree/bitmap equality and range queries.
  • All 11 new tests pass; 10 fail against the unchanged base.
  • Existing global_index_build_test.py and global_index_scalar_search_mode_test.py pass (44 tests).
  • Flake8 for changed files and git diff --check pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant