Sitelet https://github.com/formancehq/numscript/pull/3
Skip to content

Better cli - #3

Merged
ascandone merged 4 commits into
mainfrom
better-cli
Jul 26, 2024
Merged

ascandone merged 4 commits into
mainfrom
better-cli

Conversation

@ascandone

Copy link
Copy Markdown
Contributor

No description provided.

@ascandone
ascandone merged commit f34927f into main Jul 26, 2024
@ascandone
ascandone deleted the better-cli branch July 26, 2024 07:22
ascandone added a commit that referenced this pull request Sep 18, 2026
…rements

The doc said the generator rarely produces a save-overdraw followed by a
bounded-overdraft draw on the same account. Measured over the same 3000 seeds
it produces the ingredients constantly and the combination never: 686 saves,
2284 bounded overdrafts, 130 on a shared account, 7 in order, 2 that also
overdraw, 0 complete.

The reason is structural, so say it: statements are sampled independently, and
a shape needing two of them to hit the same (account, asset) in order is
quadratically suppressed.
ascandone added a commit that referenced this pull request Sep 25, 2026
* test: differential testing against the legacy ledger machine

Runs generated programs through both the numscript interpreter and a vendored
copy of the ledger repo's `internal/machine`, and compares what moved.

- `internal/oracle/` is that vendored copy, in its own module so it cannot
  drift into the interpreter's dependency graph. `smoke_test.go` checks the
  vendoring itself did not change behaviour. `DIVERGENCES.md` records every
  difference from ledger main and why it is there; `DIFFERENCES-BY-EXAMPLE.md`
  is the same list as runnable scripts.
- `internal/difftest/` is the harness, also its own module: `Compare` (what
  counts as a real difference vs a posting-granularity one), the seeded
  `TestDifferentialSweep` gate, `TestKnownBugRepros`/`TestKnownOpenDivergences`,
  and `FuzzDiff` with its corpus.

CI gets a `Difftest` job, because both are separate modules and the root
`go test ./...` reaches neither. It runs the deterministic sweep, not the
fuzzer: `-fuzztime` expiry surfaces as a failure with no divergence behind it.

The sweep is green -- 0 divergence classes and 0 tolerated over 3000 seeds.
Getting there meant changing the oracle's `kept` to consume its funding, the
way the interpreter does and unlike ledger. That is a deliberate loss of
independence on `kept` and is written up as DIVERGENCES.md §2 ②.

* refactor(gen): trim comments and unexport the internal-only helpers

Drops the running commentary about the generator's Haskell ancestry
(Gen.hs/Numscript.hs/Utils.hs, QuickCheck's `frequency`, `Ratio Int`) — none of
it is a fact about this code — along with comments that restate the field they
sit on and rationale that had grown into prose.

Kept: what cannot be read off the code — why `max`-clause amounts are never
negative, why both pools are sorted before being indexed, why the var-collision
shapes are biased for.

`GenerateScriptAST`, `ToBuilder` and `ToBuilderScript` were exported but used
only inside the package; `GenerateScript` and `RandFromBytes` are the entry
points. Unexported.

Also removes references to DIFFTEST_HANDOFF.md, which is not in the repo, and
deletes a superseded duplicate doc comment on genPresetMetadata.

internal/gen landed in #194 before this cleanup was ready, hence the separate
commit.

* docs(oracle): merge the two divergence docs into one

DIVERGENCES.md and DIFFERENCES-BY-EXAMPLE.md covered the same four differences
at two levels of detail, and the second was written on top of the first rather
than replacing it, so both had to be kept in sync by hand.

One file now. 472 lines down to 244. Each difference gets a runnable script,
what all three engines do with it, why, and the open question. The vendoring
mechanics and the drift check move to the end, where they belong.

Section numbers become #1-#4, which is what the code comments now cite.

* oracle: drop the OP_TAKE guard, tolerate the difference in Compare instead

The oracle carried a negative-amount guard on OP_TAKE that ledger does not
have. That made vm/machine.go a behaviour fork, and hid the cost: the oracle
silently stopped being ledger on every script with a negative send amount.

Copy ledger exactly instead, and absorb the difference in Compare under a named
tolerance. Both engines reject those scripts and no money moves either way;
only the error differs (`insufficient funds` vs `Cannot send negative amount`).

The tolerance is one-directional: the interpreter naming a negative amount
while the oracle blames the funds. The reverse is DIVERGENCES.md #4, where the
oracle rejects and the interpreter silently zeroes the clause and runs short,
and that must stay a mismatch. TestMissingFundsClassificationMismatchStillCaught
caught the symmetric version of this and is why it is not.

The sweep prints how often it fires -- 161 of 3000 scripts -- so the cost is
countable on every run instead of invisible. vm/machine.go is now
byte-identical to ledger apart from the vendored types, leaving `kept` as the
oracle's only behavioural divergence.

Also records in DIVERGENCES.md #4 why ledger PR #2079 was closed: clamping a
negative `max` needs a new opcode, and adding one to the machine is too
aggressive for what it buys.

* docs(oracle): replace the "rarely produces" claim about #3 with measurements

The doc said the generator rarely produces a save-overdraw followed by a
bounded-overdraft draw on the same account. Measured over the same 3000 seeds
it produces the ingredients constantly and the combination never: 686 saves,
2284 bounded overdrafts, 130 on a shared account, 7 in order, 2 that also
overdraw, 0 complete.

The reason is structural, so say it: statements are sampled independently, and
a shape needing two of them to hit the same (account, asset) in order is
quadratically suppressed.

* improve generator

* fix review issues

* apply bot suggestions

* difftest: narrow the one-sided-runtime-failure tolerance to the known negative-max gap

Compare tolerated any one-sided runtime failure once both sides agreed on
missing-funds classification, not just the known negative `max` clause
divergence (DIVERGENCES.md #4). A real bug where one engine committed money
the other refused to move, for an unrelated reason, would have passed
through silently.

Add SideResult.NegativeMaxReject, set by run_oracle.go when the failure is
ledger's OP_TAKE_MAX guard rejecting a negative max clause (untyped upstream,
so matched by its fixed error text rather than by type), and require it
before tolerating the asymmetry. Any other one-sided failure is now a
mismatch. Renamed the tolerance from "one-sided runtime failure" to
"negative max clause" to match, and updated DIVERGENCES.md and the tests that
asserted the old name.

* made the err more precise
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant