Sitelet https://github.com/formancehq/numscript/pull/194
Skip to content

feat(gen): seed-driven numscript program generator - #194

Merged
ascandone merged 3 commits into
mainfrom
feat/program-generator
Sep 18, 2026
Merged

ascandone merged 3 commits into
mainfrom
feat/program-generator

Conversation

@ascandone

@ascandone ascandone commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

What changed

internal/gen turns a seed into a numscript program plus the inputs it needs
(vars and starting balances). Programs are built as an AST and rendered through
builder.

  • Generation is a pure function of the seed. The same seed always yields
    the same program, vars and balances, so a failure reported as a seed number
    is reproducible from that number alone (determinism_test.go).
  • Every value type can appear as an inline literal or as a var, and metadata is
    read back as each of its types — not only number.
  • cleanup.go drops shapes that are not valid numscript at construction time,
    rather than emitting them and filtering downstream.

Why

Second of three PRs replacing #191. Stacked on #193 (builder/), which it uses
to render. On its own it is a generator with no consumer; #195 adds the
harness that runs what it produces through two engines.

Base is feat/numscript-builder — retargets to main automatically when
#193 merges.

Risk

LOW — new package under internal/, no callers outside tests.

Validation

  • Baseline validation: go build ./..., go vet ./..., gofmt clean
  • Targeted tests: gen_test.go, determinism_test.go, value_forms_test.go, meta_types_test.go
  • Full suite / broader validation: go test -count=1 ./... green

Architecture / behavior impact

N/A.

Review focus

Determinism is the property everything downstream relies on — if a seed stops
being reproducible, every sweep failure becomes un-actionable. determinism_test.go
is where to push.

Second: whether cleanup.go rejects shapes that are actually valid. Anything it
drops is coverage the differential sweep never gets.

Known concerns

The generator's reach is a known limit rather than a defect: it does not produce
asset scaling, account interpolation or colors, so a green sweep says nothing
about those. Recorded where it matters in #195.

Extends the script builder so it can express the shapes the differential
generator needs to emit:

- nested origins (`origin.go`), rendered in dependency order and declared as
  vars when interpolated
- allotment sources and destinations (`allotment.go`)
- `send*` / uncapped-amount statements and the remaining statement forms
  (`statement_extra.go`)

The expression layer moves from a bare string to a typed representation so a
monetary, an account and a number are no longer interchangeable at the call
site.
`internal/gen` turns a seed into a numscript program plus the inputs it needs
(vars and starting balances), for property and differential testing. Programs
are built as an AST and rendered through `builder`.

Generation is a pure function of the seed: the same seed always yields the same
program, vars and balances, so a failure reported as a seed number is
reproducible from the number alone (`determinism_test.go`).

Every value type can appear as an inline literal or as a var, metadata is read
back as each of its types, and `cleanup.go` drops the shapes that are not valid
numscript rather than emitting them and filtering later.
Azorlogh
Azorlogh previously approved these changes Sep 18, 2026
Base automatically changed from feat/numscript-builder to main September 18, 2026 09:43
@ascandone
ascandone dismissed Azorlogh’s stale review September 18, 2026 09:43

The base branch was changed.

@ascandone
ascandone enabled auto-merge (squash) September 18, 2026 09:43
@ascandone
ascandone merged commit 605460a into main Sep 18, 2026
4 checks passed
@ascandone
ascandone deleted the feat/program-generator branch September 18, 2026 09:49
ascandone added a commit that referenced this pull request Sep 18, 2026
Drops the running commentary about the generator's Haskell ancestry
(Gen.hs/Numscript.hs/Utils.hs, QuickCheck's `frequency`, `Ratio Int`) — none of
it is a fact about this code — along with comments that restate the field they
sit on and rationale that had grown into prose.

Kept: what cannot be read off the code — why `max`-clause amounts are never
negative, why both pools are sorted before being indexed, why the var-collision
shapes are biased for.

`GenerateScriptAST`, `ToBuilder` and `ToBuilderScript` were exported but used
only inside the package; `GenerateScript` and `RandFromBytes` are the entry
points. Unexported.

Also removes references to DIFFTEST_HANDOFF.md, which is not in the repo, and
deletes a superseded duplicate doc comment on genPresetMetadata.

internal/gen landed in #194 before this cleanup was ready, hence the separate
commit.
ascandone added a commit that referenced this pull request Sep 25, 2026
* test: differential testing against the legacy ledger machine

Runs generated programs through both the numscript interpreter and a vendored
copy of the ledger repo's `internal/machine`, and compares what moved.

- `internal/oracle/` is that vendored copy, in its own module so it cannot
  drift into the interpreter's dependency graph. `smoke_test.go` checks the
  vendoring itself did not change behaviour. `DIVERGENCES.md` records every
  difference from ledger main and why it is there; `DIFFERENCES-BY-EXAMPLE.md`
  is the same list as runnable scripts.
- `internal/difftest/` is the harness, also its own module: `Compare` (what
  counts as a real difference vs a posting-granularity one), the seeded
  `TestDifferentialSweep` gate, `TestKnownBugRepros`/`TestKnownOpenDivergences`,
  and `FuzzDiff` with its corpus.

CI gets a `Difftest` job, because both are separate modules and the root
`go test ./...` reaches neither. It runs the deterministic sweep, not the
fuzzer: `-fuzztime` expiry surfaces as a failure with no divergence behind it.

The sweep is green -- 0 divergence classes and 0 tolerated over 3000 seeds.
Getting there meant changing the oracle's `kept` to consume its funding, the
way the interpreter does and unlike ledger. That is a deliberate loss of
independence on `kept` and is written up as DIVERGENCES.md §2 ②.

* refactor(gen): trim comments and unexport the internal-only helpers

Drops the running commentary about the generator's Haskell ancestry
(Gen.hs/Numscript.hs/Utils.hs, QuickCheck's `frequency`, `Ratio Int`) — none of
it is a fact about this code — along with comments that restate the field they
sit on and rationale that had grown into prose.

Kept: what cannot be read off the code — why `max`-clause amounts are never
negative, why both pools are sorted before being indexed, why the var-collision
shapes are biased for.

`GenerateScriptAST`, `ToBuilder` and `ToBuilderScript` were exported but used
only inside the package; `GenerateScript` and `RandFromBytes` are the entry
points. Unexported.

Also removes references to DIFFTEST_HANDOFF.md, which is not in the repo, and
deletes a superseded duplicate doc comment on genPresetMetadata.

internal/gen landed in #194 before this cleanup was ready, hence the separate
commit.

* docs(oracle): merge the two divergence docs into one

DIVERGENCES.md and DIFFERENCES-BY-EXAMPLE.md covered the same four differences
at two levels of detail, and the second was written on top of the first rather
than replacing it, so both had to be kept in sync by hand.

One file now. 472 lines down to 244. Each difference gets a runnable script,
what all three engines do with it, why, and the open question. The vendoring
mechanics and the drift check move to the end, where they belong.

Section numbers become #1-#4, which is what the code comments now cite.

* oracle: drop the OP_TAKE guard, tolerate the difference in Compare instead

The oracle carried a negative-amount guard on OP_TAKE that ledger does not
have. That made vm/machine.go a behaviour fork, and hid the cost: the oracle
silently stopped being ledger on every script with a negative send amount.

Copy ledger exactly instead, and absorb the difference in Compare under a named
tolerance. Both engines reject those scripts and no money moves either way;
only the error differs (`insufficient funds` vs `Cannot send negative amount`).

The tolerance is one-directional: the interpreter naming a negative amount
while the oracle blames the funds. The reverse is DIVERGENCES.md #4, where the
oracle rejects and the interpreter silently zeroes the clause and runs short,
and that must stay a mismatch. TestMissingFundsClassificationMismatchStillCaught
caught the symmetric version of this and is why it is not.

The sweep prints how often it fires -- 161 of 3000 scripts -- so the cost is
countable on every run instead of invisible. vm/machine.go is now
byte-identical to ledger apart from the vendored types, leaving `kept` as the
oracle's only behavioural divergence.

Also records in DIVERGENCES.md #4 why ledger PR #2079 was closed: clamping a
negative `max` needs a new opcode, and adding one to the machine is too
aggressive for what it buys.

* docs(oracle): replace the "rarely produces" claim about #3 with measurements

The doc said the generator rarely produces a save-overdraw followed by a
bounded-overdraft draw on the same account. Measured over the same 3000 seeds
it produces the ingredients constantly and the combination never: 686 saves,
2284 bounded overdrafts, 130 on a shared account, 7 in order, 2 that also
overdraw, 0 complete.

The reason is structural, so say it: statements are sampled independently, and
a shape needing two of them to hit the same (account, asset) in order is
quadratically suppressed.

* improve generator

* fix review issues

* apply bot suggestions

* difftest: narrow the one-sided-runtime-failure tolerance to the known negative-max gap

Compare tolerated any one-sided runtime failure once both sides agreed on
missing-funds classification, not just the known negative `max` clause
divergence (DIVERGENCES.md #4). A real bug where one engine committed money
the other refused to move, for an unrelated reason, would have passed
through silently.

Add SideResult.NegativeMaxReject, set by run_oracle.go when the failure is
ledger's OP_TAKE_MAX guard rejecting a negative max clause (untyped upstream,
so matched by its fixed error text rather than by type), and require it
before tolerating the asymmetry. Any other one-sided failure is now a
mismatch. Renamed the tolerance from "one-sided runtime failure" to
"negative max clause" to match, and updated DIVERGENCES.md and the tests that
asserted the old name.

* made the err more precise
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants