Sitelet https://github.com/elastic/elasticsearch/pull/157589
Skip to content

ESQL:DS: Align declared text-binding contract docs - #157589

Open
bpintea wants to merge 4 commits into
elastic:mainfrom
bpintea:esql/ds/fix_java_docs
Open

ESQL:DS: Align declared text-binding contract docs#157589
bpintea wants to merge 4 commits into
elastic:mainfrom
bpintea:esql/ds/fix_java_docs

Conversation

@bpintea

@bpintea bpintea commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Three documentation surfaces described a positional binding contract for declared (strict) text schemas that the reader had not implemented since contract:

  • Rewrites DatasetFieldMapping's binding paragraph: declared text columns bind by header name, not positionally; headerless files self-bind via colN names; a missing declared name reads null with a warning.
  • Renames declaredPathBinding → declaredProvenanceBinding throughout (private field, SPI method withDeclaredPathBinding → withDeclaredProvenanceBinding, constructor param, all call sites and overrides) to eliminate the misleading suggestion that path presence — rather than SchemaProvenance.DECLARED — drives the flag.
  • Rewrites validateDeclaredHeaderBinding's javadoc to accurately describe its two branches: by-name dispatch for declared schemas; width tripwire for pinned inferred schemas.
  • Fixes the tripwire error message from "declared schema binds text columns in order" to "pinned schema has N columns … file may have drifted", since the tripwire fires only for pinned inferred schemas (declared schemas bypass it via the by-name path).
  • Updates the tripwire unit test comments and assertion strings to match.

Closes https://github.com/elastic/esql-planning/issues/1740

Three documentation surfaces described a positional binding contract for
declared (strict) text schemas that the reader had not implemented since
contract:

- Rewrites DatasetFieldMapping's binding paragraph: declared text columns bind
  by header name, not positionally; headerless files self-bind via colN names; a
  missing declared name reads null with a warning.
- Renames declaredPathBinding → declaredProvenanceBinding throughout (private
  field, SPI method withDeclaredPathBinding → withDeclaredProvenanceBinding,
  constructor param, all call sites and overrides) to eliminate the misleading
  suggestion that path presence — rather than SchemaProvenance.DECLARED — drives
  the flag.
- Rewrites validateDeclaredHeaderBinding's javadoc to accurately describe its
  two branches: by-name dispatch for declared schemas; width tripwire for pinned
  inferred schemas.
- Fixes the tripwire error message from "declared schema binds text columns in
  order" to "pinned schema has N columns … file may have drifted", since the
  tripwire fires only for pinned inferred schemas (declared schemas bypass it
  via the by-name path).
- Updates the tripwire unit test comments and assertion strings to match.
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Aug 25, 2026
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@quackaplop

Copy link
Copy Markdown
Member

Thesis checks out — flag's set from whether the user declared the schema, not from path. Five things:

1. The new DatasetFieldMapping line "reads null with a warning" is CSV/TSV only — warnAbsentDeclaredColumns is in CsvFormatReader and nowhere else, and sharedStrictDeclaredSchemaAbsentColumnReadsNull says the columnar readers emit none. NDJSON/Parquet/ORC read null silently. Same line's in the FormatReader javadoc you rewrote. One clause fixes both.

2. Old wording still around: ExternalSourceResolver says "text formats bind positionally" twice (multi-file guard comment, rejectDeclaredPartitionCollision javadoc), plus five in CsvFormatReader. Checked — the rejection rule doesn't need that premise, so it's just wording. Leave the inferred-path positional comments, those are right.

3. declaredPathFieldIndexes still path-named, three call sites. Four tests still testDeclaredPathBinding*.

4. FileSourceFactory: "EXTERNAL-only development knob" and "came from inline EXTERNAL" — reword while you're in there. Leave EXTERNAL_ONLY_KEYS, that's a real constant.

5. Error message says it twice ("has only 2" / "fewer columns than the pinned schema"), and still suggests header_row=false alongside drift. Pick one.

NDJSON's missing warning I'd file separately. Corner cases all check out, and sharedStrictDeclaredSchemaWithoutPathBindsByName already pins the contract, so no new test needed.

@quackaplop quackaplop left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on the five points in #157589 (comment) — mainly (1), since the new DatasetFieldMapping line claims a warning that only CSV/TSV emit.

@quackaplop quackaplop left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All five addressed, thanks. Nice catch on the "headered path-bound read" error string, I'd missed that one.

One loose end, not worth another round: in that paragraph "text" means CSV/TSV, so NDJSON now falls between the two clauses rather than being described either way. Went from wrong to unmentioned, which is fine by me — worth a word if you touch it again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL ES|QL|DS ES|QL datasources >non-issue Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.6.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants