Repository navigation

feat: Alternative table provider as spark data source - #110

Draft

timsaucer wants to merge 21 commits into

apache:mainfrom

timsaucer:feat/plain-c-scan-ffi

Commits on Jun 12, 2026
- build: consolidate Rust crates into a Cargo workspace + extract native-common
  
  timsaucer
  and
  claude
  committed
  3cf0e81
- build: inherit crate version/edition from workspace; publish datafusion-jni-common
  
  timsaucer
  and
  claude
  committed
  fd4dd74
- build: drop forward-referencing RAT excludes from foundation split
  
  timsaucer
  and
  claude
  committed
  2e50d6e
- build: drop overbroad **/*.md RAT exclude
  
  timsaucer
  and
  claude
  committed
  9d2096c
- docs: add README for datafusion-jni-common
  
  timsaucer
  and
  claude
  committed
  2514f56
- fix: scope foundation docs/excludes to what 1/6 ships
  
  timsaucer
  and
  claude
  committed
  3ff4fda
- build: point release scripts at workspace rust-target/ dir
  
  timsaucer
  and
  claude
  committed
  bb023eb
- build: mark datafusion-jni-common publish = false
  
  timsaucer
  and
  claude
  committed
  9e9de7d
Commits on Jun 15, 2026
- feat: add plain-C scan ABI crate over Arrow C Data/Stream
  
  timsaucer
  and
  claude
  committed
  e87fd64
- feat: add scan_config / scan_request protobuf for the C scan ABI
  
  timsaucer
  and
  claude
  committed
  2598322
- build: compile scan protobufs into datafusion-scan-ffi
  
  timsaucer
  and
  claude
  committed
  cfaac8d
- feat: add datafusion.listing provider decoding ScanConfig
  
  timsaucer
  and
  claude
  committed
  4cded33
- feat: add JNI shim over the plain-C scan ABI
  
  timsaucer
  and
  claude
  committed
  f95191d
- test: end-to-end JVM scan through the JNI shim
  
  timsaucer
  and
  claude
  committed
  bc91ff9
- test: projection and filter pushdown from Java
  
  timsaucer
  and
  claude
  committed
  436e2b3
- feat: wire row limit through the scan ABI
  
  timsaucer
  and
  claude
  committed
  d5eefee
- feat: add Spark DataSourceV2 connector
  
  timsaucer
  and
  claude
  committed
  0b24d7c
- feat: columnar reader on Spark 4.0 with provided Arrow
  
  timsaucer
  and
  claude
  committed
  1a1d0ec
- feat: push Spark limit into the scan (SupportsPushDownLimit)
  
  timsaucer
  and
  claude
  committed
  573b438
- docs: add DataFusion-Spark DataSource design doc
  
  timsaucer
  and
  claude
  committed
  a378bb4
Commits on Jun 16, 2026
- docs: add ADBC reuse analysis to Spark design doc
  
  timsaucer
  and
  claude
  committed
  301fd13

Navigation Menu

Repository navigation

feat: Alternative table provider as spark data source - #110

feat: Alternative table provider as spark data source#110
timsaucer wants to merge 21 commits into
apache:mainfrom
timsaucer:feat/plain-c-scan-ffi

Commits on Jun 12, 2026

build: consolidate Rust crates into a Cargo workspace + extract native-common

build: inherit crate version/edition from workspace; publish datafusion-jni-common

build: drop forward-referencing RAT excludes from foundation split

build: drop overbroad **/*.md RAT exclude

docs: add README for datafusion-jni-common

fix: scope foundation docs/excludes to what 1/6 ships

build: point release scripts at workspace rust-target/ dir

build: mark datafusion-jni-common publish = false

Commits on Jun 15, 2026

feat: add plain-C scan ABI crate over Arrow C Data/Stream

feat: add scan_config / scan_request protobuf for the C scan ABI

build: compile scan protobufs into datafusion-scan-ffi

feat: add datafusion.listing provider decoding ScanConfig

feat: add JNI shim over the plain-C scan ABI

test: end-to-end JVM scan through the JNI shim

test: projection and filter pushdown from Java

feat: wire row limit through the scan ABI

feat: add Spark DataSourceV2 connector

feat: columnar reader on Spark 4.0 with provided Arrow

feat: push Spark limit into the scan (SupportsPushDownLimit)

docs: add DataFusion-Spark DataSource design doc

Commits on Jun 16, 2026

docs: add ADBC reuse analysis to Spark design doc