fix: UDF/ODFV source rehydrate (+ Postgres / online cache) by aniketpalu · Pull Request #6655 · feast-dev/feast

aniketpalu · 2026-07-29T09:35:05Z

Summary

Fixes three issues found while testing BatchFeatureView UDFs and OnDemandFeatureViews with Spark transformations (SparkApplication / SparkComputeEngine path) against a remote feature server.

All three blocked a clean UDF/ODFV materialize → online path. This PR is scoped to that story. Shared helper feast.transformation.udf_rehydrate is introduced so the same contract can later be wired into RayTransformation.from_proto (intentionally not changed here).

1. Spark BFV UDF — prefer source over dill (exit 139)

Problem: Materializing a BatchFeatureView with a Python UDF on Spark could segfault the driver (exit 139). Registry stores both dill (body) and source (body_text / udf_string); the Spark path executed the dill-restored callable, which is unsafe with DataFrame ops on our Spark stack.

Change: SparkTransformationNode resolves the UDF via shared resolve_udf: exec trusted udf_string when present; fall back to the existing callable / dill. SparkFeatureBuilder passes udf_string through.

Files: spark/nodes.py, spark/feature_builder.py, transformation/udf_rehydrate.py

2. ODFV / local transformations — source-first `from_proto`

Problem: ODFV serve (and feast-apply rehydrate from SQL registry) used dill.loads(body) only. That fails across Python minors (e.g. apply on Spark driver 3.10, serve on feature-server 3.12) and when body_text includes @on_demand_feature_view(...) (NameError / CrashLoop if source is exec'd naively).

Change: PandasTransformation.from_proto and PythonTransformation.from_proto use the same resolve_udf (strip leading decorators, then exec; dill fallback).

Out of scope (follow-up): RayTransformation.from_proto remains dill.loads for now.

Files: pandas_transformation.py, python_transformation.py, udf_rehydrate.py

3. Postgres empty `feature_name_columns` → SELECT *

Problem: BatchFeatureView python/pandas mode signals "all columns" with empty feature_name_columns. Postgres offline store still projected only join keys + timestamps, starving the UDF of input features.

Change: When feature_name_columns is empty, pull_latest / pull_all use a.* / alias.* (aligned with Ray offline-store semantics).

Files: postgres_offline_store/postgres.py + unit tests

4. Fresh online registry reads

Problem: After materialize completes, online serving could briefly (or stickily) gate on cached MATERIALIZING / stale FV metadata.

Change: Online request path uses allow_cache=False when resolving features / feature services (utils._get_online_request_context, feature server _get_features).

Files: utils.py, feature_server.py

Out of scope

RayTransformation.from_proto / Ray compute-engine dill paths (follow-up using udf_rehydrate)
Remote materialization async failure → GENERATED wipe (separate remote-mat PR)
Redis same-timestamp idempotency

Which issue(s) this PR fixes

Found during BYOS Spark UDF/ODFV E2E; no single upstream issue required. Related discussions: Spark UDF materialize reliability, ODFV cross-image Python skew.

Checks

I've made sure the tests are passing.
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests
Integration tests
Manual tests
Testing is not required for this change

Misc

codecov-commenter · 2026-07-29T09:38:47Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 64.59627% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.92%. Comparing base (f771ea4) to head (0c6e240).

Files with missing lines	Patch %	Lines
sdk/python/feast/transformation/udf_rehydrate.py	64.65%	27 Missing and 14 partials ⚠️
.../python/feast/infra/compute_engines/spark/nodes.py	33.33%	6 Missing ⚠️
...ast/infra/compute_engines/spark/feature_builder.py	0.00%	2 Missing ⚠️
..._stores/contrib/postgres_offline_store/postgres.py	84.61%	1 Missing and 1 partial ⚠️
...thon/feast/transformation/pandas_transformation.py	75.00%	1 Missing and 1 partial ⚠️
...thon/feast/transformation/python_transformation.py	75.00%	1 Missing and 1 partial ⚠️
.../python/feast/transformation/ray_transformation.py	60.00%	1 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6655      +/-   ##
==========================================
+ Coverage   46.82%   46.92%   +0.09%     
==========================================
  Files         415      416       +1     
  Lines       50406    50549     +143     
  Branches     7215     7252      +37     
==========================================
+ Hits        23601    23718     +117     
- Misses      25155    25157       +2     
- Partials     1650     1674      +24

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`48.25% <64.59%> (+0.10%)`	⬆️

Files with missing lines	Coverage Δ
...ast/infra/compute_engines/spark/feature_builder.py	`0.00% <0.00%> (ø)`
..._stores/contrib/postgres_offline_store/postgres.py	`46.27% <84.61%> (+1.04%)`	⬆️
...thon/feast/transformation/pandas_transformation.py	`71.18% <75.00%> (-1.55%)`	⬇️
...thon/feast/transformation/python_transformation.py	`87.93% <75.00%> (+0.89%)`	⬆️
.../python/feast/transformation/ray_transformation.py	`39.39% <60.00%> (+39.39%)`	⬆️
.../python/feast/infra/compute_engines/spark/nodes.py	`26.12% <33.33%> (+0.05%)`	⬆️
sdk/python/feast/transformation/udf_rehydrate.py	`64.65% <64.65%> (ø)`

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f771ea4...0c6e240. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

franciscojavierarceo

Two blocking concerns from the remote diff:

feature_server.py and _get_online_request_context() now pass allow_cache=False for feature-service and feature-view resolution on every online request. For SQL-backed registries this moves backing-registry I/O into the serving hot path, which can materially regress latency and registry load. Please refresh/invalidate the affected cached definitions when UDF state changes (or isolate and justify this serving behavior separately) instead of globally bypassing the cache.
CodeQL's open finding in transformation/udf_rehydrate.py is valid: the decorator-stripping regex has nested unbounded repetition and processes registry-provided source text. Please replace it with a linear parser/line-based approach and add an adversarial regression case.

ntkathole · 2026-07-30T05:14:13Z

@aniketpalu Please fix CI

Post-materialize MATERIALIZING gate lag is handled by client retry / registry TTL refresh, not by bypassing the registry cache on every online request. Addresses review feedback on feast-dev#6655. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

…he) into merged-spark-e2e Resolved .secrets.baseline conflict by taking PR tip. Co-authored-by: Cursor <cursoragent@cursor.com>

Post-materialize MATERIALIZING gate lag is handled by client retry / registry TTL refresh, not by bypassing the registry cache on every online request. Addresses review feedback on feast-dev#6655. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

…istry reads Rehydrate BatchFeatureView/ODFV callables from body_text (strip leading decorators) before dill.loads to avoid Spark driver exit 139 and cross-Python serve failures. Treat empty Postgres feature_name_columns as SELECT *. Disable registry cache on the online request path so FeatureView state gates see AVAILABLE_ONLINE immediately after materialize. RayTransformation.from_proto left unchanged (follow-up). Signed-off-by: Aniket Paluskar <apaluska@redhat.com>

Signed-off-by: Aniket Paluskar <apaluska@redhat.com>

Post-materialize MATERIALIZING gate lag is handled by client retry / registry TTL refresh, not by bypassing the registry cache on every online request. Addresses review feedback on feast-dev#6655. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Avoid CodeQL ReDoS finding on nested @/newline regex when rehydrating body_text. Add multiline and adversarial @ spam unit tests. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

ntkathole · 2026-08-10T13:39:53Z

@aniketpalu please fix CI

…ache

Source-first from_proto rebuilds callables whose bytecode differs from the live repo function; requiring co_code equality made no-op feast apply rewrite ODFVs and broke universal CLI integration tests. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

…ache

Signed-off-by: Aniket Paluskar <apaluska@redhat.com>

RayTransformation now compares by udf_string when present (same contract as Pandas/Python) so a future source-first from_proto will not break no-op apply. SparkTransformationNode caches resolve_udf so strip+exec runs once per node lifetime. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

…ache

# [0.66.0](v0.65.0...v0.66.0) (2026-08-21) ### Bug Fixes * Add connection pre-warming for DynamoDB async client ([89240fa](89240fa)), closes [#6060](#6060) * Add remote registry client extra ([#6697](#6697)) ([b8dfcb0](b8dfcb0)) * Address review feedback on FIPS cipher suite configuration ([4a35fba](4a35fba)) * Allow remote-registry first apply for new projects ([39d408d](39d408d)) * Avoid importing feast.feature_store at mcp_server import time ([ddb2e9a](ddb2e9a)) * Bump pymssql to >=2.3.6 for macOS arm64 wheel support ([181eb35](181eb35)), closes [#5636](#5636) [#5193](#5193) [#5636](#5636) * Call ApplySavedDataset RPC instead of ApplyFeatureService in RemoteRegistry.apply_saved_dataset() ([934d341](934d341)) * Catch missing dbt parser dependency in dbt CLI commands ([#6534](#6534)) ([3c2ae3c](3c2ae3c)) * Default authentication to kubernetes auth ([6a4690a](6a4690a)) * Defer feature-freshness thread to post-fork to avoid Gunicorn deadlock ([#6648](#6648)) ([104ad10](104ad10)), closes [#6647](#6647) * Do not pass undeclared feature view columns to ODFV UDFs ([#6527](#6527)) ([75b9463](75b9463)) * downgrade mcp pin to 1.29.0 and fix CI lockfiles and unit tests ([98e5bca](98e5bca)), closes [#6706](#6706) * Feast apply silently ignoring ttl updates to None or timedelta(0) ([#6709](#6709)) ([97b0f25](97b0f25)), closes [#6703](#6703) * Fix mypy TorchTensor type alias error ([#6712](#6712)) ([34de6fa](34de6fa)), closes [#5563](#5563) * Fixed data source creation form gaps ([5d0f7d6](5d0f7d6)) * Handle parameterized and complex Trino types in type map ([326554d](326554d)) * Isolate default user permissions ([e37adbf](e37adbf)) * Isolate projection join key maps ([d1c709d](d1c709d)) * Map Postgres real to FLOAT instead of DOUBLE ([62db435](62db435)) * Merge shared ODFV source projections in feature resolution ([d269946](d269946)), closes [#6621](#6621) * More exhaustive athena types ([a9aaefc](a9aaefc)) * Normalize SQL registry read_path to the psycopg3 driver like path ([#6644](#6644)) ([996c6ea](996c6ea)), closes [#6643](#6643) * **operator:** add spec.services.onlineStore.disabled to opt out of the online store ([d81d4e3](d81d4e3)), closes [#6586](#6586) * Preinstall DuckDB delta extension for tests ([fd4d49d](fd4d49d)), closes [#6743](#6743) * Preserve event-time ordering within Redis online_write_batch ([40fb788](40fb788)), closes [#5163](#5163) * Prevent mutation of cached feature resolution results ([ea17419](ea17419)) * Remote feastRef FeatureStore fails first apply for a new feastProject ([9affee5](9affee5)) * Remove inert subjectaccessreviews and reorganize RBAC rules ([f771ea4](f771ea4)) * Report single-feature-view spark_application materialization success ([a9219d9](a9219d9)), closes [#6673](#6673) * Reset the global security manager after the permissions fixture ([7667215](7667215)) * Resolve kserve with pip --dry-run instead of installing it ([01da132](01da132)), closes [#6732](#6732) * Resolve write_to_offline_store feature view with a single registry lookup ([a42dc85](a42dc85)), closes [#4235](#4235) * Return False from __eq__ on cross-type comparison ([#6637](#6637)) ([0f149a9](0f149a9)), closes [#6636](#6636) * Reuse IdP-issued client tokens until near expiry ([602d752](602d752)) * Reuse the OIDC JWKS client across requests ([#6683](#6683)) ([a1e6fc2](a1e6fc2)) * Separate CronJob and feature-server ServiceAccounts ([398f643](398f643)) * Serialize UnixTimestamp proto values as raw int64 in remote online store transport ([1e7134f](1e7134f)) * Set FIPS cipher suites before pyarrow.flight import to prevent crash on IBM Power ([979b82a](979b82a)) * Support Entra ID (Azure AD) token claims in OIDC auth ([#6631](#6631)) ([f843c63](f843c63)) * UDF/ODFV source rehydrate (+ Postgres / online cache) ([#6655](#6655)) ([5fd7af7](5fd7af7)) * Updated projects-list.json in order to display newly added projects ([#6657](#6657)) ([3a6a103](3a6a103)) * Use correct image name in multi-arch imagetools push step ([faf85e0](faf85e0)) * Use join keys instead of entity names in ODFV materialization ([#6645](#6645)) ([abffebc](abffebc)), closes [#5965](#5965) * use matching proto class per feature view list in SqliteOnlineStore.plan() ([adb8c1c](adb8c1c)), closes [#6658](#6658) * Widen Athena integer type mapping for unsigned ints ([3425783](3425783)) ### Features * Add ConnectionRef to DataSource for pluggable external credential resolution ([28bde01](28bde01)) * Add Feature Service Create in UI ([0399380](0399380)) * Add hybrid to ValidOfflineStoreDBStorePersistenceTypes for HybridOfflineStore support ([#6707](#6707)) ([310ab51](310ab51)), closes [#6701](#6701) * Add MLflow integration support to Feast operator ([#6611](#6611)) ([52999f1](52999f1)) * Add opt-in filter_by_created_timestamp cutoff to get_historical_features ([#6617](#6617)) ([79b33ce](79b33ce)), closes [#6615](#6615) * Add optional OIDC token audience and issuer verification ([#6670](#6670)) ([ef307c6](ef307c6)) * Add packaged feature repository support to Feast Operator ([8112b1e](8112b1e)), closes [#6598](#6598) * add plan() support to DynamoDBOnlineStore ([51ce982](51ce982)), closes [#6658](#6658) [#6659](#6659) * Added optional namespace/colleciton to datasets ([165fcf2](165fcf2)) * Added SQL registry schema_mode and registry create command ([#6704](#6704)) ([037c4cd](037c4cd)) * Allow users to have protected project on shared registry ([f9923bc](f9923bc)) * Apply Intermediate TLS defaults on API fallback and handle transient errors ([#6587](#6587)) ([43ae993](43ae993)) * **cli:** Updated feast init demo by adding rag template ([#5946](#5946)) ([c8628eb](c8628eb)), closes [#5264](#5264) * Expose the OIDC JWKS tunables through the operator ([#6690](#6690)) ([fef4e78](fef4e78)), closes [#6683](#6683) * Making feast vector store with open ai search api compatible ([#6121](#6121)) ([54da19a](54da19a)) * Multi-arch publish for feast operator image ([b221036](b221036)) * OpenLineage lineage enhancements - full object coverage, richer UI, and API-level sync ([#6719](#6719)) ([120a868](120a868)) * **operator:** Add spec.services.initImage for init container image override ([#6598](#6598)) ([ca355cb](ca355cb)) * Pass optional OIDC audience and issuer through the operator ([#6677](#6677)) ([a13ed7b](a13ed7b)), closes [#6670](#6670) * **server:** Remote Materialization ([#6649](#6649)) ([b7ae488](b7ae488)), closes [#4526](#4526) * Support Lineage configs via operator ([bf1e54a](bf1e54a)) * Updated datasets UI to support grouping ([7ae64ec](7ae64ec))

…#6655) * fix: prefer UDF source over dill; Postgres SELECT *; fresh online registry reads Rehydrate BatchFeatureView/ODFV callables from body_text (strip leading decorators) before dill.loads to avoid Spark driver exit 139 and cross-Python serve failures. Treat empty Postgres feature_name_columns as SELECT *. Disable registry cache on the online request path so FeatureView state gates see AVAILABLE_ONLINE immediately after materialize. RayTransformation.from_proto left unchanged (follow-up). Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * Lint Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * fix: drop global allow_cache=False on online path Post-materialize MATERIALIZING gate lag is handled by client retry / registry TTL refresh, not by bypassing the registry cache on every online request. Addresses review feedback on feast-dev#6655. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * fix: replace UDF decorator strip regex with linear scan Avoid CodeQL ReDoS finding on nested @/newline regex when rehydrating body_text. Add multiline and adversarial @ spam unit tests. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * fix: cast resolve_udf result to FunctionType for mypy Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * fix: compare UDF transformations by udf_string when present Source-first from_proto rebuilds callables whose bytecode differs from the live repo function; requiring co_code equality made no-op feast apply rewrite ODFVs and broke universal CLI integration tests. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * lint Signed-off-by: Aniket Paluskar <apaluska@redhat.com> * fix: align Ray __eq__ with source identity; cache Spark UDF resolve RayTransformation now compares by udf_string when present (same contract as Pandas/Python) so a future source-first from_proto will not break no-op apply. SparkTransformationNode caches resolve_udf so strip+exec runs once per node lifetime. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

aniketpalu requested a review from a team as a code owner July 29, 2026 09:35

github-advanced-security Bot found potential problems Jul 29, 2026

View reviewed changes

Comment thread sdk/python/feast/transformation/udf_rehydrate.py Fixed

aniketpalu changed the title ~~fix: Fixes three issues found while testing BatchFeatureView UDFs and OnDemandFeatureViews with Spark transformations (SparkApplication / SparkComputeEngine path) against a remote feature server~~ fix: UDF/ODFV source rehydrate (+ Postgres / online cache) Jul 29, 2026

franciscojavierarceo requested changes Jul 29, 2026

View reviewed changes

aniketpalu force-pushed the fix/udf-source-rehydrate-postgres-online-cache branch from fd2178d to 3028ff9 Compare August 6, 2026 10:28

aniketpalu and others added 5 commits August 10, 2026 18:51

Lint

12ae434

Signed-off-by: Aniket Paluskar <apaluska@redhat.com>

fix: replace UDF decorator strip regex with linear scan

45ad64a

Avoid CodeQL ReDoS finding on nested @/newline regex when rehydrating body_text. Add multiline and adversarial @ spam unit tests. Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

fix: cast resolve_udf result to FunctionType for mypy

708ff26

Signed-off-by: Aniket Paluskar <apaluska@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>

ntkathole force-pushed the fix/udf-source-rehydrate-postgres-online-cache branch from 648ef47 to 708ff26 Compare August 10, 2026 13:21

ntkathole added the ok-to-test label Aug 10, 2026

aniketpalu and others added 4 commits August 11, 2026 14:35

Merge branch 'master' into fix/udf-source-rehydrate-postgres-online-c…

f2878fb

…ache

Merge branch 'master' into fix/udf-source-rehydrate-postgres-online-c…

9dbe4f1

…ache

lint

4e3aaf4

Signed-off-by: Aniket Paluskar <apaluska@redhat.com>

aniketpalu requested a review from franciscojavierarceo August 13, 2026 07:36

aniketpalu and others added 2 commits August 17, 2026 13:29

Merge branch 'master' into fix/udf-source-rehydrate-postgres-online-c…

0c6e240

…ache

ntkathole approved these changes Aug 17, 2026

View reviewed changes

ntkathole merged commit 5fd7af7 into feast-dev:master Aug 17, 2026
23 checks passed

Navigation Menu

Uh oh!

fix: UDF/ODFV source rehydrate (+ Postgres / online cache) - #6655

fix: UDF/ODFV source rehydrate (+ Postgres / online cache)#6655
ntkathole merged 11 commits into
feast-dev:masterfrom
aniketpalu:fix/udf-source-rehydrate-postgres-online-cache

aniketpalu commented Jul 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

codecov-commenter commented Jul 29, 2026 •

edited

Loading

Uh oh!

franciscojavierarceo left a comment

Uh oh!

ntkathole commented Jul 30, 2026

Uh oh!

ntkathole commented Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

aniketpalu commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

1. Spark BFV UDF — prefer source over dill (exit 139)

2. ODFV / local transformations — source-first from_proto

3. Postgres empty feature_name_columns → SELECT *

4. Fresh online registry reads

Out of scope

Which issue(s) this PR fixes

Checks

Testing Strategy

Misc

Uh oh!

Uh oh!

codecov-commenter commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

franciscojavierarceo left a comment

Choose a reason for hiding this comment

Uh oh!

ntkathole commented Jul 30, 2026

Uh oh!

ntkathole commented Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

aniketpalu commented Jul 29, 2026 •

edited

Loading

2. ODFV / local transformations — source-first `from_proto`

3. Postgres empty `feature_name_columns` → SELECT *

codecov-commenter commented Jul 29, 2026 •

edited

Loading