Sitelet https://github.com/feast-dev/feast/issues/6783
Skip to content

Monitoring metrics crash on NaN/Inf feature values, silently dropping the whole feature view #6783

Description

@piyush182004

Expected Behavior

Computing monitoring metrics for a numeric feature should succeed even when some values are NaN or +-Infinity, which occur naturally in feature engineering (e.g. a ratio whose denominator is zero).

Current Behavior

MetricsCalculator.compute_numeric() passes values straight into np.histogram(), which raises on non-finite input:

ValueError: autodetected range of [0.05, inf] is not finite

compute_all() has no per-column error handling, so this exception discards metrics for the entire feature view - including columns that are perfectly well-formed.

The failure is silent to the caller: POST /monitoring/compute (and the auto_compute endpoint the Feast UI's "Compute Metrics" button calls) still returns HTTP 200 with "status": "completed" and "computed_features": 0. Nothing surfaces to the UI - the feature view is simply absent from the Monitoring page, with no error shown anywhere.

The same unguarded np.histogram call is duplicated in the Dask offline store (_dask_compute_numeric_metrics), so it fails there too.

Steps to reproduce

import pyarrow as pa
from feast.monitoring.metrics_calculator import MetricsCalculator

calc = MetricsCalculator()
arr = pa.array([0.05, 0.06, float("inf"), 0.08], type=pa.float64())
calc.compute_numeric(arr)
# ValueError: autodetected range of [0.05, inf] is not finite

Or end-to-end: define a FeatureView with a feature computed as clicks / impressions where one row has impressions = 0, enable data_quality_monitoring in feature_store.yaml, and run feast apply. The apply succeeds but logs:

ERROR:feast.monitoring.monitoring_service:Failed to compute baseline for feature view 'campaign_stats'
ValueError: autodetected range of [0.05, inf] is not finite

and the feature view never appears on the Monitoring page, even though feast apply itself reports no error.

Specifications

  • Version: master
  • Platform: Linux (also affects the Dask offline store code path)
  • Subsystem: monitoring / metrics_calculator

Possible Solution

Filter non-finite values out after dropping nulls, before computing any statistic, and apply the existing _safe_float/opt_float helpers (which already exist for this exact purpose but were only applied to mean and stddev) to min_val, max_val, and the quantiles as well. Fix in both feast/monitoring/metrics_calculator.py and feast/infra/offline_stores/dask.py. Submitted as a PR alongside this issue.

FIX :- #6782

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions