[Feature] Built-in feature drift detection with alerting

[rehan243]

Description

As a feature store, Feast is in a unique position to detect data drift between training and serving feature distributions. Built-in PSI/KS-test monitoring with configurable alerts would be very valuable.

Use Case

• Detect when feature distributions shift significantly
• Alert ML engineers before model performance degrades
• Integrate with existing monitoring (Prometheus, Grafana)

[rehan243]

Hey, love the idea of built-in drift detection with Feast since it’s sitting right at the intersection of training and serving data. PSI/KS tests are a solid call for catching distribution shifts, and hooking into Prometheus for alerting makes total sense — we’ve been using that combo for monitoring feature pipelines at scale with 100k+ feature updates daily. Honestly, the trickiest bit is setting thresholds for alerts that don’t spam you with false positives; took us a few tries to dial that in.

One quick thought: Feast could expose a simple API to plug in custom drift metrics or even pre-trained detectors if you’ve got something fancy. Something like this could work:

from feast.drift import DriftDetector
detector = DriftDetector(metric="ksi", threshold=0.3)
feature_view.register_drift_detector(detector, alert_channel="prometheus")

Turns out, integrating this at the feature view level keeps the noise down compared to global monitoring.

[reallyticsai]

Implementing feature drift detection directly in Feast makes a lot of sense, especially since it’s already tightly integrated into data pipelines. We've integrated PSI (Population Stability Index) and KS-tests in production for similar purposes, typically leveraging libraries like scipy.stats for KS-test and custom scripts for PSI calculations.

A practical approach is to run these tests on a daily or hourly basis, store the metrics in Prometheus, and create alert rules based on thresholds that reflect significant shifts (e.g., PSI > 0.2 or KS p-value < 0.05). These can then trigger alerts in Grafana or via email. For a more scalable solution, consider batching these checks into a separate monitoring job that can be triggered via Feast’s batch ingestion or via a sidecar process, ensuring minimal impact on the primary feature serving pipeline.

[franciscojavierarceo]

@jyejare

[jyejare]

Hello @rehan243 and @reallyticsai thanks for the interest in feast monitoring feature.

I have been working towards that from past few weeks now and opened 2 pull requests #6340 and #6202 that builds the foundation of monitoring / drift detection and alerting.

Please help review those PRs and / or feel free to contribute in drift detection and alerting.