Phase 61.1 Spec & Technical: DataQualityAssessor — Data Profiling & Label Noise Detection

[web3guru888]

DataQualityAssessor — Specification & Technical Discussion

Related Issue: #1146

Planning Discussion: #1145

This thread covers the technical specification and implementation details for the DataQualityAssessor component.

Key Topics

1. Statistical Profiling: Best approaches for feature-level statistics (moments, quantiles, distribution fitting)
2. Outlier Detection: Isolation Forest vs. LOF vs. Mahalanobis — which methods for which data modalities?
3. Label Noise Detection: Implementing Confident Learning (Cleanlab) — cross-validation strategies and threshold tuning
4. Data Completeness: Metrics for missing data patterns (MCAR, MAR, MNAR classification)
5. Streaming Profiling: Incremental statistics for large-scale datasets (Welford's algorithm, HyperLogLog)
6. Integration: How DataQualityAssessor feeds into ActiveLearner and DataVersionController

Open Questions

• Should we use exact or approximate statistics for datasets > 10M rows?
• How to handle mixed-type features (numeric + categorical)?
• What confidence threshold for label noise flagging? (Cleanlab default: 0.5)