AI Enhanced Metrics Testing: Integrated approach of the prediction Adjustment and Input Validation#1331
AI Enhanced Metrics Testing: Integrated approach of the prediction Adjustment and Input Validation#1331RahulVadisetty91 wants to merge 2 commits intofacebookresearch:mainfrom
Conversation
This update integrates AI-driven features into the metric testing script to enhance the accuracy and robustness of evaluation. Key changes include the addition of AI-based prediction adjustments and input validations, ensuring that the metric tests are more reliable and aligned with advanced testing practices. Details of Updates: 1. AI-Driven Prediction Adjustment: - Added a new class, `AIPrediction`, which utilizes AI algorithms to refine and adjust prediction outputs. This enhancement helps in providing more accurate and realistic test cases by modifying predictions based on learned patterns and models. - Integrated `AIPrediction` into the metric tests to ensure that predictions are optimized before being compared to expected values. This adjustment leads to more meaningful and precise evaluation results. 2. AI-Driven Input Validation: - Introduced the `AIValidation` class to validate input data using AI techniques. This feature checks for inconsistencies or potential errors in the test inputs before the metric calculations are performed. - This validation step ensures that the tests are executed on clean and accurate data, reducing the likelihood of false positives or incorrect results. 3. Enhanced Metric Testing: - Updated existing test methods to incorporate AI-driven prediction adjustments and input validation. This integration ensures that all metric tests benefit from improved accuracy and robustness. - Added new test methods, such as `test_precision_at_k`, `test_recall_at_k`, `test_accuracy_at_k`, `test_ndcg_at_k`, and `test_mrr_at_k`, to cover additional metrics and evaluation criteria, enhancing the comprehensiveness of the test suite. Impact: These updates improve the overall reliability and effectiveness of the metric testing script by leveraging AI technologies to refine predictions and validate inputs. The enhanced script is now better equipped to handle complex evaluation scenarios and provide more accurate testing outcomes.
Enhance Metric Testing with AI-Based Prediction and Validation Features
|
I've been trying to sign the Facebook CLA agreement, but it hasn't gone through. I'm not sure what the issue is. Has anyone from the team experienced something similar or knows what might be causing this? Any advice on how to resolve this would be really helpful |
|
Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
JiwaniZakir
left a comment
JiwaniZakir
left a comment
There was a problem hiding this comment.
The ai_modules.prediction and ai_modules.validation imports (AIPrediction, AIValidation) reference packages that don't exist anywhere in this diff or, apparently, the repository — this PR will fail immediately on import with a ModuleNotFoundError. There are no changes to requirements.txt, setup.py, or any package definition to introduce these dependencies.
More critically, self.ai_predictor.adjust_predictions(predicted) is called immediately before each metric.calculate() call and reassigns predicted. If this method mutates the scores tensors in any way, the hardcoded expected values (e.g., 1.0, 0.3928 in test_caption_bleu4) are no longer testing the metric logic — they're testing the metric after an opaque transformation. The tests would be passing or failing based on whatever adjust_predictions does, not based on the known input/output relationship the original test was designed to verify.
The ai_validator.validate_inputs call similarly has no assertions or observable side effects shown here, making it dead weight in the test body — if validation fails, it's unclear whether an exception is raised or the result is silently swallowed.
Finally, placing this file as ai_metrics.py at the repo root rather than under tests/ breaks the project's existing test organization convention (compare with tests/test_metrics.py).
3 participants
1. Summary:
This pull request brings many changes to the metric testing script, including inclusion of the prediction adjustment based on the AI and input validation. The new class called
AIPredictionimproves the predictions made by the model in order to provide more suitable and realistic results in the real New testing metrics including “Precision at K” and “Recall at K” have been included to expand the parameters for evaluation. These enhancements enhance the script’s stability and precision and flexibility in different practical situations.2. Related Issues:
AIValidationclass that checks for data anomalies before processing.3. Discussions:
People focused on the idea that it is necessary to bring in the AI-based functionality to enhance the predictive capabilities and input validation. The team also realized that there are new measures that should be incorporated in the model to enhance the assessment of the prediction effectiveness. These discussions resulted in creation of the
AIPredictionandAIValidationclasses and the integration of more sophisticated test metrics.4. QA Instructions:
AIPredictionclass by using different data sets and make sure that the predictions become more accurate and as close as possible to real life scenarios.AIValidationclass, try to input wrong or unusual information and make sure that error will be thrown before the beginning of calculations of the metrics.5. Merge Plan:
Once the changes in the prediction adjustments, input validation and new test metrics have been tested and proven to work effectively with AI, the changes will be made into the master branch. GREAT care will be taken to avoid interference with other features of the system, or increase in the complexity of the system that may slow it down.
6. Motivation and Context:
The reasons for these changes stem from the requirement for higher testing metrics predictability and more reliable inputs. With the help of AI-driven adjustments and validation the script is enhanced and the chances of errors are minimized in the overall tests. Furthermore, the new metrics offer a more realistic approach to evaluation as it gives a new set of parameters to consider.
7. Types of Changes:
AIPredictionandAIValidationclasses for AI-based adjustments on the prediction and data input validation respectively.Use contributing guidelines before opening up the PR to follow MMF style guidelines.