Currently, the Task Runner will save the best model based on the accuracy score acquired during aggregated model validation. A higher accuracy score indicates better performance (ref). This is good practice in general, but is limited to metrics where higher score indicates better performance. There will be cases where a user will use a loss metric for validation. In this case, lower values indicate better performance and the Task Runner will not save the best model since it is looking for the highest value.
This configuration should be via the enabled via plan.yaml
A natural location to consider would be the task_*.yaml (ref)
Currently, the Task Runner will save the best model based on the accuracy score acquired during aggregated model validation. A higher accuracy score indicates better performance (ref). This is good practice in general, but is limited to metrics where higher score indicates better performance. There will be cases where a user will use a loss metric for validation. In this case, lower values indicate better performance and the Task Runner will not save the best model since it is looking for the highest value.
This configuration should be via the enabled via
plan.yamlA natural location to consider would be the
task_*.yaml(ref)