Skip to content

Commit abcf16f

Browse files
mshuaibiilevineds
mshuaibii
and
levineds
authored
Add OMol evaluations (#1412)
* init omol evals * remove absolute paths * add omol recipe unit tests * fix omol imports * add omol to workflow * bump sella version * cpu test * update gpu dependencies * add omol evaluator * reorganize * update imports * placeholder docs * allow reducer to handle no targets * add leaderboard documentation * untrack omol configs * revert all config changes * update documentation * update tests * Update leaderboard.md * add omol eval runners * Update leaderboard.md * Update leaderboard.md * Update leaderboard.md * Update src/fairchem/core/components/calculate/omol_runner.py Co-authored-by: Daniel Levine <levineds@meta.com> * Update leaderboard.md * Update leaderboard.md * Update leaderboard.md * Update leaderboard.md * Update leaderboard.md --------- Co-authored-by: Daniel Levine <levineds@meta.com>
1 parent fa44409 commit abcf16f

13 files changed

Lines changed: 1504 additions & 6 deletions

File tree

‎configs/uma/benchmark/omol-geom.yaml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
defaults:
2+
- cluster: v100
3+
- checkpoint: uma_sm
4+
- _self_
5+
6+
benchmark_name: geom_conformers
7+
8+
job:
9+
run_name: ${checkpoint.model_name}
10+
run_dir: ${cluster.run_dir}
11+
device_type: ${cluster.device}
12+
debug: ${cluster.debug}
13+
scheduler:
14+
mode: ${cluster.mode}
15+
distributed_init_method: FILE
16+
num_array_jobs: 400
17+
slurm:
18+
partition: ${cluster.partition}
19+
mem_gb: ${cluster.mem_gb}
20+
timeout_hr: 72
21+
logger:
22+
_target_: fairchem.core.common.logger.WandBSingletonLogger.init_wandb
23+
_partial_: true
24+
entity: fairchem
25+
project: uma-benchmarks
26+
group: ${checkpoint.model_name}
27+
job_type: ${benchmark_name}
28+
29+
runner:
30+
_target_: fairchem.core.components.calculate.omol_runner.OMolRunner
31+
calculator:
32+
_target_: fairchem.core.FAIRChemCalculator.from_model_checkpoint
33+
name_or_path: ${checkpoint.ckpt_path}
34+
task_name: omol
35+
input_data: ${cluster.data_root_dir}/conformer_inputs.pkl
36+
benchmark_name: ${benchmark_name}
37+
benchmark:
38+
_target_: fairchem.core.components.calculate.recipes.omol.conformers
39+
_partial_: True
40+
41+
reducer:
42+
_target_: fairchem.core.components.benchmark.omol_reducer.OMolReducer
43+
benchmark_name: ${benchmark_name}

‎configs/uma/benchmark/omol-ieea.yaml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
defaults:
2+
- cluster: v100
3+
- checkpoint: uma_sm
4+
- _self_
5+
6+
benchmark_name: ie_ea
7+
8+
job:
9+
run_name: ${checkpoint.model_name}
10+
run_dir: ${cluster.run_dir}
11+
device_type: ${cluster.device}
12+
debug: ${cluster.debug}
13+
scheduler:
14+
mode: ${cluster.mode}
15+
distributed_init_method: FILE
16+
num_array_jobs: 10
17+
slurm:
18+
partition: ${cluster.partition}
19+
mem_gb: ${cluster.mem_gb}
20+
timeout_hr: 72
21+
logger:
22+
_target_: fairchem.core.common.logger.WandBSingletonLogger.init_wandb
23+
_partial_: true
24+
entity: fairchem
25+
project: uma-benchmarks
26+
group: ${checkpoint.model_name}
27+
job_type: ${benchmark_name}
28+
29+
runner:
30+
_target_: fairchem.core.components.calculate.omol_runner.OMolRunner
31+
calculator:
32+
_target_: fairchem.core.FAIRChemCalculator.from_model_checkpoint
33+
name_or_path: ${checkpoint.ckpt_path}
34+
task_name: omol
35+
input_data: ${cluster.data_root_dir}/ieea_inputs.pkl
36+
benchmark_name: ${benchmark_name}
37+
benchmark:
38+
_target_: fairchem.core.components.calculate.recipes.omol.ieea
39+
_partial_: True
40+
41+
reducer:
42+
_target_: fairchem.core.components.benchmark.omol_reducer.OMolReducer
43+
benchmark_name: ${benchmark_name}

‎configs/uma/benchmark/omol-pocket.yaml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
defaults:
2+
- cluster: v100
3+
- checkpoint: uma_sm
4+
- _self_
5+
6+
benchmark_name: ligand_pocket
7+
8+
job:
9+
run_name: ${checkpoint.model_name}
10+
run_dir: ${cluster.run_dir}
11+
device_type: ${cluster.device}
12+
debug: ${cluster.debug}
13+
scheduler:
14+
mode: ${cluster.mode}
15+
distributed_init_method: FILE
16+
num_array_jobs: 1
17+
slurm:
18+
partition: ${cluster.partition}
19+
mem_gb: ${cluster.mem_gb}
20+
timeout_hr: 72
21+
logger:
22+
_target_: fairchem.core.common.logger.WandBSingletonLogger.init_wandb
23+
_partial_: true
24+
entity: fairchem
25+
project: uma-benchmarks
26+
group: ${checkpoint.model_name}
27+
job_type: ${benchmark_name}
28+
29+
runner:
30+
_target_: fairchem.core.components.calculate.omol_runner.OMolRunner
31+
calculator:
32+
_target_: fairchem.core.FAIRChemCalculator.from_model_checkpoint
33+
name_or_path: ${checkpoint.ckpt_path}
34+
task_name: omol
35+
input_data: ${cluster.data_root_dir}/ligand_pocket_inputs.pkl
36+
benchmark_name: ${benchmark_name}
37+
benchmark:
38+
_target_: fairchem.core.components.calculate.recipes.omol.ligand_pocket
39+
_partial_: True
40+
41+
reducer:
42+
_target_: fairchem.core.components.benchmark.omol_reducer.OMolReducer
43+
benchmark_name: ${benchmark_name}

‎configs/uma/benchmark/omol-prot.yaml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
defaults:
2+
- cluster: v100
3+
- checkpoint: uma_sm
4+
- _self_
5+
6+
benchmark_name: protonation_energies
7+
8+
job:
9+
run_name: ${checkpoint.model_name}
10+
run_dir: ${cluster.run_dir}
11+
device_type: ${cluster.device}
12+
debug: ${cluster.debug}
13+
scheduler:
14+
mode: ${cluster.mode}
15+
distributed_init_method: FILE
16+
num_array_jobs: 400
17+
slurm:
18+
partition: ${cluster.partition}
19+
mem_gb: ${cluster.mem_gb}
20+
timeout_hr: 72
21+
logger:
22+
_target_: fairchem.core.common.logger.WandBSingletonLogger.init_wandb
23+
_partial_: true
24+
entity: fairchem
25+
project: uma-benchmarks
26+
group: ${checkpoint.model_name}
27+
job_type: ${benchmark_name}
28+
29+
runner:
30+
_target_: fairchem.core.components.calculate.omol_runner.OMolRunner
31+
calculator:
32+
_target_: fairchem.core.FAIRChemCalculator.from_model_checkpoint
33+
name_or_path: ${checkpoint.ckpt_path}
34+
task_name: omol
35+
input_data: ${cluster.data_root_dir}/protonation_inputs.pkl
36+
benchmark_name: ${benchmark_name}
37+
benchmark:
38+
_target_: fairchem.core.components.calculate.recipes.omol.protonation
39+
_partial_: True
40+
41+
reducer:
42+
_target_: fairchem.core.components.benchmark.omol_reducer.OMolReducer
43+
benchmark_name: ${benchmark_name}

‎configs/uma/benchmark/omol-scale.yaml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
defaults:
2+
- cluster: v100
3+
- checkpoint: uma_sm
4+
- _self_
5+
6+
benchmark_name: distance_scaling
7+
8+
job:
9+
run_name: ${checkpoint.model_name}
10+
run_dir: ${cluster.run_dir}
11+
device_type: ${cluster.device}
12+
debug: ${cluster.debug}
13+
scheduler:
14+
mode: ${cluster.mode}
15+
distributed_init_method: FILE
16+
num_array_jobs: 200
17+
slurm:
18+
partition: ${cluster.partition}
19+
mem_gb: ${cluster.mem_gb}
20+
timeout_hr: 72
21+
logger:
22+
_target_: fairchem.core.common.logger.WandBSingletonLogger.init_wandb
23+
_partial_: true
24+
entity: fairchem
25+
project: uma-benchmarks
26+
group: ${checkpoint.model_name}
27+
job_type: ${benchmark_name}
28+
29+
runner:
30+
_target_: fairchem.core.components.calculate.omol_runner.OMolRunner
31+
calculator:
32+
_target_: fairchem.core.FAIRChemCalculator.from_model_checkpoint
33+
name_or_path: ${checkpoint.ckpt_path}
34+
task_name: omol
35+
input_data: ${cluster.data_root_dir}/distance_scaling_inputs.pkl
36+
benchmark_name: ${benchmark_name}
37+
benchmark:
38+
_target_: fairchem.core.components.calculate.recipes.omol.distance_scaling
39+
_partial_: True
40+
41+
reducer:
42+
_target_: fairchem.core.components.benchmark.omol_reducer.OMolReducer
43+
benchmark_name: ${benchmark_name}

‎configs/uma/benchmark/omol-spin.yaml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
defaults:
2+
- cluster: v100
3+
- checkpoint: uma_sm
4+
- _self_
5+
6+
benchmark_name: spin_gap
7+
8+
job:
9+
run_name: ${checkpoint.model_name}
10+
run_dir: ${cluster.run_dir}
11+
device_type: ${cluster.device}
12+
debug: ${cluster.debug}
13+
scheduler:
14+
mode: ${cluster.mode}
15+
distributed_init_method: FILE
16+
num_array_jobs: 10
17+
slurm:
18+
partition: ${cluster.partition}
19+
mem_gb: ${cluster.mem_gb}
20+
timeout_hr: 72
21+
logger:
22+
_target_: fairchem.core.common.logger.WandBSingletonLogger.init_wandb
23+
_partial_: true
24+
entity: fairchem
25+
project: uma-benchmarks
26+
group: ${checkpoint.model_name}
27+
job_type: ${benchmark_name}
28+
29+
runner:
30+
_target_: fairchem.core.components.calculate.omol_runner.OMolRunner
31+
calculator:
32+
_target_: fairchem.core.FAIRChemCalculator.from_model_checkpoint
33+
name_or_path: ${checkpoint.ckpt_path}
34+
task_name: omol
35+
input_data: ${cluster.data_root_dir}/spingap_inputs.pkl
36+
benchmark_name: ${benchmark_name}
37+
benchmark:
38+
_target_: fairchem.core.components.calculate.recipes.omol.spin_gap
39+
_partial_: True
40+
41+
reducer:
42+
_target_: fairchem.core.components.benchmark.omol_reducer.OMolReducer
43+
benchmark_name: ${benchmark_name}

‎configs/uma/benchmark/omol-strain.yaml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
defaults:
2+
- cluster: v100
3+
- checkpoint: uma_sm
4+
- _self_
5+
6+
benchmark_name: ligand_strain
7+
8+
job:
9+
run_name: ${checkpoint.model_name}
10+
run_dir: ${cluster.run_dir}
11+
device_type: ${cluster.device}
12+
debug: ${cluster.debug}
13+
scheduler:
14+
mode: ${cluster.mode}
15+
distributed_init_method: FILE
16+
num_array_jobs: 400
17+
slurm:
18+
partition: ${cluster.partition}
19+
mem_gb: ${cluster.mem_gb}
20+
timeout_hr: 72
21+
logger:
22+
_target_: fairchem.core.common.logger.WandBSingletonLogger.init_wandb
23+
_partial_: true
24+
entity: fairchem
25+
project: uma-benchmarks
26+
group: ${checkpoint.model_name}
27+
job_type: ${benchmark_name}
28+
29+
runner:
30+
_target_: fairchem.core.components.calculate.omol_runner.OMolRunner
31+
calculator:
32+
_target_: fairchem.core.FAIRChemCalculator.from_model_checkpoint
33+
name_or_path: ${checkpoint.ckpt_path}
34+
task_name: omol
35+
input_data: ${cluster.data_root_dir}/ligand_strain_inputs.pkl
36+
benchmark_name: ${benchmark_name}
37+
benchmark:
38+
_target_: fairchem.core.components.calculate.recipes.omol.ligand_strain
39+
_partial_: True
40+
41+
reducer:
42+
_target_: fairchem.core.components.benchmark.omol_reducer.OMolReducer
43+
benchmark_name: ${benchmark_name}

0 commit comments

Comments
 (0)
⚡