omol lbfgs evals (#1488)

mshuaibii · web-flow · commit 51b85366dfdf · 2025-09-15T20:15:26.000Z
* lbfgs optimizer

* Update leaderboard.md

* Update leaderboard.md
diff --git a/docs/molecules/leaderboard.md b/docs/molecules/leaderboard.md
@@ -7,11 +7,22 @@ The leaderboard is broken into two different sections - "S2EF" and "Evaluations"
 Structure to Energy and Forces (S2EF) is the most straightforward evaluation for MLIPs - given a structure, how well can you predict the total energy and per-atom forces.
 Evaluations correspond to several chemistry relevant tasks (spin gap, ligand-strain, etc.) introduced in OMol25 to evaluate MLIPs beyond simple energy and force metrics (see the [paper](https://arxiv.org/pdf/2505.08762) for more details).
 
-The simplest way to get started is to have an ASE-compatible MLIP calculator that can make energy and force predictions.
+The simplest way to get started is to have an ASE-compatible MLIP calculator that can make energy and force predictions. Input data for the different benchmarks can be downloaded below.
+
+## 💾 Download
+
+| Benchmarks | URL | 
+|----------|----------|
+| S2EF (Val/Test)   | https://huggingface.co/facebook/OMol25/blob/main/DATASET.md#dataset-splits     |
+| Evaluations    | https://huggingface.co/facebook/OMol25/blob/main/DATASET.md#evaluation-data     | 
 
 ## S2EF
-The leadebroard supports S2EF evaluations for both the OMol25 "Validation" and "Test" sets. The download links for these datasets can be found [🔗 here](https://huggingface.co/facebook/OMol25/blob/main/DATASET.md#dataset-splits).
-Validation and Test sets should contain 2,762,021 and 2,805,046 samples, respectively.
+The leadebroard supports S2EF evaluations for both the OMol25 "Validation" and "Test" sets. Validation labels are already accessible in the released dataset for local benchmarking and debugging, so we highly encourage users to make Test submissions to fairly and accurately compare models. The size of each split is as follows:
+
+| Split | Size | 
+|----------|----------|
+| Val   | 2,762,021 |
+| Test    | 2,805,046     | 
 
 Predictions must be saved as ".npz" files and shall contain the following information:
 ```
@@ -51,6 +62,7 @@ for idx in range(len(dataset)):
     energy.append(atoms.get_potential_energy())
     forces.append(atoms.get_forces())
 
+### Do not forget this! Your submission will fail.
 forces = np.concatenate(forces)
 
 np.savez_compressed(
@@ -74,18 +86,21 @@ The following evaluations are currently available on the OMol25 leaderboard:
 * Conformers: Identifying the lowest energy conformer is a crucial part of many biological and pharmaceutical tasks.
 * Protonation: As a proxy to pKa prediction, we evaluate energy differences of structures differing by one proton.
 * Distance scaling: Short range and long range intermolecular interactions are essential for observable properties like phase changes, density, etc.
+* IE/EA: The addition, removal, and transfer of electrons is central to many redox processes.
+* Spin gap: Differences between spin states can play a critical role of molecular optic devices and photactive catalysts.
 
 For a detailed descripion of each task we refer people to the original [manuscript](https://arxiv.org/pdf/2505.08762).
-The download links for evaluation inputs can be found [🔗 here](https://huggingface.co/facebook/OMol25/blob/main/DATASET.md#evaluation-data).
 
 To generate prediction files for the different tasks, we have released a set of [recipes](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py) to be used with ASE-compatible calculators.
 Each evaluation task has its own unique structure, a detailed description of the expected output is provided in the recipe docstrings. The following recipes should be used to evaluate the corresponding task:
 
-* [Ligand pocket](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L321)
-* [Ligand strain](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L370)
-* [Conformers](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L138)
-* [Protonation](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L186)
-* [Distance scaling](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L437)
+* [Ligand pocket](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L323)
+* [Ligand strain](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L372)
+* [Conformers](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L140)
+* [Protonation](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L188)
+* [Distance scaling](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L439)
+* [IE/EA](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L237)
+* [Spin gap](https://github.com/facebookresearch/fairchem/blob/main/src/fairchem/core/components/calculate/recipes/omol.py#L284)
 
 As an example:
 
diff --git a/src/fairchem/core/components/calculate/recipes/omol.py b/src/fairchem/core/components/calculate/recipes/omol.py
@@ -34,7 +34,7 @@
     from ase import Atoms
     from ase.calculators.calculator import Calculator
 
-from fairchem.data.omol.orca.calc import TIGHT_OPT_PARAMETERS
+from fairchem.data.omol.orca.calc import EVAL_OPT_PARAMETERS
 from pymatgen.io.ase import MSONAtoms
 from tqdm import tqdm
 
@@ -178,7 +178,7 @@ def conformers(input_data: dict[str, Any], calculator: Calculator) -> dict[str,
         for conformer in conformers:
             sid = conformer["sid"]
             initial_atoms = conformer["initial_atoms"]
-            results = relax_job(initial_atoms, calculator, TIGHT_OPT_PARAMETERS)
+            results = relax_job(initial_atoms, calculator, EVAL_OPT_PARAMETERS)
             conformer_results[sid] = results
 
         all_results[molecule_family] = conformer_results
@@ -226,7 +226,7 @@ def protonation(input_data: dict[str, Any], calculator: Calculator) -> dict[str,
         states = input_data[molecule_family]
         for state in states:
             initial_atoms = states[state]["initial_atoms"]
-            results = relax_job(initial_atoms, calculator, TIGHT_OPT_PARAMETERS)
+            results = relax_job(initial_atoms, calculator, EVAL_OPT_PARAMETERS)
 
             state_results[state] = results
 
@@ -427,7 +427,7 @@ def ligand_strain(input_data: dict[str, Any], calculator: Calculator) -> dict[st
         # Gas-phase conformers parts
         conformer_prediction = {}
         for idx, initial_atoms in enumerate(ligand_system["conformers"]):
-            results = relax_job(initial_atoms, calculator, TIGHT_OPT_PARAMETERS)
+            results = relax_job(initial_atoms, calculator, EVAL_OPT_PARAMETERS)
 
             conformer_prediction[idx] = results
         complex_results["gas_phase"] = conformer_prediction
diff --git a/src/fairchem/data/omol/orca/calc.py b/src/fairchem/data/omol/orca/calc.py
@@ -13,6 +13,7 @@
 
 from ase import Atoms
 from ase.calculators.orca import ORCA, OrcaProfile
+from ase.optimize import LBFGS
 from sella import Sella
 
 # ECP sizes taken from Table 6.5 in the Orca 5.0.3 manual
@@ -164,6 +165,12 @@
         "internal": True,
     },
 }
+EVAL_OPT_PARAMETERS = {
+    "optimizer": LBFGS,
+    "fmax": 0.01,
+    "max_steps": 500,
+    "optimizer_kwargs": {},
+}
 
 
 class Vertical(Enum):
