Reproduce olmov2 SFT

[ahmeda14960]

Description

Match Olmo v2 SFT on core evals (MMLU) and instruction following (alpacaeval)

Hypothesis or Goal

Trying to match the perf of Olmo v2 SFT given this dataset

Links

• WandB Report: (link)
• Data Browser: (link)
• Experiment JSON: (link)
• (etc.)

Results

(What did you find, including relevant evaluation metrics, etc.)

[ahmeda14960]

This is currently blocked by a strange bug somewhere that prevents transformers from updating to 4.47.0, regardless of whether we update pip_deps or pyproject.toml

[ahmeda14960]

Experiment link for tulu3: https://marlin-subtle-barnacle.ngrok-free.app/experiment?path=gs%3A//marin-us-central2/experiments/606_sft-30c565.json

Official Claimed Tulu 3 SFT MMLU: 65.9

Tulu 3 SFT Lmeval numbers MMLU: 63.73

Marin Tulu 3 Reproduction on MMLU: 62.41

There is still a gap here, but the Tulu paper states that they do 0-shot MMLU eval with chain of thought which we don't currently support

The gap for Alpaca eval is non-trivial:

Official Tulu 3 SFT Alpaca-LC: 12.4

Marin Tulu 3 SFT Alpaca-LC: 8.35

[ahmeda14960]

More seeds (will update as results come in)

Seed 1:

• MMLU: 62.69
• Alpaca-LC: 9.54

Seed 2:

• MMLU: 62.42
• Alpaca-LC: 7.31

Seed 42:

• MMLU: 62.85
• Alpaca-LC: 6.91

average variance for Alpaca-LC is 0.11, for MMLU is 0.39

[ahmeda14960]

allenai__Llama-3.1-Tulu-3-8B-SFT

"results": {
"alpaca_eval": {
"win_rate": 9.343216827175963,
"standard_error": 0.8874822488050991,
"n_wins": 63,
"n_wins_base": 736,
"n_draws": 6,
"n_total": 805,
"discrete_win_rate": 8.198757763975156,
"mode": "community",
"avg_length": 1174,
"length_controlled_winrate": 13.993435307067855,
"lc_standard_error": 0.6121563734674437,
"num_examples": 805,
"completion_rate": 1.0
}

marin tulu 3 reproduction all docs (7335 steps)

"results": {
"alpaca_eval": {
"win_rate": 14.224671738135687,
"standard_error": 1.0475504031399154,
"n_wins": 112,
"n_wins_base": 690,
"n_draws": 3,
"n_total": 805,
"discrete_win_rate": 14.09937888198758,
"mode": "community",
"avg_length": 1522,
"length_controlled_winrate": 17.769976813835335,
"lc_standard_error": 0.6499752005333808,
"num_examples": 805,
"completion_rate": 1.0
}
},

[ahmeda14960]

Below are results that more faithfully capture the setting under which the Tulu 3 SFT checkpoint was produced. Before we were training without sequence packing so both models were seeing the same number of tokens but with many more gradient updates for our model. Now with sequence packing, we should have a similar number of gradient updates as the Tulu 3 SFT checkpoint and a similar number of tokens.

Our discrete win_rate and LC win rate matches the numbers for the official Tulu 3 SFT checkpoint within the standard error, which validates our pipeline. Interestingly it seems that without sequence packing our reproduction of Tulu 3 SFT is stronger on alpaca eval (9.88 -> 14 on discrete win rate, 13.53 -> 17.76 on LC win rate) so it seems more gradient steps per example is more beneficial. We can now fully conclude this reproduction.

Sequence packing seems to save ~17% wall clock time then training over the same number of tokens without sequence packing.

Summary:

Official Tulu 3 SFT Alpaca results:
win rate: 9.34 LC: 13.99

Our Tulu 3 SFT with sequence packing results:
win rate: 9.88 LC: 13.53

Our Tulu 3 SFT with no sequence packing and thus more gradient steps results:
win rate: 14.22. LC: 17.76

{
"results": {
"alpaca_eval": {
"win_rate": 9.886481271929224,
"standard_error": 0.9065225540381237,
"n_wins": 73,
"n_wins_base": 731,
"n_draws": 1,
"n_total": 805,
"discrete_win_rate": 9.130434782608695,
"mode": "community",
"avg_length": 1435,
"length_controlled_winrate": 13.530757774168148,
"lc_standard_error": 0.5566744587867744,
"num_examples": 805,
"completion_rate": 1.0
}
},
"config": {
"model": "hf",
"model_args": "pretrained=/hai/scratch/ahmedah/llama_3_tulusft_2eps/step-2573,tokenizer=meta-llama/Llama-3.1-8B-Instruct",
"batch_size": "4",
"batch_sizes": [],
"device": null,
"use_cache": null,
"limit": null,
"annotator_model": "auto",
"gen_kwargs": null,
"random_seed": 0,
"numpy_seed": 1234,
"torch_seed": 1234,
"fewshot_seed": 1234,
"model_num_parameters": 8030277632,
"model_dtype": "torch.float32",
"model_revision": "main",
"model_sha": ""
},