Experiment: Tootsie Phoenix Cooldown (sensible-starling)

[dlwh]

Description

Code-Name: sensible-starling

It's time to cool down Tootsie Phoenix. Based on #847 we're going to use our datasets as well as nemotron and dolmino.

I'm going to throw in zloss and possibly megamath, depending.

Hypothesis or Goal

MMLU to the moon (and everything else)

Links

(Delete any that aren't applicable)

• WandB Report: https://wandb.ai/marin-community/marin/reports/Tootsie-8B-phoenix-cooldown-starling---VmlldzoxMjQ2MjM5Ng
• Run: https://wandb.ai/marin-community/marin/runs/tootsie-8b-sensible-starling?nw=nwuserdlwh
• Data Browser: (link)

Results

(What did you find, including relevant evaluation metrics, etc.)