Name	Name	Last commit message	Last commit date
parent directory ..
README.md	README.md
aup_utils.py	aup_utils.py
data_dream.yaml	data_dream.yaml
data_dream_coder.yaml	data_dream_coder.yaml
data_llada.yaml	data_llada.yaml
main.py	main.py
plot_aup_illustration.py	plot_aup_illustration.py
plot_histogram.py	plot_histogram.py
plot_lines.py	plot_lines.py
plot_radar.py	plot_radar.py
plot_speed_accuracy.py	plot_speed_accuracy.py

Name

Last commit message

Last commit date

aup_utils.py

data_dream.yaml

data_dream_coder.yaml

data_llada.yaml

main.py

plot_aup_illustration.py

plot_histogram.py

plot_lines.py

plot_radar.py

plot_speed_accuracy.py

🏆 Diffusion LLM Leaderboard using AUP as the Metric

📊 Introducing a New Metric: AUP

Traditional throughput metrics (tokens per second) are hardware-dependent, making fair comparisons difficult. We introduce AUP (Accuracy Under Parallelism), a hardware-independent metric that jointly measures efficiency and performance.

AUP captures both parallelism (tokens per forward pass) and accuracy, with a weighting function that penalizes accuracy degradation

Key insight: AUP uses tokens per forward (TPF) instead of tokens per second (TPS), making it device-independent. A higher AUP score means the model maintains accuracy while achieving high parallelism.