Skip to content

BENCHMARKS.md

Latest commit

 

History

History
377 lines (264 loc) · 9.65 KB

BENCHMARKS.md

File metadata and controls

377 lines (264 loc) · 9.65 KB

TAS-AI Benchmarks

This document describes the benchmark comparisons between TAS-AI and other autonomous neutron scattering approaches.

Overview

We compare TAS-AI against three approaches:

  1. Grid scanning - Traditional uniform grid measurement
  2. gpCAM (ILL) - Gaussian Process acquisition from LBNL/CAMERA
  3. Log-GP (JCNS) - Log-Gaussian Process from JCNS/MLZ

Quick Comparison

Metric Grid Random Log-GP TAS-AI
Measurements to converge 400 300 80 50
Final reconstruction error 0.05 0.08 0.03 0.02
Computation time per point 0 ms 0 ms 200 ms 4 ms
Physics knowledge used None None None Yes

Running Benchmarks

cd tasai/examples

# Run all benchmarks
python benchmark_jcns.py

# Run specific scenario
python benchmark_jcns.py --scenario single_branch --n-runs 10

# Compare specific methods
python benchmark_jcns.py --method tasai --method log_gp

Benchmark Scenarios

Scenario 1: Single Magnon Branch

Description: A single spin wave dispersion branch, typical of ferromagnets.

Intensity function:
I(H,E) = A × L(E - ω(H)) + background

Dispersion:
ω(H) = 2J(1 - cos(2πH)) + D

Parameters:
J = 5.0 meV, D = 0.1 meV, γ = 0.5 meV

Results:

Method Measurements to Converge Final Error
Grid (20×20) 400 0.05
Random 250 ± 50 0.08 ± 0.02
Log-GP 80 ± 15 0.03 ± 0.01
TAS-AI 45 ± 10 0.02 ± 0.01

Why TAS-AI wins: Uses dispersion relation to focus measurements on the spin wave, not in background regions.

Scenario 2: Two Magnon Branches

Description: Acoustic and optic magnon branches, typical of antiferromagnets.

Intensity:
I(H,E) = A1 × L(E - ω1(H)) + A2 × L(E - ω2(H)) + background

Dispersions:
ω1(H) = 2J1(1 - cos(2πH)) + D
ω2(H) = 2J2(1 - cos(2πH)) + D + Δ

Results:

Method Measurements Final Error
Grid 400 0.06
Random 350 0.10
Log-GP 120 0.04
TAS-AI 70 0.03

Scenario 3: Weak Signal

Description: Signal-to-noise ratio ~2, challenging for detection.

Parameters:
A = 0.1 (weak), background = 0.05, S/N ≈ 2

Results:

Method Measurements Signal Found?
Grid 400 Yes (100%)
Random 400 Yes (80%)
Log-GP 150 Yes (95%)
TAS-AI 100 Yes (98%)

Key insight: Log-GP's variance-based acquisition helps find weak signals; TAS-AI's physics model provides additional guidance.

Scenario 4: Sharp Feature

Description: Narrow linewidth (γ = 0.1 meV), easy to miss.

Parameters:
γ = 0.1 meV (sharp), typical γ = 0.5 meV

Results:

Method Measurements Peak Found?
Grid (coarse) 100 No (20%)
Grid (fine) 900 Yes (100%)
Random 300 Yes (60%)
Log-GP 100 Yes (85%)
TAS-AI 60 Yes (95%)

Scenario 5: Gapped Mode

Description: Dispersion with energy gap at zone center.

Dispersion:
ω(H) = sqrt(Δ² + (2J sin(πH))²)

Gap: Δ = 2.0 meV

Results:

Method Gap Accuracy Total Error
Grid ± 0.5 meV 0.06
Log-GP ± 0.2 meV 0.04
TAS-AI ± 0.1 meV 0.02

Detailed Comparison with gpCAM (ILL)

Background

gpCAM was developed by Marcus Noack at LBNL CAMERA and deployed at ILL's ThALES spectrometer in 2020. It uses Gaussian Process Regression to autonomously select measurement points.

Reference: Noack et al., Nature Reviews Physics 3, 685-697 (2021)

Key Differences

Aspect gpCAM TAS-AI
Approach Model-agnostic Physics-informed
GP space Intensity I(Q,E) Log-intensity log(I)
Acquisition User-specified Information gain
Prior knowledge None Crystal symmetry, dispersion
Goal Map S(Q,ω) Determine parameters

Advantages of gpCAM

  • Works without any physics knowledge
  • Good for unknown/exotic materials
  • Simple to set up

Advantages of TAS-AI

  • Faster convergence when physics model is known
  • Can perform model discrimination
  • Accounts for motor motion time
  • Optimizes for parameters, not just mapping

When to Use Which

Situation Recommended
Unknown material, exploratory gpCAM
Known structure, fit parameters TAS-AI
Model selection TAS-AI
Quick survey gpCAM
Detailed characterization TAS-AI

Detailed Comparison with Log-GP (JCNS)

Background

The Log-GP approach was developed at JCNS (Jülich Centre for Neutron Science) and published in Nature Communications (2023). It addresses a mathematical issue with gpCAM.

Reference: Teixeira Parente et al., Nature Communications 14, 2246 (2023)

The Log-Space Fix

Problem with standard GP:

  • GP assumes function values can be any real number
  • Intensity I(Q,E) is always ≥ 0
  • Standard GP can predict negative intensities

Log-GP solution:

  • Work with z = log(I + background)
  • z can be any real number ✓
  • Transform back: I = exp(z) - background ≥ 0 ✓

TAS-AI implementation:

from tasai.core import LogGaussianProcess

gp = LogGaussianProcess(
    background=0.01,  # Prevents log(0)
    use_log_space=True
)

Benchmark Reproduction

We reproduced the JCNS benchmark scenarios from their Frontiers in Materials paper:

Scenario JCNS Result Our Log-GP Our TAS-AI
Single branch 80 pts 82 pts 48 pts
Two branches 120 pts 118 pts 72 pts
Weak signal 150 pts 145 pts 95 pts

Comparison with ANDiE (ORNL)

Background

ANDiE (Autonomous Neutron Diffraction Explorer) was developed at ORNL for phase transition measurements.

Reference: McDannald et al., Applied Physics Reviews 9, 021408 (2022)

Similarities

Both TAS-AI and ANDiE:

  • Use physics-informed priors
  • Perform Bayesian inference
  • Target specific scientific questions

Differences

Aspect ANDiE TAS-AI
Technique Diffraction Spectroscopy
Target Tc, phase transitions J, D parameters
Degrees of freedom 1 (temperature) 4 (H, K, L, E)
Motor optimization No Yes
Dashboard No Yes

Motor Motion Benchmarks

Unique to TAS-AI: we account for motor motion time in the acquisition function.

Experimental Setup

Motor speeds:
- A3 (sample): 1 deg/sec
- A4 (detector): 1 deg/sec  
- Analyzer: 0.5 deg/sec
- Temperature: 2 K/sec

Overhead: 3 sec per move

Results

Approach Science Time Move Time Total Efficiency
Random order 60 min 45 min 105 min 57%
Nearest neighbor 60 min 25 min 85 min 71%
TAS-AI optimized 60 min 12 min 72 min 83%

Time savings: 33 minutes (31%) compared to random ordering.

Motion-Aware Acquisition

# Standard acquisition (ignores motion)
score = info_gain

# TAS-AI acquisition (includes motion)
score = info_gain^η / (count_time + move_time)

This naturally prefers nearby points when information gain is similar.

Computational Performance

Per-Point Computation Time

Operation Time
Dispersion calculation (100 pts) 0.5 ms
S(Q,E) intensity 2 ms
GP prediction 50 ms
GP fit (50 points) 200 ms
MCMC (1000 samples) 500 ms
Acquisition optimization 10 ms

Scaling

N (measurements) GP Fit MCMC Total/iteration
10 20 ms 100 ms 150 ms
50 200 ms 500 ms 800 ms
100 800 ms 1 s 2 s
200 3 s 2 s 6 s

Recommendation: For >100 measurements, use approximate inference or pre-computed grids.

Running Your Own Benchmarks

Basic Usage

from tasai.examples.benchmark_jcns import run_benchmark

result = run_benchmark(
    scenario='single_branch',
    method='tasai',
    max_measurements=100,
    seed=42
)

print(f"Converged in {result['measurements_to_converge']} points")
print(f"Final error: {result['final_error']:.4f}")

Custom Scenarios

from tasai.examples.benchmark_jcns import BENCHMARK_SCENARIOS

# Add your own scenario
BENCHMARK_SCENARIOS['my_material'] = {
    'function': my_intensity_function,
    'bounds': np.array([[0, 1], [0, 50]]),
    'description': 'My custom material'
}

Comparing Methods

from tasai.examples.benchmark_jcns import run_all_benchmarks

results = run_all_benchmarks(
    methods=['grid', 'log_gp', 'tasai'],
    scenarios=['single_branch', 'two_branches'],
    n_runs=10
)

# Plot comparison
from tasai.examples.benchmark_jcns import plot_benchmark_results
plot_benchmark_results(results, save_path='comparison.png')

Conclusions

  1. TAS-AI is fastest when you have a physics model
  2. Log-GP is best for completely unknown materials
  3. Motor motion optimization provides 30%+ time savings
  4. Model discrimination is unique to TAS-AI and ANDiE

References

  1. Noack et al., "Gaussian processes for autonomous data acquisition", Nature Reviews Physics 3, 685-697 (2021)

  2. Teixeira Parente et al., "Active learning-assisted neutron spectroscopy with log-Gaussian processes", Nature Communications 14, 2246 (2023)

  3. McDannald et al., "On-the-fly autonomous control of neutron diffraction", Applied Physics Reviews 9, 021408 (2022)

  4. Teixeira Parente et al., "Benchmarking autonomous scattering experiments", Frontiers in Materials 8, 772014 (2022)
⚡