Local plotting functions fail on simulation benchmarks

[jordandsullivan]

Repro:

1. Run simulation benchmarks locally: ucc-bench benchmarks/simulation_benchmarks.toml
2. Run plotting function: python plotting/plot_latest_benchmark.py .local_results Jordans-MacBook-Pro.local

Result:

(ucc-bench) jordansullivan@Jordans-MacBook-Pro ucc-bench % `python plotting/plot_latest_benchmark.py .local_results Jordans-MacBook-Pro.local`

Traceback (most recent call last):
  File "/Users/jordansullivan/UnitaryFund/ucc-bench/plotting/plot_latest_benchmark.py", line 190, in <module>
    main()
  File "/Users/jordansullivan/UnitaryFund/ucc-bench/plotting/plot_latest_benchmark.py", line 163, in main
    plot_compilation(df, latest_date, out_path, args.pdf)
  File "/Users/jordansullivan/UnitaryFund/ucc-bench/plotting/plot_latest_benchmark.py", line 85, in plot_compilation
    generate_plot(df, plot_configs, latest_date, out_path, use_pdf)
  File "/Users/jordansullivan/UnitaryFund/ucc-bench/plotting/plot_latest_benchmark.py", line 30, in generate_plot
    circuit_names = sorted(df["benchmark_id"].unique())
                           ~~^^^^^^^^^^^^^^^^
  File "/Users/jordansullivan/UnitaryFund/ucc-bench/.venv/lib/python3.12/site-packages/pandas/core/frame.py", line 4107, in __getitem__
    indexer = self.columns.get_loc(key)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jordansullivan/UnitaryFund/ucc-bench/.venv/lib/python3.12/site-packages/pandas/core/indexes/range.py", line 417, in get_loc
    raise KeyError(key)
KeyError: 'benchmark_id'

Workaround

The following code succeeds with the --plot simulation argument provided:
python plotting/plot_latest_benchmark.py --plot simulation .local_results Jordans-MacBook-Pro.local

[bachase]

@jordandsullivan I was not able to reproduce but tracing the error sounds like the results file might've been empty. But its not clear how the simulation plot still worked then.

Could you share the .json file from the benchmark run? That would be in .local_results/Jordans-MacBook-Pro.local/simulation_benchmarks/YYYYMMDD/YYYYMMDDHHMMSS.random_uid.json.

[jordandsullivan]

Here's the file: 20250822104704.3e8c865b-a728-4aa4-8d3f-01da4b983754.json

and the filepath:
/Users/jordansullivan/UnitaryFoundation/ucc-bench/.local_results/Jordans-MacBook-Pro.local/simulation_benchmarks/20250822/20250822104704.3e8c865b-a728-4aa4-8d3f-01da4b983754.json

[bachase]

New theory

If you run without the --plot simulation argument, the script tries to plot both compilation and simulation results. The latter is in the .local_results/Jordans-MacBook-Pro.local/simulation_benchmarks directory, but the former pulls from .local_results/Jordans-MacBook-Pro.local/compilation_benchmarks. That is, this plot is usually run in CI and it assumes by default you've most recently run all the benchmarks.

Here, is it possible you've only most recently been running the simulation_benchmarks? So when it goes to load the compilation_benchmarks the directory is either empty, or maybe it's an older format? Testing the empty case locally does crash the script, but with a different error (Which I will fix). But can you check and share JSON for the most recent run you have of the compilation_benchmarks?

[jordandsullivan]

Indeed I was only running the simulation benchmarks. The latest compilation benchmarks are not from the same run.

20250829101612.ec2b82a8-33c9-4931-91e5-cc3a29591ec2.json

[bachase]

I wasn't able to recreate this even with the update json file on my end. Might need to do a screenshare to get to the bottom of it.

[jordandsullivan]

Huh, fine to deprioritize for how then. I'll just see if I run into the issue again in the future