Notebook tests are slow

[mpharrigan]

The notebooks test is the slowest CI check with the following 10 slowest notebooks

Slowest 10 notebooks:
210.25s /qualtran/bloqs/phase_estimation/phase_estimation_of_quantum_walk.ipynb
62.87s /qualtran/bloqs/hamiltonian_simulation/hamiltonian_simulation_by_gqsp.ipynb
41.47s /qualtran/bloqs/rotations/quantum_variable_rotation.ipynb
26.64s /qualtran/bloqs/optimization/k_xor_sat/kikuchi_guiding_state_tutorial.ipynb
22.56s /qualtran/bloqs/arithmetic/error_analysis_for_fxp_arithmetic.ipynb
18.72s /qualtran/bloqs/phase_estimation/kaiser_window_state.ipynb
18.21s /qualtran/bloqs/arithmetic/comparison.ipynb
17.09s /qualtran/cirq_interop/cirq_interop.ipynb
16.96s /qualtran/bloqs/multiplexers/apply_gate_to_lth_target.ipynb
16.90s /qualtran/bloqs/chemistry/resource_estimation.ipynb

I think we may be hitting a timeout sometimes as well.

[mpharrigan]

I think there was a performance regression introduced in #1732

[mpharrigan]

running hamiltonian_simulation_by_gqsp is nearly instantaneous from the juypter interface but takes tens of second from the executor script on my machine

[mpharrigan]

Another fundamental problem is that if it takes a minimum of 5 seconds per notebook (which seems to be the case: spinning up a kernel, doing imports) .. we have 151 notebooks now. That's already a minimum of 12 notebooks (single threaded). In reality, the script will run in parallel but there's only so much parallelization afforded by the github runners. Also, 5 seconds seems to be the absolute minimum. Most seem to take around 10 seconds with some taking more.

[fdmalone]

would jupytext help? (rewrite as decorated python scripts and test these which pay be faster, and generate the notebooks from these)