Distribute docs build over number of CPUs.

[cosenal]

Description

An attempt to speed up docs build (related to #2083, but not the same as that's about RTD).
It looks like the bottleneck is the notebook execution. MyST-NB is parallel-friendly, hence this attempt.

Some data points from testing this on my local machine (Apple M2 Pro):

Build	time make docs-clean
Serial	962.92s user 424.57s system 193% cpu 11:57.48 total
Parallel	1052.97s user 250.09s system 443% cpu 4:53.88 total

---

License

• I license this contribution under the terms of the GNU GPL, version 3 and grant Unitary Fund the right to provide additional permissions as described in section 7 of the GNU GPL, version 3.

Before opening the PR, please ensure you have completed the following where appropriate.

N/A - [ ] I added unit tests for new code.
N/A - [ ] I used type hints in function signatures.
N/A - [ ] I used Google-style docstrings for functions.
N/A - [ ] I updated the documentation where relevant.

• Added myself / the copyright holder to the AUTHORS file

[github-actions]

Hello @cosenal, thank you for submitting a PR to Mitiq! We will respond as soon as possible, and if you have any questions in the meantime, you can ask us on the Unitary Fund Discord.

[nathanshammah]

Nice! It's failing here, not sure why.

[cosenal]

@nathanshammah Yes, I cancelled the workflow, because I saw it failing on the docs build, which I am investigating. I am leaving it as a Draft PR until the build is fixed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.13%. Comparing base (02bc290) to head (e8c3969).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2235   +/-   ##
=======================================
  Coverage   98.13%   98.13%           
=======================================
  Files         103      103           
  Lines        4730     4730           
=======================================
  Hits         4642     4642           
  Misses         88       88           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[nathanshammah]

Is this blocked by the PyData Sphinx theme action?

[cosenal]

@nathanshammah indeed – I have just dropped a comment there

[cosenal]

Since v0.15.4, the Sphinx theme enabled parallel write. The next blocking factor is Jupyter-cache not supporting parallel execution, see executablebooks/jupyter-cache#37 and sphinx-doc/sphinx#12481.

[natestemen]

@cosenal I'm confused by that ticket on jupyter-cache given they do support two parallel modes according to https://jupyter-cache.readthedocs.io/en/latest/using/cli.html#executing-the-notebooks. Maybe local-parallel/temp-parallel aren't what we need?

[cosenal]

@cosenal I'm confused by that ticket on jupyter-cache given they do support two parallel modes according to https://jupyter-cache.readthedocs.io/en/latest/using/cli.html#executing-the-notebooks. Maybe local-parallel/temp-parallel aren't what we need?

I am also confused by why the builds are passing now, and nothing has changed in jupyter-cache since 🤔 Maybe just a fortunate race condition.

[natestemen]

In order to get the build to work locally I (read Claude) had to make some alterations to avoid a race condition. See 7970a86.

Once the issue was fixed I was able to run make docs-clean on main and this branch. The results:

Build	time make docs-clean
Serial	1731.05s user 394.69s system 152% cpu 23:11.25 total
Parallel	1769.04s user 358.97s system 383% cpu 9:15.05 total

[cosenal]

I triggered a docs-build manually on this branch: https://github.com/unitaryfoundation/mitiq/actions/runs/23434142191 (I had to merge main into this for it to work)

[natestemen]

1. were you able to build the docs locally on this branch before my addition?
2. are you able to build docs locally on this branch?

[cosenal]

1. were you able to build the docs locally on this branch before my addition?

yes, no error. But that's the unpredictability of race conditions

2. are you able to build docs locally on this branch?

running a make install && make docs-clean now

[cosenal]

Locally it failed with

Sphinx parallel build error:
KeyError: 'Notebook file does not exist for cache record PK: 2'
[IPKernelApp] ERROR | Exception in message handler:
Traceback (most recent call last):
  File "/Users/cosenal/Coding/unitaryfoundation/mitiq/.venv/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 478, in dispatch_shell
    await result
  File "/Users/cosenal/Coding/unitaryfoundation/mitiq/.venv/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 372, in execute_request
    await super().execute_request(stream, ident, parent)
  File "/Users/cosenal/Coding/unitaryfoundation/mitiq/.venv/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 857, in execute_request
    reply_msg: dict[str, t.Any] = self.session.send(  # type:ignore[assignment]
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cosenal/Coding/unitaryfoundation/mitiq/.venv/lib/python3.12/site-packages/jupyter_client/session.py", line 867, in send
    stream.send_multipart(to_send, copy=copy)
  File "/Users/cosenal/Coding/unitaryfoundation/mitiq/.venv/lib/python3.12/site-packages/zmq/sugar/socket.py", line 749, in send_multipart
    self.send(msg, zmq.SNDMORE | flags, copy=copy, track=track)
  File "/Users/cosenal/Coding/unitaryfoundation/mitiq/.venv/lib/python3.12/site-packages/zmq/sugar/socket.py", line 698, in send
    return super().send(data, flags=flags, copy=copy, track=track)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "zmq/backend/cython/_zmq.py", line 1152, in zmq.backend.cython._zmq.Socket.send
    def send(self, data, flags=0, copy: bint = True, track: bint = False):
    ^^^^^^^^^^^
  File "zmq/backend/cython/_zmq.py", line 1194, in zmq.backend.cython._zmq.Socket.send
    _check_closed(self)
    ^^^^^^^^^^^
  File "zmq/backend/cython/_zmq.py", line 1339, in zmq.backend.cython._zmq._check_closed
    raise ZMQError(ENOTSOCK)
    ^^^^^^^^^^^

[cosenal]

btw, this shouldn't block any release, it's just a nice to have, especially now that we are not running docs-build at every PR.

[cosenal]

for future reference, the docs-build on the GitHub worker succeeded ✅