Modernise Python setup for repo

[spwoodcock]

After #1904 is well tested and merged, I would happily contribute a few things:

• Upgrade from using standard pip --> uv. Running Python via venv is the recommended approach, regardless of container usage. See comment 24.04 #1904 (comment). uv would add much better dependency resolution and conflict checking. Users could run outside a container too, without trashing their system python install.
• Move most Python metadata to pyproject.toml, such as requirements.txt that would go under [dependencies].
• Update all links to requirements.txt to first export this via uv export, having the resolved pyproject.toml and uv.lock as a single source of truth.
• Update the dockerfiles to be a bit more efficient with caching / multi-stage builds, where possible.

[smathermather]

Thanks @spwoodcock -- a pull request would be most welcome! 24.04 is now merged and released.