Tools for a sources mirror

[antonlacon]

This adds tools for building a sources mirror. Its broken into three parts:

genbuildplan.py: modified to add command line arguments for use by other scripts

genmirrorlist.py: new script to output a list of $PKG_NAME and $PKG_URL, one per line, of the requested images to be built. May be invoked by setting PROJECT/DEVICE/ARCH, invoking --all, or invoking --all with --builddirs. --all generates a list for all images (takes 20-30 minutes to put together the list - if the uboot based images don't vary in their package needs, this can be reduced significantly). --builddirs reduces --all to only those images with a build directory present. PROJECT/DEVICE/ARCH is for that specific image. Script outputs to stdout or to a file with the --export option.

build-mirror: standalone shell script that will download the output of genmirrorlist.py into the specified output directory with $PKG_NAME subdirectories. This is separated from genmirrorlist.py because genmirrorlist.py needs to run in the git source tree, and this doesn't. Add -v to show download progress.

Examples:

PROJECT=RPi DEVICE=RPi5 ARCH=aarch64 tools/genmirrorlist.py | tools/build-mirror -o mirror/

tools/genmirrorlist.py --all --export package-list-20241001.txt && tools/build-mirror -i package-list-20241001.txt -o mirror/

[vpeter4]

What about this two?
#6270
#8279

[antonlacon]

6270 works if you're only interested in downloading sources for one device. If this merges, I may extend build-mirror to use parallel for multiple downloads and then close it. I'm not expecting 6270 to merge.

8279 is complimentary. It shares some logic in figuring out what files are needed, but it's meant for pruning a builder's sources directory, instead of populating a mirror. I don't remember why it's marked draft, but I'll revisit it soon. Long term it might make sense to merge it and genmirrorlist.py together, but I'm not in a rush and finding reviewers for python is already tricky.

[antonlacon]

Updated to remove devices that no longer receive test builds, use of DISTRO_VERSION, and to make genmirrorlist.py multithreaded. Reduces time to figure out needed packages when running for all project/device/arch combinations from ~40 minutes to ~8 minutes on 16 cores.