Testing universal2 wheel on arm64 macOS cp38 runner result in `not a supported wheel on this platform`

[laggykiller]

Description

Currently, to test a wheel, wheels are first installed with pip. On arm64 macOS github runner (macos-14), wheels are installed with arch -x86_64 pip install .... This results in ERROR: xxx_universal2.whl is not a supported wheel on this platform..

On my local M1 macmini, I am able to recreate the issue by:

1. Install python3.8.10 x86_64 version from (Note that x86_64 version is used in python3.8 even for arm64 macOS runner according to Fix cp38-macosx universal2 and arm64 Python pkg URL #1169): https://www.python.org/ftp/python/3.8.10/python-3.8.10-macosx10.9.pkg
2. python3.8 -m pip install apngasm_python-1.2.3-cp311-cp311-macosx_11_0_universal2.whl.

If I rename the problematic wheel name to -macosx_10_9_universal2.whl, the problem does not occur. This does not make sense as we are testing on macos-14, which is much newer than macos 10.9 or 11.0.

This problem does not occur for universal2 python versions (i.e. python >3.8).

Relevant issue: #1746

You can download sample universal2 wheels from artifact of this run (https://github.com/laggykiller/apngasm-python/actions/runs/8063189658), or directly down here:

apngasm_python-1.3.0-cp38-cp38-macosx_11_0_universal2.whl.zip
apngasm_python-1.3.0-cp39-cp39-macosx_11_0_universal2.whl.zip
apngasm_python-1.3.0-cp310-cp310-macosx_11_0_universal2.whl.zip
apngasm_python-1.3.0-cp311-cp311-macosx_11_0_universal2.whl.zip
apngasm_python-1.3.0-cp312-abi3-macosx_11_0_universal2.whl.zip

(.zip was added to the end of whl file name end so that github allow me to upload it. Just remove .zip extension from the file name after downloading.)

Build log

https://github.com/laggykiller/apngasm-python/actions/runs/8061744757/job/22020065912#step:4:1347

EDIT: I have tried with cibuildwheel master branch, and the problem still exists: https://github.com/laggykiller/apngasm-python/actions/runs/8074988563/job/22061164760#step:4:1419

CI config

https://github.com/laggykiller/apngasm-python/blob/c046caa64b633fe1cc957f67fd4fbc0f6abd7b40/.github/workflows/build.yml

[laggykiller]

As I investigate further, it seems like the problem is caused by x86_64 (non-universal2) version of python 3.8 running on arm64 macOS, which has bug in checking wheel tag such that *-macosx_11_universal2.whl is misrecognized as invalid.

This seems to be caused by the same issue as #1603 and I can confirm that adding SYSTEM_VERSION_COMPAT=0 would allow the wheel to be installed

Should we add SYSTEM_VERSION_COMPAT=0 for the user in cibuildwheel during pip install built wheel? I can submit a PR if this is a good idea. Otherwise let's just close this issue.

Other hacks, workarounds:

• Use universal2 python3.8, but as discussed in Fix cp38-macosx universal2 and arm64 Python pkg URL #1169 it would increase the minimum MacOS version for the x86_64 arch from 10.9 to 11.0, so this is probably no go.
• Install universal2 python3.8 in CI ourselves as suggested in Fix cp38-macosx universal2 and arm64 Python pkg URL #1169 (comment)
• Change deployment target of wheel to <11.0 instead of >=11.0
• Skip testing cp38 universal2 wheel

[joerick]

Nice work debugging this. So you'll be hitting this error due to these lines in your config:

https://github.com/laggykiller/apngasm-python/blob/c046caa64b633fe1cc957f67fd4fbc0f6abd7b40/.github/workflows/build.yml#L75-L76

Is there a specific reason you set MACOSX_DEPLOYMENT_TARGET to 11.0 on this build? It's 10.15 on the x86_64 one. macOS 11.0 is the first to support arm64, so it's implicit that you'd need 11.0 or later on arm64.

(Also, just FYI, unless there's something going on here that I've missed, you shouldn't need to set _PYTHON_HOST_PLATFORM yourself, cibuildwheel will do that for you.)

As for the broader issue, I think you're right, we should set SYSTEM_VERSION_COMPAT=0 when testing universal2 Python 3.8 wheels on x86_64. Otherwise, as you have discovered, it's not possible to set MACOSX_DEPLOYMENT_TARGET to anything greater than 10.16. I don't think that Python 3.9+ would need this though, because they'll be built with a newer SDK, so macOS won't do any version-compat shimming. I see that you only had the issue with 3.8, so I guess that's correct.

[laggykiller]

Is there a specific reason you set MACOSX_DEPLOYMENT_TARGET to 11.0 on this build?

Fortunately no, so I can just set it back to 10.15 to avoid this problem. I originally thought that the targets should match for both the arm64 and x86_64 part of universal2 wheel. Turns out this is not necessary.

you shouldn't need to set _PYTHON_HOST_PLATFORM yourself

This is because I need to set x86_64 deployment target to 10.15, as the C++ code I am compiling requires it. The default 10.9 won't work for me.

we should set SYSTEM_VERSION_COMPAT=0 when testing universal2 Python 3.8 wheels on x86_64.

Opened a PR for this: #1768

[joerick]

For reference, this is really a bug in pip that we're working around, ref pypa/pip#11715

[joerick]

To be clear, this is a dupe of #1603 , i'm just more inclined to do something about it given that a fix in pip doesn't appear to be imminent...

[bdarnell]

My builds are producing warnings that link to this issue. (sample build) Is any action required? (If not, is the log message necessary?) Does this mean that users of my package will have to use SYSTEM_VERSION_COMPAT=0 for py38 on macos?

Note that I am not setting MACOS_DEPLOYMENT_TARGET unless cibuildwheel is doing it for me somewhere.