macOS arm64 requires numpy 1.20.0 or above by kbourgoin · Pull Request #20 · scipy/oldest-supported-numpy

kbourgoin · 2021-05-27T15:02:46Z

Fixes #19

rgommers

Thanks @kbourgoin. This makes sense to add. I'm wondering if this also needs

platform_system=='darwin'

because there may be Windows arm64 machines.

I can't find any place that authoritatively defines platform_machine, so hard to be sure.

rgommers · 2021-05-27T19:22:18Z

    numpy==1.20.0; python_version=='3.7' and platform_python_implementation=='PyPy'

+    # arm64 supports Python 3.8 and above requires numpy>=1.20.0
+    numpy==1.20.0; python_version=='3.8' and platform_machine=='arm64'


We should add a note to this: while Python 3.8 on arm64 works, there is no way to cross-compile from an x86_64 machine to arm64 - that's only possible for Python >= 3.9. Hence effectively it's almost impossible to provide 3.8 wheels, because projects do that on CI - and there's no macOS M1 CI.

We should add a note to this: while Python 3.8 on arm64 works, there is no way to cross-compile from an x86_64 machine to arm64 - that's only possible for Python >= 3.9

Why not?

You need a machine with macOS 11, as the final CPython 3.8 binary release does have an experimental Universal2 build, it does not work on macOS 10.9+ like the 3.9 releases. I think once macOS 11 is readily available on CI, it might be easier to compile 3.8 universal or cross-compiled arm64 binaries.

You can also use the Xcode python 3.8, which is universal, and supports 10.15+, but that will only work as long as Xcode doesn't replace it with 3.9 in the future.

(yes, "there is no way to cross-compile" is not true, it's a bit hard due to the issues listed above, but not impossible)

Thanks for the info. We could use the regular python 3.8 build. Universal build is not necessary to build wheels.

Works just fine for numpy. See MacPython/numpy-wheels#115

I thought the resolution here was that CPython 3.8 universal was built without MACOS_DEPLOYMENT_TARGET and as such wouldn't work. Looks like that's a macOS 10.15 runner, though, for MacPython? Ah, you are building universal2 from python-3.8.10-macosx10.9.pkg. I didn't realize you could build a universal2 build from a non-universal Python. Is there a way I can download the wheel artifact and try it out on my M1?

(I really wish I'd left a machine on 10.15 for tests like this...)

Is there a way I can download the wheel artifact and try it out on my M1?

If that PR is merged, then yes. cc @mattip

kbourgoin · 2021-05-28T02:31:41Z

@rgommers as far as I could find while looking, arm64 is specific to Apple's implementation of 64-bit ARM. Otherwise, it's called aarch64 by platform. It's not currently possible to run Windows on an M1, but that may not always be true.

I could add it to lines 48/49, but then it makes 42/43 much more complex. Is it worth it to makes those lines harder to read? If so, I can try to make it work.

rgommers · 2021-05-28T07:32:01Z

Otherwise, it's called aarch64 by platform. It's not currently possible to run Windows on an M1, but that may not always be true.

The Windows case I hadn't even thought about - I was more thinking about about other custom chips. I don't think they're guaranteed to all be named aarch64.

I could add it to lines 48/49, but then it makes 42/43 much more complex. Is it worth it to makes those lines harder to read? If so, I can try to make it work.

How about this, move 48/49 above the block of lines for default numpy requirements, which is a good idea anyway. And then make 42/43 more specific, but leave 48/49 as is.

rgommers · 2021-05-28T13:57:05Z

Just a note here on universal2 as well to not forget: I suspect it needs to be treated as a separate architecture. Cibuildwheel lists this in its matrix:

cp39-macosx_x86_64
cp39-macosx_universal2
cp39-macosx_arm64

That of course is very weird, since there's no such a hardware architecture, but it is what's done at https://github.com/pypa/cibuildwheel/pull/484/files.

If we build a universal2 wheel, we do want both binaries to be built against the same version of numpy, and not treat x86_64 and arm64 requirements separately.

The multibuild setup is different though, see explanation at https://github.com/matthew-brett/multibuild/pull/383/files. So there we don't need universal2. I think it's the cleaner way of doing it, but on the other hand I'm not entirely happy that we then get to pull in two numpy versions to build against for a single wheel.

Cc @joerick and @isuruf, could you comment on whether the pyproject.toml changes in this are sufficient for cibuildwheel and multibuild respectively?

kbourgoin · 2021-05-28T14:38:16Z

Alright, gave it an update. Let me know if this works. It works on my machine, but in a strange quirk of life, I don't have anything but an M1 to test this on right now.

joerick · 2021-05-28T14:42:30Z

Hmm. Tricky... when cibuildwheel builds arm64 and universal2 wheels, we're cross compiling from x64_64 by setting _PYTHON_HOST_PLATFORM and ARCHFLAGS during the build. So I don't think that platform_machine will be set to arm64 in that case (AFAIK it's just platform.machine()). Are you able to specify using anything more expressive than the pip environment markers?

I'll also cc @henryiii who might have some ideas!

henryiii · 2021-05-28T19:43:57Z

@rgommers I don't actually see what's different in Multibuild vs. cibuildwheel. In both systems, you set "universal2", in both systems you get -arch x86_64 -arch arm64 when building, and in both systems, you get a universal2 wheel tag. In cibuildwheel, you have a build selector, but that's just an indicator for what sort of wheel you want to build.

I'm building boost-histogram Python 3.8 wheels with a (patched) cibuildwheel using the x86_64 Python, and you are correct; I was able to get an arm64 wheel that works on my M1. (Universal2 actually produced x86_64 tagged wheels, not sure why - the size is correct for the dual arch's, though)

henryiii · 2021-05-28T21:55:41Z

@rgommers Sorry, now I see "what's different" in the new docs in that PR - multibuild does option 2 and cibuildwheel does option 1. Might be something to look at in the future as an option.

rgommers · 2021-05-29T18:04:33Z

Alright, gave it an update. Let me know if this works. It works on my machine, but in a strange quirk of life, I don't have anything but an M1 to test this on right now.

Thanks @kbourgoin, I think that looks good. I pushed a small change to also add the =='Darwin' part to the macOS M1 case.

It looks like the arm64 case is indeed unique right now, but it's not guaranteed - the CPython repo does have code using arm64 for Windows Azure config and a win-amd64 platform tag, so better safe than sorry.

I do have a non-M1 machine, will do a quick test now.

Changes taken over from scipygh-14145 and from scipy/oldest-supported-numpy#20

rgommers · 2021-06-06T15:32:48Z

In it goes. Thanks @kbourgoin

This is the first NumPy version with universal2 wheels, and 1.21.0 was just released so we can do the version bump. This is a follow-up to scipygh-19 and scipygh-20

arm64 requires 1.20.0 or above

c1786e7

rgommers reviewed May 27, 2021

View reviewed changes

Make this more specific and rearrange things.

c239b6c

henryiii mentioned this pull request May 28, 2021

feat: support building universal/arm64 from x64_86 Python 3.8 pypa/cibuildwheel#704

Merged

Add platform_system=='Darwin' to the macOS M1 cases

7846558

rgommers added a commit to rgommers/scipy that referenced this pull request May 29, 2021

BLD: update pyproject.toml - add macOS M1, drop py36

a74e827

Changes taken over from scipygh-14145 and from scipy/oldest-supported-numpy#20

rgommers mentioned this pull request May 29, 2021

BLD: update pyproject.toml - add macOS M1, drop py36 scipy/scipy#14151

Merged

rgommers added a commit to rgommers/scipy that referenced this pull request Jun 2, 2021

BLD: update pyproject.toml - add macOS M1, drop py36

0bd0706

Changes taken over from scipygh-14145 and from scipy/oldest-supported-numpy#20

rgommers merged commit 37a7530 into scipy:master Jun 6, 2021

rgommers changed the title ~~arm64 requires 1.20.0 or above~~ macOS arm64 requires numpy 1.20.0 or above Jun 6, 2021

kbourgoin deleted the fix-arm64 branch June 7, 2021 18:19

rgommers mentioned this pull request Jun 25, 2021

Update macOS ARM64 requirement to 1.21.0 #22

Merged

ljlamarche mentioned this pull request Mar 15, 2022

BUG: Bus error: 10 aburrell/apexpy#84

Closed

henryiii mentioned this pull request Jul 8, 2022

Fix cp38-macosx universal2 and arm64 Python pkg URL pypa/cibuildwheel#1169

Closed

Uh oh!

Conversation

kbourgoin commented May 27, 2021

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

henryiii May 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kbourgoin commented May 28, 2021

Uh oh!

rgommers commented May 28, 2021

Uh oh!

rgommers commented May 28, 2021

Uh oh!

kbourgoin commented May 28, 2021

Uh oh!

joerick commented May 28, 2021

Uh oh!

henryiii commented May 28, 2021

Uh oh!

henryiii commented May 28, 2021

Uh oh!

rgommers commented May 29, 2021

Uh oh!

rgommers commented Jun 6, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

henryiii May 28, 2021 •

edited

Loading