Uploading wheels first broken on macOS High Sierra (OR, wheels should always be uploaded first)

[jamadden]

It is important to upload .whl files to PyPI first (see pypa/twine#106 for motivation). Up until very recently, this always Just Worked for me. But now, using twine 1.9.1 and zest.releaser 6.12.4, I've been seeing the sdist get uploaded first. Example from this morning:

Upload to pypi (Y/n)?
Uploading zope.annotation-4.6.0.tar.gz
Uploading zope.annotation-4.6.0-py2.py3-none-any.whl

What changed? I'm now running macOS High Sierra on its new filesystem APFS.

twine ensures this happens when you use the command line and specify upload dist/*, but it looks like zest.releaser just does an os.listdir() and feeds the results of that, one filename at a time, to twine, so twine can't reorder the dists appropriately.

Why did this work before? Prior to APFS, the macOS filesystem was HFS+ which was internally organised as a BTree based on filenames. So os.listdir() would return results in sorted filename order, and the first differing character in the sdist and wheel filenames ('.' and '-') sort such that the wheel comes first:

In [1]: chars = ['.', '-']

In [2]: sorted(chars)
Out[2]: ['-', '.']

In [3]: filenames = ['zope.annotation-4.6.0.tar.gz', 'zope.annotation-4.6.0-py2.py3-none-any.whl']

In [4]: sorted(filenames)
Out[4]: ['zope.annotation-4.6.0-py2.py3-none-any.whl', 'zope.annotation-4.6.0.tar.gz']

Under APFS, filenames are returned apparently at random:

Calling readdir(2) on a directory in APFS returns filenames in hash order, whereas HFS+ returns filenames in lexicographical order.

This started as an unexplained support request trying to figure out what happened, but now it's turned into a feature request 😄 : can zest.releaser group wheels first, just like twine does? That would do the right thing on all filesystems (I know that most linux filesystems don't return filenames ordered).

[reinout]

Why is the order so important? I've read the twine link and that sounds like there might be a slight delay for showing the metadata when the tgz and whl are uploaded in a different order. I doubt that half a second delay makes any difference.

Or is there actually data missing when the tgz is uploaded first?

Of course a quick .sort() is easy to add, I'll do that :-)

[jamadden]

Or is there actually data missing when the tgz is uploaded first?

Indeed, there is missing data when the sdist is uploaded first. Compare the metadata section for zope.annotation where the sdist was uploaded first with that for zope.traversing where I uploaded the wheel first. You'll see that the entire "Requires Distribution" section is missing when the sdist was first.

This difference is also visible in the PyPI APIs.

There may be other metadata that's only extracted from wheels, but that's the most visible example.

[jamadden]

Of course a quick .sort() is easy to add, I'll do that

Thank you! twine needs a more complicated "grouping" solution because the command line can specify arbitrary files to upload, potentially from multiple projects. But here we know we are dealing with just one project and the filename standards should guarantee that sorting works out, so the simple solution should suffice. Awesome!

[reinout]

I've released 6.12.5 which ought to fix the problem.