What metadata/installability checks should Warehouse check uploads for?

[msabramo]

Recently, a project that I worked on pushed a new version to PyPI and it turned out to be completely broken, because the setup.py was referencing a README.rst file that was not present in the sdist.

It would be awesome if PyPI could so checking of packages that are uploaded.

To start with, it could create a virtualenv and try pip installing the package and make sure that exits with status 0. Perhaps it could also try easy_install to make sure that works too.

There are more elaborate things that could be done like run tests if they are included (many packages don't bundle their tests though) or try to validate the RST, but I think just pip installing is a very good first start, as it detects packages that are completely broken.

[sontek]

What I'd love is to see the packaging/build tooling help prevent this before it ever gets up to pypi/warehouse, although it might be nice to do some sanity checking of its own as well.

Its a weird issue since the way this is done is people reading files in setup.py, but would be nice if this was somehow detected and alert you that you are referencing things that wont end up in the dist.

[agronholm]

I don't think executing untrusted code on the PyPI server is an option. Can this be done without?

[r1chardj0n3s]

This is partly what the cheesecake project attempted to do; I like the idea in theory but it requires a bunch of work and support to make happen in practise.

[msabramo]

I guess tox would've caught this. Not everyone uses tox though and even people who do sometimes forget. Also tox could probably pass inadvertently because of a file in the dev's workspace that's not pushed up to the canonical git repo when using setuptools_git

I note that CPAN does automated testing of packages - I have no idea how they sandbox. VMs would be my guess. Obviously Travis CI and drone.io sandbox code to run tests. A system with VMs or Docker containers could help.

That said I recognize that this requires non-trivial resources so it's not easy.

[msabramo]

How awesome would it be to search PyPI and be able to sort by test coverage?!

What if setup.py had a field for ci_server and you could plug in Travis, Circle, Drone.io, etc. and they could run on your release tarball rather than your VCS repo?

[agronholm]

What would be even more awesome is if we ran pyroma against uploads and users could filter out projects with low scores? I would personally find that VERY helpful. I would even vote for rejecting uploads that get lower than a minimum score. IIRC pyroma doesn't even require sandboxing.

[msabramo]

Example of how CPAN runs tests on packages:

http://ppm4.activestate.com/i686-linux/5.16/1600/M/MS/MSABRAMO/App-TarColor-0.011.d/log-20120801T133902.txt

We can't let Perl show us up! 😄

ActiveState is active in Python as well as Perl -- maybe they are interesting in hosting a service like this like they do for the Perl community?

[dstufft]

@agronholm Ok so Looking at pyroma it has the following checks

1. The package should have a name, a version and a Description. If it does not, it will recieve a rating of 0.
2. The version number should be a string. A floating point number will work with distutils, but most other tools will fail.
3. The version number should comply to PEP386.
4. The long_description should be over a 100 characters.
5. Pyroma will convert your long_description to HTML using Docutils, to verify that it is possible. This guarantees pretty formatting of your description on PyPI. As long as Docutils can convert it, this passes, even if there are warnings or error in the conversion. These warnings and errors are printed to stdout so you will see them.
6. You should have a the following meta data fields filled in: classifiers, keywords, author, author_email, url and license.
7. You should have classifiers specifying the sypported Python versions.
8. If you are using setuptools or distribute you should specify zip_safe, as it defaults to "true" and that's probably not what you want.
9. If you are using setuptools or distribute you can specify a test_suite to run tests with 'setup.py test'. This makes it easy to run tests for both humans and automated tools.
10. If you are checking on a PyPI package, and not a local directory or local package, pyroma will check the number of owners the package has on PyPI. It should be three or more, to minimize the "Bus factor", the risk of the index owners suddenly going off-line for whatever reason.
11. If you are checking on a PyPI package, and not a local directory or local package, Pyroma will look for documentation for your package at pythonhosted.org and readthedocs.org. If it can't find it, it prints out a message to that effect. However, since you may have documentation elsewhere, this does not affect your rating.

So going through these checks!

1. Yes, Yes, Description or Summary? Either way, Maybe.
2. Yes, but I don't think PyPI can actually look at this without executing the setup.py
3. Yes, once PEP440 is finalized I want to enforce this.
4. Maybe
5. Pyroma probably does it wrong, because just because it works in docutils doesn't mean PyPI will render it. But I agree with the check in spirit.
6. * Classifiers -> Yes, but specific ones
   • Keywords -> No, they are kinda dumb
   • Author -> Sure
   • Author Email -> No
   • URL -> No, sometimes the PyPI Page is the URL
   • License -> I think this is better handled by classifiers
7. Yes-ish, In reality a dedicated metadata would be better for this but this is what we have ATM
8. We can't determine this from PyPI w/o executing
9. Useful, but I'm not sure we want to be hardcoding in setup.py stuff like this right now
10. Ehh, I'm not real excited about this one. 3 is an arbitrary number and I don't think giving people a negative mark because they didn't convince two other people to maintain their library isn't very nice.
11. Documentation is a big one i'd like to do as well, Docs on pythonhosted.org we can detect super easily, however RTD is harder. Crate did this and had to maintain a table of mappings because it had to guess at what the RTD link would be. This will be easier if we key it off a specially named project-url, but setuptools doesn't submit project urls ATM so it's hard to give people a negative mark for something that they can't actually give us yet if they aren't using pythonhosted.org

[agronholm]

What I propose is:

1. Enforcing that all the minimum files for installation (setup.py and egg-info/dist-info/whatever) is in place
2. Enforcing that name, description, long_description and license are filled out
3. Enforcing that the version conforms to the relevant PEPs (for new projects at least)
4. Enforcing that the supported range of Python versions is somehow indicated in the metadata
5. Filtering out projects in search that don't have downloadable distributions available

[dstufft]

1. Yes!, PyPI Legacy "tries" to do this now, but I don't think it does a very good job at it.
2. Techincally License field isn't supposed to be filled out if a license classifier fits, so as long as we have that logic in place, then OK by me.
3. Need to figure out what sort of coverage the related PEPs have over the current versions on Python, but I agree.
4. Probably OK
5. Yes!

[brainwane]

This would be a fantastic feature. I'm only moving this to a future milestone because it is not on the critical path for the immediate goal of switching from the old PyPI to Warehouse.

[jayfk]

Is this something people are still interested in?

If we had a system in place that allows to fully install a package there’s lots of other useful information we can extract from it. Transitive dependencies, metadata, even tests.

[agronholm]

Yes, people are still interested in this but it's been deferred until the immediate goals have been met and Warehouse has replaced Cheeseshop as the official PyPI.