Proposal: Add `--platform` flag to `pip install` and `pip wheel`

[chriskuehl]

What's the problem this feature will solve?

By default, the wheel platform detection pip does on Linux isn’t very advanced: it produces a platform tag like linux_x86_64, which is the same across nearly all Linux installations.

For packages with compiled bits that link against system-installed shared objects (.so files), it’s necessary to use different wheels for systems with different versions of the shared libraries. For example, a wheel built on Ubuntu 14.04 will not necessarily work on Ubuntu 16.04.

Being able to specify a custom platform name would allow building the same package version on different systems to produce two wheels with unique filenames that can be served from the same PyPI registry. For example, you could produce two wheels with the (already valid) filenames:

• numpy-1.9.2-cp36-cp36m-linux_ubuntu_14_04_x86_64.whl
• numpy-1.9.2-cp36-cp36m-linux_ubuntu_16_04_x86_64.whl

These wheels can be distributed to package consumers (e.g. internally inside a company), who get the benefits of quick wheel installation (no compile-on-install) and the possibility to work from several different platforms.

Describe the solution you'd like

We would like to add a --platform flag to the pip install and pip wheel commands, with identical behavior to the existing --platform flag on the pip download command. This change would allow building and installing wheels with a custom, user-provided platform name.

At $COMPANY, we’ve been building wheels with a custom platform name (just like the numpy example above) for several years. We upload these wheels to our internal PyPI registry, and install them using using pip-custom-platform, a wrapper around pip that adds in the --platform flag being requested in this issue.

Because we have hundreds of different internal Python projects, upgrading from one OS version to another is a long process during which we need to support development (and wheels!) on multiple OSes.

Overall, we’ve been very happy with the custom platform approach. The burden on projects and friction for developers is very low, and conceptually it fits in very nicely with the “platform” concept of a wheel (similar to how macOS wheels specify versions in the wheel’s platform tag). The downside for us right now, of course, is that we have to use a monkey-patched version of pip instead of the real thing.

pip-custom-platform has also started to be used by others in interesting use cases such as building packages for AWS Lambda to specify their own platform names.

The scope of the proposed change to pip is identical to the --platform flag for pip download: it’s just a flag that allows users to manually specify a platform to use. pip-custom-platform does have functionality to automatically generate platform names based on the Linux distribution, but we think it’s better to leave this complexity out of pip, and instead have users construct the platform name themselves and either pass in this flag manually or as an environment variable (possibly set at the system level by a system administrator, e.g. in /etc/environment).

Alternative solutions

We considered several potential alternative solutions to this problem:

• Abandon wheels entirely and compile on installation. This works but makes installation unrealistically slow (20+ minutes for some projects) and requires installing build tooling and library headers (gcc, fortran, libxxx-dev, cmake, etc.) everywhere.
• Running one PyPI registry per distribution, where each serves wheels with the linux_86_64 platform tag but which have been compiled on the corresponding platform. This could work, but is pretty unwieldy: every project would need some hackery in their build scripts to select the correct server, plus we’d have to run a ton of these registries (we currently support 3 Ubuntu versions, plus about half a dozen random platforms which are used in specialty cases but are still important to support, e.g. random Amazon Linux AMIs used for EMR). Additionally, if you were to accidentally use the wrong PyPI server and download wheels built for the wrong platform, pip would happily install them without problem and you wouldn’t notice until you get runtime errors in your code.

Additional context

Why can’t we use manylinux wheels?

manylinux wheels are a method of providing a single built wheel that works across most Linux installations. In general they do this job pretty well, but unfortunately we’re unable to use them for many packages due to the security concerns associated with them.

Specifically, for packages which need to depend on C libraries outside of the manylinux1 profile (for example, anything that depends on libssl, or almost all of the scientific Python libraries), the choices for producing a manylinux wheel are to either statically link in the dependencies, or to bundle the .so files in the wheel. In both of these cases, it is very difficult to roll out security updates across hundreds of different services, as it may involve patching and rebuilding the affected wheels, then building and deploying all of the services that consume them.

By contrast, rolling out security updates to shared libraries when services don’t bundle or statically link them is typically as easy as your distro’s equivalent of apt-get upgrade to pull in the latest patched version.

[asottile]

(full disclosure: I helped with this writeup) I'd be happy to assist with implementation as well if this proposal seems good

[lorencarvalho]

hey @chriskuehl

Curious if you think my open PR #5404 would satisfy your use case? It basically ports all the dist options from being exclusive to download to working with install (only when used in conjunction with --target) though it would make it easy to consume in the wheel subcommand as well, possibly.

[asottile]

It's close, though we would want to be able to use those options even without --target -- is there a reason you've restricted them to --target (there's already the requirement on either --no-deps or --only-binary which should make it "safe")

[lorencarvalho]

That was a request from @pfmoore in my initial issue #5355 -- and I tend to agree with their reasoning... it would become very easy to fundamentally break folks python installations if a local-incompatible wheel was installed. That's the whole reason for the local pep425 checks in the first place, as i understand it.

If you did want to install a locally in-compatible wheel, you could with my PR, you'd just have to specify --target /path/to/your/local/site-packages. I'd be open to something more explicit too, like --skip-compatilibity-check or something, but restricting to --target (for install at least) makes sense.

[chriskuehl]

Thanks for linking to that PR @sixninetynine, this looks great!

I totally agree that we want to be explicit and obvious so that nobody accidentally installs incompatible wheels. I think that the --platform flag does that pretty well, though -- I feel like a command like pip install --platform linux_ubuntu_16_04_x86_64 is pretty explicit already that you're installing wheels for a specific platform.

I can definitely see why something like PIP_IGNORE_PEP425 could be a concern (easy to get copy-pasted around without understanding what's going on), but I wonder if the same concerns still apply when explicitly naming one specific platform to be allowed (as opposed to disabling the safety mechanism)?

[lorencarvalho]

That's a really good point, merely specifying the platform stuff is making your intention pretty explicit. I don't hold too strong of an opinion on that, I'm definitely open to changing the PR based on what the community and maintainers feel is best 🙂

[asottile]

Requiring --target doesn't really help our usecases described above so I'd love for an amend on that 😆

Our goal would be to do something like:

echo 'PIP_PLATFORM=linux_ubuntu_16_04_x86_64' >> /etc/environment
echo 'PIP_INDEX_URL=https://pypi.mycompany.com/simple' >> /etc/environment
echo 'PIP_ONLY_BINARY=:all:' >> /etc/environment

and then pip install x would just work

[chadrik]

I tested the changes in PR #5404 and I agree that it would be more convenient to able to use --platform, etc without needing to use --target. For example, how can I install entry-point scripts when using --target?

[lorencarvalho]

@chadrik what do you mean by

how can I install entry-point scripts when using --target?

@asottile you can still accomplish what you want, you just need to also set PIP_TARGET to the site-packages dir of your choice (including the system's site-packages dir, if you wish).

(tmp_venv) darwin ~ $ PIP_PLATFORM="manylinux1_x86_64" PIP_ONLY_BINARY=":all:" PIP_TARGET="local/path" python3 -m pip install numpy
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/fe/94/7049fed8373c52839c8cde619acaf2c9b83082b935e5aa8c0fa27a4a8bcc/numpy-1.15.1-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.15.1
(tmp_venv) darwin ~ $ tar cf site-packages.tar local/
(tmp_venv) darwin ~ $ scp site-packages.tar xx@xx:~/
site-packages.tar                                                                                                                           100%   57MB   1.3MB/s   00:45
(tmp_venv) darwin ~ $ ssh root@159.65.110.140
Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-30-generic x86_64)
hemera:~# tar xf site-packages.tar
hemera:~# PYTHONPATH=local/path/ python3
Python 3.6.5 (default, Apr  1 2018, 05:46:30)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>

[asottile]

@sixninetynine I can't set PIP_TARGET statically and use pip. Each caller of pip needs to know about it which doesn't actually help at all (I would need a wrapper script that essentially undoes the --target requirement). (EDIT: in that it's not better than what pip-custom-platform has to offer)

It also doesn't help with putting things into the ./bin directory which is what @chadrik is asking about.

[lorencarvalho]

I can't set PIP_TARGET statically and use pip

@asottile I don't understand this, why not? Setting it to your interpreter's site-packages means it's set explicitly (thus satisfying the existing requirement to use --target) but it will behave like a normal pip invocation. I'm thinking something along the lines of

PIP_TARGET=`python3 -c "import sys;print(sys.path.pop())"`

I'm not saying that I'm right and you are wrong, I just want to make sure I fully grok the use cases you are describing. I'm still confused on what you and @chadrik mean by scripts or ./bin too... not sure I'm following there, an example would be super helpful!

That said, @ncoghlan mentioned on the pypa mailing list:

The basic idea seems like a good one to me, and starting out with the --target restriction doesn't hurt - it's much easier to relax restrictions like that later than it is to figure out whether or not you've correctly handled all the edge cases that arise without them.

So relaxing that requirement is definitely not out of the question.