Add a reqwest backend to rustup

[brson]

rustup currently supports a robust collection of HTTP backends: curl, hyper + native-tls and hyper + rustls. Add a fourth based on reqwest, making sure it has proxy and resume support. This will probably become the default backend.

It should be a matter of following the example in src/download, and adding RUSTUP_USE_REQWEST to src/utils/src/utils.rs.

cc #1181

[brson]

Like hyper, this backend can be enabled by default in the root Cargo.toml.

[brson]

@seanmonstar says the version of reqwest that supports proxies is not released yet.

[sebasmagri]

@brson I'd like to work on this. After a first run, it seems like we'll going to have to bump the hyper version to at least ^0.10.0 and url to ^1.2 to satisfy the reqwest requirements.

I started to move to hyper = "0.10.12" and url = "^1.2" but got stuck at implementing the new hyper::client::ProxyConfig constructor with this error:

the trait bound `S: hyper::hyper::net::SslClient<hyper::hyper::net::HttpsStream<<S as hyper::hyper::net::SslClient>::Stream>>` is not satisfied

Any guidance on this is totally welcome.

The changes can be seen at my WIP diff. There is some rustfmt noise on it. Sorry about that. The lines specific to the above error start here.

[LukeBousfield]

I've started work on this, but it's blocked by inejge/env_proxy#1

[brson]

@sebasmagri Ah, yes. I actually suspect the situation potentially requires even more complexity.

So as I recall, the version of reqwest that will support HTTP proxies is not yet released, and proxy support is a requirement for rustup. And that version of reqwest is going to be built on async hyper, that is, hyper 0.11.

And that makes the situation in the rustup download crate super awkward, because it is built on sync hyper.

So even if you manage to upgrade hyper from 0.9 to 0.10, that is not sufficient to satisfy the dependency reqwest is ultimately going to need in order to reach feature-parity with the curl backend in rustup.

So there are a few options:

• Bake two versions of hyper into rustup, at least when all HTTP backends are compiled in
• Upgrade the existing 2 hyper backends to async hyper
• Drop the existing 2 hyper backends

I'm pretty torn about the direction here. The reason rustup supports so many backends is basically because of vanity - I like having rustls be an option (and in fact want to turn on the rustls backend in the official builds); and I have some ambition to extract the download crate, document it thoroughly and publish it as a learning tool.

That said, if sync hyper is a deadend, then the value of having the hyper backends as a learning tool is going to decrease rapidly, and maintaining the sync hyper backend is going to be a burden (maybe @seanmonstar can comment on the future of sync hyper).

And since the curl backend remains the default, it might be simplest to just rip out the existing hyper backends and replace them with a single reqwest backend. I'd be sad to drop rustls support, but nobody is actually using it.

It should be possible to have both the sync hyper and async hyper crates linked to the same rustup binary, but again, maybe it's not worth it.

My only worry with dropping the hyper backend is that maybe somebody is using it to work around some bug in the curl backend in some particular scenario.

So I guess the most prudent course of action is:

• Drop the hyper and rustls backends, and the ca-loader crate
• Add a reqwest 0.6 backend now
• Upgrade to reqwest with proxy support when it is published
• Ask people to start testing the reqwest backend
• Turn reqwest on by default
• Drop the curl backend (someday).

I think I'll post an issue to users asking if anybody is using the hyper backend.

[brson]

@CodingPower472 thank you for making that fix to env_proxy.

@sebasmagri @CodingPower472 since you have both started on this issue it seems like you'll need to coordinate a bit to decide who continues here.

[brson]

Here's the urlo thread https://users.rust-lang.org/t/does-anybody-use-the-rustup-rustup-use-hyper-flag/11716

[LukeBousfield]

@sebasmagri can continue on this issue. I am not currently working on it. In my work, I simply panicked if there was a proxy present.

[seanmonstar]

(maybe @seanmonstar can comment on the future of sync hyper).

The 0.10.x branch of hyper won't receive any more attention. If someone finds a drastic bug and submits a PR, a version could be released. But that's all.

I'd be sad to drop rustls support, but nobody is actually using it.

I'd love for something like rustls to be viable in the future, too! Maybe someday, it (or any Rust TLS library) can replace openssl in the native-tls crate, and we're all for the better. For now, it's not really prudent to recommend using it by default. It only supports TLS 1.2 and 1.3, and it hasn't been audited. Certainly something to use in personal projects, if feeling adventurous.

I'd say you bullet list plan at the end there is what I also would recommend.

[sebasmagri]

I've updated my experimental branch removing the hyper and rustls backends. It's functional with the reqwest backend but of course does not support proxies yet. It can be updated when reqwest supports it. It doesn't support partial downloads neither. I'll look at this as a next step.

[seanmonstar]

Just wanted to update that reqwest 0.7 is released that includes proxy support.

[sebasmagri]

WooHoo! will try it ASAP @seanmonstar