HTTPS by default

[konklone]

HTTPS is already configured at https://api.data.gov. So, api.data.gov should make it the recommended and default protocol.

I'm sure the team here already understands the various benefits of HTTPS, and I don't mean to preach to the choir, but here are a few:

• Broader client-side compatibility. Any APIs under api.data.gov wishing to support CORS or JSONP for in-browser operation will be completely blocked on HTTPS websites, as active mixed content.
• Enhanced privacy for apps and users using APIs at api.data.gov -- HTTP headers and query string parameters (among other things) will be encrypted. For APIs accepting particularly sensitive data (like lat/long or zip), this is critical.
• The primary benefit of HTTPS: a stronger guarantee that you're speaking with the real api.data.gov.

Much of the performance hit involved in SSL can be mitigated - @igrigorik's excellent istlsfastyet.com contains many references for doing this.

There are a few ways this issue could be tackled:

• Update the documentation to always use https:// in the examples.
• Update all links to api.data.gov to use https://.
• Begin redirecting all http:// requests to https:// for the main api.data.gov website.
• Begin redirecting all http:// requests to https:// for any contained API.
• Set a HSTS header that tells clients to always use https:// going forward.

[GUI]

Great suggestion. At launch we didn't have HTTPS available on this api.data.gov subdomain, so a lot of this stems from that, but I agree HTTPS should be the default. I'll get the documentation updated soon.

However, I think HTTPS redirection might be slightly more problematic to introduce for existing APIs. While technically HTTP redirects should be followed by clients, I've experienced my share of problems with "dumb" clients not following redirects, so introducing them to an API can cause things to break. This is never really a problem in web browsers that properly follow the location headers, but if you're fetching an API using a lower-level HTTP library (as is relatively common for APIs), you often have to explicitly account for redirections in your code. You could definitely argue that redirects should be handled by a proper client, but in my experience that often doesn't happen, and by starting to return redirects I'm just afraid we'll break existing API usage.

If you have any experience introducing HTTP redirects for APIs, though, I'd love to hear about it (maybe it's not as big of a deal as I think it might be). Or if you, or anyone else, has more general thoughts this potential redirection issue, those would be great to here. If everyone thinks tackling forced HTTPS is important enough, one options is to try and tackle this migration with all our current users (notify them of the upcoming change, give them ample time to change their code, give them more warnings, and then finally roll out the change at a later date).

Thanks for bringing this up!

[konklone]

Actually, I had the exact same concern about redirection that you've expressed here - when I announced HTTPS by default in January for an API I manage in my work, I didn't include HSTS or forced redirection, and we continue to support HTTP. This is primarily because I wanted to make that announcement right away, and so I put off tackling the harder aspects.

It's hard for me to know how difficult it is for api.data.gov to migrate its users, but maybe we can trade experiences together here. I'm recording the user agent of all requests, and just in the last few minutes started capturing the protocol of all requests in propublica/sunlight-congress@bb86221. That'll help me get a sense of how much http: traffic there is, and what API keys among those are using user agents other than web browsers.

Over the last couple months, I've tried to bend the curve of http: traffic downwards by converting our own applications that dogfood the API to exclusively make HTTPS queries. I also went and filed pull requests on the most popular Python, Ruby, and Node libraries that people use to integrate with our API, all of which were merged and released. (I actually just went and filed a PR with a PHP library a few minutes ago.)

I've also just trolled around Github's code search looking for domain references, and filed random pull requests updating the protocol wherever it looked like the code was in use. There aren't too many for api.data.gov, though of course they could be referenced in private repos and indirectly. On that note, I could probably go further by searching for people who are using outdated versions of the the main client libraries I pushed to use https:.

Finally, I could also announce a migration window before we disable HTTP support, and send an email out a warning to all API keys that have hit our API with an http: protocol and a non-browser user-agent over some span of time.

Hopefully me vocalizing my inner monologue provides some ideas for what to do next -- I'm game for pushing a migration window if you are. 😸

[konklone]

Yesterday, I sent out the below email to ~170 people who had made unencrypted requests in a 3-day span to our Congress API.

Hi,

This is the Sunlight Foundation!

We've noticed that in the last few days, your Sunlight Foundation API key has been making unencrypted requests to our Congress API.

We have an encrypted (https://) endpoint that we would like you to use instead. We are considering disabling or redirecting unencrypted use of the API in the future.

We feel strongly that usage of our API should be encrypted, so as to protect the confidentiality and privacy of you and your users. We recently explained why we feel strongly about this on our blog.

Fixing this is likely to be very simple:

• If you use a helper library to use our API, check to see if they've shipped an updated version that uses our HTTPS endpoint. If so, update your version of that library. If not, request that they do so, or make the change yourself in your copy.
• If you hit our API directly, without a helper library, toggle http:// to https://. In most contexts, this is all you need to do.

If you have any issues addressing this, you can reply to me directly, or write to our API discussion list. I'm happy to answer any questions that may come up.

Thanks,
Eric Mill

(Actually, about half of them got a more strongly worded variant, if their requests were sending user lat/longs or zip codes.) And this was just from usage in the last 3 days, as that's the size of our rolling analytics window at the moment -- I'm working on expanding it. There are probably more than 170 keys making unencrypted requests.

I've fielded ~25 responses from users since then, most of which were "Oh! Thanks, I fixed it" or "Oh! Thanks, I'll fix it". At least one of them was using the PHP client that I successfully nudged the week before to ship an update using HTTPS. For many others, it was just a one-character change. A few people had no idea they were even using our API, but either changed it or didn't care if their use was cut off.

No one complained! The quick, positive response makes me feel good about:

1. Sending another blast from a longer analytics window, to people not yet emailed
2. Announcing a "we're going to force HTTPS" window for upgrading.

No. 2 is something to talk over more here before actually doing it. But I feel good about the whole thing so far.

[konklone]

This thread was the inspiration for some of the content that @nacin wrote up for this project: https://https.cio.gov/apis/

As api.data.gov migrates, feeding some of the experience back into that page, managed for now at whitehouse/https, would be awesome for other agencies looking to do something similar.

[GUI]

I saw that API page when the HTTPS site got released and thought it sounded vaguely familiar to our conversation here. :) But awesome work to everyone involved in the HTTPS standard--it looks really great.

But as an extremely belated followup, we are finally planning to at least tackle part of this for api.data.gov. In the next couple weeks we're planning to rollout functionality that will force all new APIs we host to HTTPS (and you can thank the HTTPS standard site going live for jogging my memory). Even though I've been trying to informally encourage all new users to use HTTPS, it easily gets missed as soon as people start exchanging links with HTTP in them. This should at least help us stem the tide of non-HTTPS APIs and ensure all new agencies moving forward on api.data.gov will be HTTPS-only (I just wish we had tackled this earlier). I may also try to implement something to try to ensure that new API keys that are signed up for are only used via HTTPS, even for existing APIs (but I'm less sure about how feasible that will be to implement).

Then the second part of this task becomes migrating existing APIs to HTTPS-only. As we've discussed, that's a little more complicated and will require a little more coordination/communication, but it's certainly doable. But we at least wanted to start with the simpler task of ensuring new clients are HTTPS-only, and then we can begin to work with existing agencies on potential migration.

And yeah, I'd definitely be happy to share our experiences and things we learn.

Thanks again for everything you're doing to push HTTPS forward, and sorry for the super-long delay in following up!

[konklone]

In the next couple weeks we're planning to rollout functionality that will force all new APIs we host to HTTPS (and you can thank the HTTPS standard site going live for jogging my memory).

Awesome.

I may also try to implement something to try to ensure that new API keys that are signed up for are only used via HTTPS, even for existing APIs (but I'm less sure about how feasible that will be to implement).

That's fascinating. If that is feasible, that's exactly the kind of case study I'd like to document.

Then the second part of this task becomes migrating existing APIs to HTTPS-only. As we've discussed, that's a little more complicated and will require a little more coordination/communication, but it's certainly doable. But we at least wanted to start with the simpler task of ensuring new clients are HTTPS-only, and then we can begin to work with existing agencies on potential migration.

Absolutely. I don't underestimate the challenge. The kinds of steps you think of will also be useful.

Thanks again for everything you're doing to push HTTPS forward, and sorry for the super-long delay in following up!

Thank you for being so supportive of the transition!