Create .torrent files as part of planet creation

[Firefishy]

Currently we publish all planet files via https on https://planet.openstreetmap.org/
We'd like to start publishing bittorrent .torrent files for the large files eg: https://planet.openstreetmap.org/planet/2020/planet-200720.osm.bz2 ((92GB)

Requirements:

• Low maintenance.
• Webseed support in the .torrent files pointing to the file on https://planet.openstreetmap.org/ and any other suitable mirrors
• Reasonable torrent chunk size to keep .torrent size reasonable
• Use reputable public tracker(s) which are stable
• Standard client compatible

Goal:

• Lower the usage of server(s) supporting https://planet.openstreetmap.org
• Give users another potentially faster option for downloading the planet file

[mmd-osm]

// cc: @cquest who's already running a torrent service on http://osm.cquest.org/torrents/ and might share some ideas re. requirements.

[hbogner]

Planet was torrented long time ago also: https://github.com/mnalis/osm-torrent
cc @mnalis

[hbogner]

@mnalis you said something about one liner and rss, could you please elaborate more.

[mnalis]

As the person who was running https://github.com/mnalis/osm-torrent, I'm willing to help in any way needed. Let me know.

As for the comments on requirements:

• Low maintenance.

Sure. In it's most basic, it was just fire-and-forget apt-get install mktorrent and adding (somewhat longish oneliner, but still) oneliner in script which is called after .osm.bz2 file is generated. No further maintenance was needed. Mine was for example basically:

# this is our full featured torrent file: redundant trackers, tcp+udp, ipv4+ipv6, webseed
mktorrent -l $CHUNKSIZE \
  -c "See http://osm-torrent.torres.voyager.hr/ -- $LICENSE" \
  -a http://ipv4.tracker.osm-torrent.torres.voyager.hr/announce \
  -a http://ipv6.tracker.osm-torrent.torres.voyager.hr/announce \
  -a udp://tracker.ipv6tracker.org:80/announce,http://tracker.ipv6tracker.org:80/announce \
  -a udp://tracker.publicbt.com:80/announce,http://tracker.publicbt.com:80/announce \
  -a udp://tracker.ccc.de:80/announce,http://tracker.ccc.de/announce \
  -a udp://tracker.openbittorrent.com:80/announce \
  -w $URL_PLANET2 -w $URL_PLANET $FILE_PLANET -o ${FILE_TORRENT}.tmp \
  && mv -f ${FILE_TORRENT}.tmp ${FILE_TORRENT} \
  && ln -sf ${FILE_TORRENT} ${FILE_TORRENT_LATEST}

In addition to that, it would be nice to have RSS/Atom of the changes (I was providing RSS too with simple shell script) as at least few GUI clients offer to follow/autoshare files from RSS (and I know at least few people would want to help share the load that way, myself included). Of course, dedicated non-GUI sharers could just put in cron oneliner like wget https://planet.openstreetmap.org/latest.torrent -O /incoming/rtorrent/, but still, more community support could be had with RSS.

• Webseed support in the .torrent files pointing to the file on https://planet.openstreetmap.org/ and any other suitable mirrors

Yes, webseed is working in example above with -w options.

• Reasonable torrent chunk size to keep .torrent size reasonable

Yes, it's configurable. I was using at the time:

CHUNKSIZE=22		# 2^20=1MB, 2^22=4MB, etc. mktorrent 1.0 default ( 2^18=256kB) is too small for our ~15GB files

but one could choose any value s/he likes.

• Use reputable public tracker(s) which are stable

Any tracker which is public is by definition susceptible to be used by individuals/groups/actions one does not approve of.
Listed above are few example which were fine by me back then, but that probably changed.

Only way to be absolutely sure is to provide your own. Note that most torrent software do not actually require that you use a tracker at all (using DHT / PEX / LPD [instead / in addition] to find peers) so you could probably go trackerless (at least in start), but having a tracker or two in there is probably better idea.

I was providing at the time ipv4/ipv6.tracker.osm-torrent.torres.voyager.hr, which were running simple GPL PeerTracker PHP software using sqlite on standard shared Apache hosting using suexec+forking for each script invocation - for simplicity (and IPv6 tweaking which was lacking at the time, and another subject I was interested in). It was sure reputable and worked fine at the time, but if I had high load in mind, I'd probably used dedicated one that run as daemon.

• Standard client compatible

Sure. Worked just fine at least with uTorrent on Windows, mainstream / rtorrent / transmission / Vuze on Linux.

I'm glad bittorrent idea is finally getting traction. Let's get on with it!

[hbogner]

I had no idea it was that simple !!!
Just used you one liner to create a torrent for some iso files I'm mirroring, and it started webseed download from all the mirrors, with no seeders to start with.
Now i need to to build a planet seed machine :)

[cquest]

The only problem so far, is that in order to generate the torrent file, you have to download the planet file entirely.
That's why generating an official torrent file on planet.openstreetmap.org will save time and bandwidth for everybody.

As soon as this torrent is available, clients can start downloading parts from the web seed at planet.osm.org and share them with other torrent clients.

[grischard]

@cquest's scripts are available at http://osm.cquest.org/torrents/.pbf/. They handle web seeds and rss. Thank you Christian!

[mmd-osm]

What would be the absolute minimum version of this? Can we skip rss.xml, meta4, .m4j and running our own tracker, and leave that for later? What about history pbf and osm.bz2? Are they in high demand or can we skip them for the time being?

• install mktorrent
• iterate over all planet-20*.osm.pbf files in planet directory https://planet.openstreetmap.org/pbf/
• call mktorrent unless the respective .torrent file already exsits,
• move resulting .torrent file to torrents directory

Do we care about osm.pbf files that have been deleted after a while? Do we want to remove the .torrent file as well in that case?

[mnalis]

@mmd-osm

• As I noted above, absolute minimum is just installing mktorrent and adding call to it in script which generates latest .pbf file. You don't even need to iterate over existing planets, as usually the latest one is the most interesting from bandwidth perspective (and it would be irellevant for bandwidth after 2 month anyway, as older PBFs get removed from planet.openstreetmap.org). And if you're looking at absolute minimum of work, no need to remove old torrents or anything else. Just add that one oneliner to your script.

• however, as adding RSS support is as simple as copy/pasting already written and tested shell script I included, I see no big reason not to include RSS support at least (as not having RSS is going to exclude all people running Microsoft Windows OS, and many others not comfortable in writing commands in terminal, from helping share the load - as they would have to manually go to site and add newest torrent EVERY TIME it is released to help share the load. That would get boring/annoying very fast, so I guess support would drop drastically)

[verdy-p]

@cquest: your initial list of trackers is not correct, and it is a bit too small to make it resistant.
Here is the list I use:

udp://tracker.coppersurfer.tk:6969/announce

udp://tracker.opentrackr.org:1337/announce

udp://tracker.torrent.eu.org:451/announce

udp://tracker.leechers-paradise.org:6969/announce

udp://tracker-udp.gbitt.info:80/announce

http://tracker.gbitt.info/announce

http://tracker.cquest.org:6969/announce

http://tracker.computel.fr:80/announce

The last two do not work most of the time (including your own local track). As well you used a tracker with a non-routable domain name (in ".local") which should never be there (removed in the list above).
The UDP trackers in the list are almong the most reliable for now, and they are generally faster than HTTP (or worse HTTPS, but HTTPS is great to protect the network and get a secure restart; may be we should have an HTTPS tracker from OSM.org itself directly on its planet server, even if it has a small capacity, as long as it also hosts UDP, and if it annoucnes itself as well on a few other "open" trackers; and along with Webtracker pointing to the existing HTTP(S) server on OSM.org, it would allow faster propagation, as for now you still need to perform a full download once a week on your server, before you can announce it on your small tracker which is not available for long enough : you really need other external trackers capable of supporting more clients, even if now most torrent clients can also use PEX and the DHT to discover many more clients as they've started to download a few kilobytes).

Note that downloading about 128MB of data per file is always fast. But at ~127.5MB (128MB is for the extra overhead on the filesystem for allocating free space on the volume and writing/updating the directory entries), most downloads will stall for about half an hour or more, before resuming at must lower speed. During that initial time (half an hour or more), you cannot even shutdown your PC on Windows (this is a problem inside Libtorrent which affects both Windows and Linux, and used by many BitTorrent clients) because of the way it manages the file storage allocation and limitation inside OS'es for pending I/O to the disk. This time of ~1/2 hour may be reduced only if you download planet files entirely on an SSD, it may be much larger if you download to a network mounted drive (NAS) or a RAID (with parity or redundancy)... This problem is not at all a problem of the CPU performance or a memory barrier: the first 128MB are loaded very fast in just one second, then nothing for a long time where you'll notice a huge disk activity inside the OS, but after that you'll notice that the download restarts and resume at decent speed after just a few minutes (generally at about 700 megabits/second if there's a dozen of peers available), still limited to the disk write bandwidth (if downloading from a Gigabit Fiber internet access, or the Internet download bandwidth only with DSL access).

For some planet files I could then reshare them with a good ratio (for some planets shared since last January I have a ratio reaching about 1.5, but for the most recent planet file reshared fast, the ratio grows rapidly to about 0.25 in just one day, with about 10-20 peers connected constantly.

So even with a modest amount of peers on torrents, we save Terabytes on the OSM.org's planet server! This could be much more if OSM.org implemented this tracker directly on its server to greatly accelerate the distribution (this could also be used to download some small RSYNC datafile and redistribute the minute diffs listed in it also as torrents instead of RSYNC; classic mirrors are not fast enough to significantly reduce the workload on OSM.org's servers)

[verdy-p]

Also I can provide some statistics about the current growth of the planet file (in .pbf format):
It grows at a significant but almost constant speed each week:

• 48.89 GiB on 2020-01-20
• 49.00 GiB on 2020-01-27
• 49.11 GiB on 2020-02-03
• 49.22 GiB on 2020-02-10
• 49.33 GiB on 2020-02-17
• 49.44 GiB on 2020-02-24
• 49.55 GiB on 2020-03-02
• 49.67 GiB on 2020-03-09
• 49.78 GiB on 2020-03-16
• 49.91 GiB on 2020-03-23
• 50.04 GiB on 2020-03-30
• 50.18 GiB on 2020-04-06
• 50.31 GiB on 2020-04-13
• 50.46 GiB on 2020-04-20
• 50.61 GiB on 2020-04-27
• 50.75 GiB on 2020-05-04
• 50.89 GiB on 2020-05-11
• 51.04 GiB on 2020-05-18
• 51.18 GiB on 2020-05-25
• 51.30 GiB on 2020-06-01
• 51.41 GiB on 2020-06-08
• 51.52 GiB on 2020-06-15
• 51.64 GiB on 2020-06-22
• 51.77 GiB on 2020-06-29
• 52.03 GiB on 2020-07-13
• 52.14 GiB on 2020-07-20
• (there was no update on your RSS tracker on 2020-07-27, may be you were in holiday or you had a problem downloading it from OSM.org in that week)
• 52.43 GiB on 2020-08-03
• 52.55 GiB on 2020-08-10
• 52.66 GiB on 2020-08-17
• 52.78 GiB on 2020-08-24
• (there was no update on your RSS tracker on 2020-08-31, may be you were in holiday or you had a problem downloading it from OSM.org in that week)

I can predict that today's planet file (in .osm.pbf format) for 2020-09-07 will be ~52.89 GiB, given that each week it grows by about 0.11 GB. There was no effect on this growth caused by the COVID-19 pandemic even if more people were at home (but not only for working on OSM, and less people going outdoor, many have working on "cleanup tasks" or just refined the geometry of existing objects around them only to match the most recent imagery; bots are still active to remove unnecessary duplicates and new data imports from public agencies are coming at slower rates, as many of them have been delayed).

---

The prediction was correct and is still valid today:

• 52.91 GiB on 2020-08-31
• 53.05 GiB on 2020-09-07

[verdy-p]

Also not that sharing on the same host not only a large OSM planet dump, but also a popular Linux distrib ISO significantly helps maintaining the DHT connected (the DHT is used across multiple shares).

I host for example the ISO for Linux Mint 20 (64-bit: 3 versions for its choice of desktop environment: Xfce, Mate and Cinnamon the most popular): these ISO are not very large (about 1.8 GiB) and they constantly have a large number of connected peers > 1000), and they are updated with one or two versions each year. Even the Linux Mint 19 is still popular (version 20 no longer has a 32-bit version, so the 32-bit version is still widely requested in version 19).

This allows faster boot time for the DHT and improves the stability for faster discovery of clients interested in OSM data, with various network types (IPv4 or IPv6 ; mobile, fiber, DSL and dedicated servers on backbones/datacenters) and lot of countries worldwide.

[hbogner]

Just a tought
If you create planet.torrent on ironbelly after planet.pbf is created could all the mirrors get it by torrent too and not wait for rsync of the whole file?

[verdy-p]

I do agree that syncing mirrors with torrents would potentially allow a faster start of the swarm and better stability without pressing the initial server with excessive loads that mirrors (and all individual downloaders) will attempt to do for hours or days.

For now getting a new weekly dump or even a daily dump is just a "dream". And this causes significant problems, such as the lack of map renderers (remember that Wikimedia has already announced it will severely limit access to its own tile server).

We need a solution to better scale the distribution and largely increase our own "CDN" which is very fragile, and very slow to update to reflect changes (or corrections after some small errors, like most of England turned to a giant beach recently because a tiny change in a small beach, or because incomplete changesets that were broken prematurely for variable technical reasons).

How can we offer stability to the network? We need to demand LESS to mirrors and servers, and allow more people to contribute a share of their computing or networking power. Torrents are a good solution towards this goal (this is not the only solution of course). But if servers of the Foundation are less stressed, that power will have better use for more powerful tools not doing the same things massively and repeatedly (and having difficulties to synchronize each other in reasonable time).

That's why OSM still has to play in the same garden as commercial solutions (that have developed a much more massive CDN allowing their contents to be updated and delivered faster and more reliably).

I only see Rsync useful only for internal synchronization of servers managed with a unified administratioàn and using the same tools and a strong interaction between their local administrators: access to the Rsync could then be restricted to just a few internal mirrors. But I am convinced that a P2P protocol (torrent or Kademlia- based) would scale better than the current solution based on a single central administration: we could use a more heterogenous array of hosts more easily deployable worldwide, and we should be able to use the growing networking capabilities of individual fiber-based internet accesses without necessarily having to develop our own sets of unstable peerings poorly administered and not balanced at all.

If you want a minimum balancing of workload, torrents fit quite well and they require almost no development. The protocol is already there. We just lack a few maintenance scripts in the OSM servers to start seeding the dumps in a more cooperative way (and without requiring a strong and fast commitment of each participant).

Implementing the root tracker directly on the OSM servers is not something complicate to do (after all Christian Quest did that with his own HTTP downloads for servers in OSM France) and opensourced that solution. But he also complained that it is difficult to get the first whole download.

As well we need a few more open trackers than just the two ones listed by Christian (these two are not very stable, and not even Christian's tracker which regularly goes offline or cannot accept many requests): generally the webseeds located in Germany make an additional link needed, but it also has limitations: downloading by torrent a 50GB dump still requires 5 to 12 hours, too much for most users that want to be able to use their PC just for a couple of hours, or be able to reboot it (for installing system updates or their own software) without waiting for the same time.

I've see several occasions where these torrents, started to download one week, never finished completely before the next week when a new dump was already available: with this new version, the number of seeders dramatically decreased instantly. If you do not observe that, you may not be able to get the next download completely after you finish the first one (and you cannot perform them in parallel: most seeders limit the total bandwidth for all the files you attempt to download in parallel.

Clearly we have an unsolved scalability problem. And the cause is located precisely at OSM's servers not using the best options that are already available today, but still not used.

[mnalis]

So, is there anything more we can do to help implement planet torrents, and in a way that minimizes load on sysadmin team and other overloaded OSM people? Write a pull request? What would be appreciated? I do get the irony that even asking such questions is generating more load