Crawl politely

[SylvainDe]

Probably an interesting read: https://blog.scrapinghub.com/2016/08/25/how-to-crawl-the-web-politely-with-scrapy/ . Might be related to #36 too.

[SylvainDe]

https://thekindlyone.github.io/lessons/2015/04/03/scraping-tutorial/