[PSA] twitter rate limiting twice in a row does not mean there is no more media

[inverteddensity]

I don't know if I'm allowed to open an "issue" like this, since it is just me projecting my tech illiteracy. But I never knew this, so I wanted to spread awareness of it, if I could.

I have been using gallery-dl for years, long before elon musk bought twitter. At some point after he bought twitter, he added rate limiting to it, if you download media older than the "media" tab of a profile. Until now, I thought if you triggered the rate limit twice in a row, you had downloaded all the media of a profile, since I have never received media after waiting such out. But today I did for the first time, retroactively rendering all my previous scrapes I've closed before they exhausted the rate limit potentially incomplete.

Again, this is probably obvious to everyone else, but I had no idea rate limiting was relative to date, not bandwidth. So it triggers when for example checking any given six month period for media, as opposed to only triggering when a threshold for bandwidth is reached. In hindsight, I guess I could've figured this, since I did not understand why sometimes waiting out a rate limit would return ones of media, but other times dozens. But I just assumed it was elon musk being inconsistently stingy, not him setting a literal standard time elapsed required to download all media from a profile depending on its age, which only coincidentally limits bandwidth depending on the size and amount of media the profile has posted across which dates.

I guess if I had to make this an actual feature request, assuming it were possible at all, I wish the rate limit message stated the date it has currently reached in checking for media at the time of being rate limited, so the layman could see that even though they are not receiving media, it hasn't finished checking everything, as evidenced by the date currently checked not having reached the date the account was created yet. Further, you could also state the date the account was created in the same message, so the user could calculate how many rate limits remain. Or if it is this consistent, you could estimate how many rate limits remain outright, sparing the user of dates entirely. Though I don't know if any of this is possible.

Again, this must be obvious to everyone else, but the way elon musk designed it seemed irrational to me, as opposed to rate limiting based on bandwidth. So I did not assume such to be the case at all. But this isn't blaming gallery-dl nor expecting nor hoping for anything of it; this is just me trying to raise awareness of possibly only my own unique incorrect, self-destructive assumption, which I am only just realizing was the case the entire time.

[vlf]

Be careful when hitting rate limits, one of my accounts got banned just for that and it wasn't even aggressive scraping, just one sngle media feed. Since then I run in extremely slow mode never hitting rate limit.

[inverteddensity]

Be careful when hitting rate limits, one of my accounts got banned just for that and it wasn't even aggressive scraping, just one sngle media feed. Since then I run in extremely slow mode never hitting rate limit.

Oh, thank you for the fair warning. There were a few things I didn't mention in the original post, but that reminds me that one of them was that I assumed if triggering the rate limit didn't download any media, that the bandwidth cost must be negligible, so twitter would not retaliate over it. So often I would scrape a twitter, it would hit rate limit, and I would scrape others freely, also hitting rate limit on them, and just keep the cmd instances of them open, for the rate limit on all of them to expire at the exact same moment. I assumed since it hit rate limit once, the chances of them having more media is virtually zero, so the cost to twitter's servers must be the same. But with the knowledge I now have, I see that I should really only wait out a single rate limit at a time. But truthfully I will not do that, because the prospect of having to wait 15 minutes (possibly several times over, from hitting rate limit over and over while gaining no new media) before scraping another twitter- to only scrape one at a time after rate limit- is dreadful. I guess I will just continue as I ever have, except let my many instances of gallery-dl hit rate limit over and over until it exhausts them for each account.

There was at least one other aspect of rate limiting I probably made a false assumption on, but in formatting the previous paragraph, I've forgotten it. But if I remember it and feel it can help others, I'll share it.

Edit: the other aspect of rate limiting that I wanted to mention but forgot about was that sometimes I would notice that the "media" tab of a twitter would display disproportionately fewer media than I was expecting. This has happened from accounts with supposedly thousands of media to accounts with only ones of them, but either way, the media tab only shows very few. I always wondered why this was the case, but I imagine now I know why: what is displayed in the "media" tab is based on age, not media count.

I have been guilty before of not understanding why an account with very few media had such a short media tab, and cancelling the download after hitting rate limit once, since I figured I would not get any more by waiting it out. Though now I know to always exhaust every rate limit, regardless of what can be seen in the "media" tab.

Edit2: Editing this instead of bumping this again. But is the standard workflow really exhausting all rate limits? It happens over and over again until it hits a:

twitter: 404 Not Found ()
twitter: Use '-o cursor=3_1886188099300110775/' to continue downloading from the current position

which has now happened to me so many times that I now have to make a list of each "cursor" and the associated twitter to get back to later, because if i do too many at once I become unable to browse twitter, since it refuses to load anything for me. Further, I literally scraped the following twitter:
https://x.com/linabun

where the oldest media visible in their "media" tab was posted "Feb 2, 2025":
https://x.com/linabun/status/1886188099300110775

and their account creation date is "February 2025". Yet after downloading that media, I continued to hit rate limit on their twitter nine more times until the 404 error I pasted above happened. So while it is true that there is a nonzero chance there could be media after hitting rate limit twice in a row, there is clearly not a standardized period of dates it is afforded checking each time, and the odds of it receiving new media might effectively be zero.

Providing the oldest date checked and the age of the profile might be necessary for useful feedback on this after all, since it seems gallery-dl will continue hitting rate limit effectively needlessly in some cases, and the user would have no way to know without manually checking the oldest media date and comparing against the creation date of the profile.

I am still trying to be paranoid of missing media after hitting rate limit twice in a row, but it is a tedious endeavor, and seems fruitless in virtually all cases.