Raw file update lags changes by minutes?...

[mogplus88]

Hi All,

I don't use github, I'm just a hobbyist programmer playing with a few microcontrollers, just for fun.

But recently I have been experimenting with over the air updates for ESP32 microcontrollers. The best way I've found so far (and the only way I've managed to actually get to work) is to maintain the latest version on github and download it from there to write to the microcontroller. It works very well, except for one minor issue I don't quite understand. It's taken me days to realise this, as I've been trying to debug a problem that doesn't exist.

So I've been making a change to the program, then testing it only to find that it hasn't downloaded the modified file, even though I just changed it and committed the change. Much head scratching and trying a few different things, even putting a couple of print statements in the program to try to see what the heck is going on, and suddenly it works! No rhyme or reason, it just seemed to work sometimes, then not.

I had assumed that the raw file would be the same file as the "actual" file (for wont of a better term) I just updated, but apparently not. It seems the actual file gets copied somewhere where the rawgithubuser site can find it. I'm guessing there is some background program running that scans github for updated files, and copies them to rawguthubuser? Or something similar. Anyway, it takes time. I finally found this out by updating my "actual" file, then opening the raw file in my browser, to find the changes hadn't got there yet. Lots of refreshing and it still wasn't there, until, at last, it was. It took well over a minute.

Is there a way around this? Other than putting a five minute delay in my program before it tries to download the raw file? Or making a cup of tea after I've updated the file, then download it to my ESP32?

All advice gratefully received.

[nuran-command]

I’ve noticed this kind of delay before with raw GitHub files-it’s usually just caching. When you commit a change, it can take a short time before the raw file URL actually shows the update. It’s not a bug; it’s just how the server and caching work.
A few things you can do:
• Give it a few seconds after committing before your program downloads the file. Even 30–60 seconds usually fixes it.
• Manually check the raw file in your browser first to make sure it has the latest changes before trying to download it programmatically.
• Add a unique query string to the URL (like ?t=timestamp) so the server treats it as a new request and fetches the newest version.
Basically, just be aware that the raw URL isn’t always “instant” after a commit, and a short wait or cache-bypass trick usually solves it.

[mogplus88]

Thanks for the reply. I figured it must be something like that. A bit irritating for my particular use case, where I want to download the raw file immediately after making a change. Not sure what you mean by adding ?t=timestamp. You you mean adding it to the url? With timestamp set to now?

I'm thinking the cup of tea option might be the solution for me, as it's not something I'm going to be doing often.

Thanks again,

;-) Ian

[nuran-command]

Ian, about the ?t=timestamp trick- it’s basically a way to bypass caching so you get the newest version of the file.
Normally you might request something like:

https://raw.githubusercontent.com/user/repo/main/firmware.bin

If you instead request:

https://raw.githubusercontent.com/user/repo/main/firmware.bin?t=1710000000

the server treats it as a different request. GitHub’s CDN usually caches files based on the exact URL. If the URL is identical, it may return a cached copy for a short time. Adding ?t= with a number makes the URL look new, so the server fetches the latest version instead of giving the cached one.
You can easily automate this in your ESP32 code. Just append something that changes each time you request the file.If your device has the time from NTP, you can append the current timestamp. If not, even something simple like millis() or a random number works.
Example idea:

String url = "https://raw.githubusercontent.com/.../file.bin?v=" + String(millis());

That way each request is slightly different and avoids cached responses.
tho, i think sometimes GitHub’s raw service itself takes 30–60 seconds to propagate a new commit across their servers. In that case, even the cache‑busting trick might still show the old version for a short while.Thank u

[mogplus88]

So much to learn, so little time...

Thanks ;-)

[nuran-command]

good luck, mate