Move ESGF scrape to GitHub actions

[znichollscr]

Description

Move ESGF scrape to GitHub actions

An example of the PRs it creates: #267

Fix #115

Checklist

Please confirm that this pull request has done the following:

• Data released on ESGF
• ESGF update pulled in here
• Documentation added (where applicable)
• Changelog item added to changelog/
• Did a new release after merging

[github-actions]

No changes to the database between 'main' branch and 7979031

[github-actions]

No changes to the database between 'main' branch and e7c5436

[durack1]

@znichollscr nice, if this can be enabled we can monitor to see how it evolves.

Just thinking aloud, as we are the only ones that are publishing to the project, it's a little overkill to have this triggering regularly, we only need this to happen when some input4MIPs status changes, and currently, none of this is automatic without our explicit involvement

[github-actions]

No changes to the database between 'main' branch and 59d68ae

[znichollscr]

Just thinking aloud, as we are the only ones that are publishing to the project, it's a little overkill to have this triggering regularly, we only need this to happen when some input4MIPs status changes, and currently, none of this is automatic without our explicit involvement

I agree. I've got it happening once per day at the moment. We can also make it less often e.g. once per week or only on manual trigger. Let me know if you have a strong feeling.

[durack1]

Let me know if you have a strong feeling.

My strong feeling would be to automate this at a high frequency now, and see how often it breaks. We need to start testing the system before CMIP7 pressure starts to accumulate, and we're in a perfect position to do this on a comparatively tiny live project - my NERSC/perlmutter cron is 6 hrly, why not do the same here and start logging success/failures?

[github-actions]

No changes to the database between 'main' branch and 3252ddc

[durack1]

Which action should I be watching the "udpate.." or this https://github.com/PCMDI/input4MIPs_CVs/actions/workflows/update-esgf-scrape.yaml

[znichollscr]

This one: https://github.com/PCMDI/input4MIPs_CVs/actions/workflows/update-esgf-scrape.yaml Not sure how we ended up with duplicates

…

On Thu, Jul 10, 2025, 15:45 Paul J. Durack ***@***.***> wrote: *durack1* left a comment (PCMDI/input4MIPs_CVs#276) <#276 (comment)> Which action should I be watching the "udpate.." or this https://github.com/PCMDI/input4MIPs_CVs/actions/workflows/update-esgf-scrape.yaml — Reply to this email directly, view it on GitHub <#276 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3KEXHZ3AU5LAJMPSU3NLIL3HZU53AVCNFSM6AAAAACBG6UIIOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANJXGUYTSNJUGA> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[durack1]

My NERSC scrape is firing off as expected, but the time taken has been interesting to watch, and also had a couple of outages/failures. Definitely something to keep an eye on over an extended period.

perlmutter:login31:~> tail -n 20 /global/cfs/projectdirs/m4931/gsharing/user_pub_work/input4MIPs/esgf-input4MIPs.log
...
Sat 05 Jul 2025 11:04:27 AM PDT
Sat 05 Jul 2025 05:00:10 PM PDT
Sat 05 Jul 2025 11:00:10 PM PDT
Sun 06 Jul 2025 05:00:17 AM PDT
Sun 06 Jul 2025 11:00:17 AM PDT
Sun 06 Jul 2025 05:00:18 PM PDT
Sun 06 Jul 2025 11:00:17 PM PDT
Mon 07 Jul 2025 05:00:18 AM PDT
Mon 07 Jul 2025 11:02:01 AM PDT
Mon 07 Jul 2025 05:01:57 PM PDT
Mon 07 Jul 2025 11:00:15 PM PDT
Tue 08 Jul 2025 05:00:24 AM PDT
Tue 08 Jul 2025 11:00:11 AM PDT
Tue 08 Jul 2025 05:00:19 PM PDT
Tue 08 Jul 2025 11:05:47 PM PDT
Wed 09 Jul 2025 05:07:21 AM PDT
Wed 09 Jul 2025 11:05:27 AM PDT
Wed 09 Jul 2025 05:06:10 PM PDT
Wed 09 Jul 2025 11:07:26 PM PDT
Thu 10 Jul 2025 05:09:42 AM PDT

[durack1]

Looks like it triggered, as expected - without any changes, it's a little hard to ascertain if it worked - are there logs somewhere? All I could see is the job info at https://github.com/PCMDI/input4MIPs_CVs/actions/workflows/update-esgf-scrape.yaml

[znichollscr]

Can you see https://github.com/PCMDI/input4MIPs_CVs/actions/runs/16202657392/job/45745305418 It triggered, saw that there were no changes after running the script and as a result did nothing so working as it's meant to

…

On Thu, Jul 10, 2025, 21:43 Paul J. Durack ***@***.***> wrote: *durack1* left a comment (PCMDI/input4MIPs_CVs#276) <#276 (comment)> Looks like it triggered, as expected - without any changes, it's a little hard to ascertain if it worked - are there logs somewhere? All I could see is the job info at https://github.com/PCMDI/input4MIPs_CVs/actions/workflows/update-esgf-scrape.yaml — Reply to this email directly, view it on GitHub <#276 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3KEXH2IVDRI5B6O4TUWALD3H267TAVCNFSM6AAAAACBG6UIIOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANJYHAYDCMZYGI> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>