update pollESGF SOLR->Globus (#246, #115)

[durack1]

Description

This PR updates the ESGF index polling from LLNL SOLR to ORNL Globus. There are a couple of quirks we'll need to keep track of. Intermittent timeouts were an issue (see esgf2-us/esg_fastapi#76), but I have successfully run this numerous times without issue, so hopefully this will be the case on another host

Ref #246
Ref #115

Checklist

Please confirm that this pull request has done the following:

• Data released on ESGF
• ESGF update pulled in here
• Documentation added (where applicable)
• Changelog item added to changelog/
• Did a new release after merging

[durack1]

We are missing a couple of pmount entries, as both CEDS-CMIP-2025-04-18 and DRES-CMIP-BB4CMIP7-2-1 are only published at NERSC.

@sashakames has now replicated the complete LLNL: /p/ structure on NERSC: /global/cfs/projectdirs/m4931/gsharing/user_pub_work/input4MIPs (some perm tweaks are required for access). So, we should be good to migrate the entire processing, from publication to poll to query across to NERSC, reproduce everything (including the MRI-JRA55-do-1-6-0 data, which is live again here).

Will see if addition to the pollESGF.py update, if running the database query code gets things live on NERSC..

[durack1]

Just deployed a scrontab on NERSC, so should start seeing auto-updates happening 2-hourly to start. Once these are humming along will drop back to 6-hourly.

Once there is a perms wash over /global/cfs/projectdirs/m4931/gsharing/user_pub_work/input4MIPs I should be able to update the database completed, hopefully we're all green, and then we're off and running again on a "production" system

[durack1]

Nope, it seems that there is some dependency that requires a system library which is not currently available on NERSC - #265

[durack1]

Ok this looks promising, converting to PR (from draft)

[durack1]

@znichollscr I might have managed to stumble my way through all this - still need to update the PIK-CMIP-1-0-0 info, but otherwise, we might be close to a merge - if you have a window, a peek would be appreciated

[znichollscr]

@znichollscr I might have managed to stumble my way through all this - still need to update the PIK-CMIP-1-0-0 info, but otherwise, we might be close to a merge - if you have a window, a peek would be appreciated

Yep nice. I added a CHANGELOG in #268. This also has the double benefit of checking whether the CI will behave. The changes to the database comment is failing, but that makes sense as we're changing basically every entry. So, suggested steps:

1. merge Add CHANGELOG #268
2. merge this PR (i.e. update pollESGF SOLR->Globus (#246, #115) #264)
3. in a separate, follow up MR, re-run the scrape script and do the PIK update stuff (I suggest doing this separately as there might be surprises and it'll be easier if we have fewer moving parts)
   • whether you do this before or after Resync HTML pages #262, up to you