Skip to content

Resolve the cache self test failure when drop privs is enabled#2120

Merged
jhiemstrawisc merged 5 commits into
PelicanPlatform:mainfrom
h2zh:self-monitor-xrdhttp-pelican
May 22, 2025
Merged

Resolve the cache self test failure when drop privs is enabled#2120
jhiemstrawisc merged 5 commits into
PelicanPlatform:mainfrom
h2zh:self-monitor-xrdhttp-pelican

Conversation

@h2zh

@h2zh h2zh commented Mar 20, 2025

Copy link
Copy Markdown
Contributor

This PR needs to work together with another PR in the xrdhttp-pelican plugin repo. That was already merged in and picked up by Pelican so you don't need to do any extra steps except pulling a Pelican image after 7.15.

  • This PR resolves the failed self test in cache when drop privs is enabled. Origin self test won't be affected.
  • A new package self_monitor to resolve the import cycles (xrootd package can't be imported to cache package). This package integrates the entry point for both origin and cache's self test.
  • Modified the functions in self test package impacted by the drop privs mode
  • Tell the plugin what are the self-test file transplant destinations by setting the environment variables (hardcoded the destination paths)
  • The self-test file is named "self-test-*.txt" when it created in its birthplace, but once it is transplanted to the selfTest dir, it is renamed to "self-test-cache-server.txt" and overwrite the previous file. The original file in the birthplace will be deleted after the transplant.

How to test

Run Pelican Cache in drop privs mode by setting the following configs in pelican.yaml

Server:
  DropPrivileges: true
  UnprivilegedUser: pelican

Before this PR, the web UI is unable to open. With this one, the web UI works fine.

Update:
Done in a subsequent PR #2292 : Found logs in Cache logs suggesting other 6 directories needed to set correct permissions. They don't crash the server abruptly though.

WARNING[2025-03-20T17:06:33Z] Failed to open auth database for reload:open /etc/pelican/server-web-passwd: permission denied
WARNING[2025-03-20T17:07:03Z] Failed to load key /etc/pelican/issuer-keys/pelican_generated_1740770124479586092_2096306543.pem: failed to read key file: open /etc/pelican/issuer-keys/pelican_generated_1740770124479586092_2096306543.pem: permission denied
WARNING[2025-03-20T17:07:03Z] Failure during cache director-based health test clean up routine: stat /run/pelican/cache/namespace/pelican/monitoring: permission denied
WARNING[2025-03-20T17:08:03Z] Failure during server TLS maintenance routine: open /etc/pelican/certificates/tls.key: permission denied
ERROR[2025-03-20T17:08:03Z] Failure when generating authfile: Failed to create a generated authfile /run/pelican/xrootd/cache/authfile-cache-generated: open /run/pelican/xrootd/cache/authfile-cache-generated: permission denied
ERROR[2025-03-20T17:08:03Z] Failure when emitting the scitokens.cfg: Failed to create a temporary scitokens file /run/pelican/xrootd/cache/scitokens-generated.cfg.tmp: open /run/pelican/xrootd/cache/scitokens-generated.cfg.tmp: permission denied

@h2zh h2zh added bug Something isn't working cache Issue relating to the cache component labels Mar 20, 2025
@h2zh h2zh requested a review from bbockelm March 20, 2025 00:29
@h2zh h2zh force-pushed the self-monitor-xrdhttp-pelican branch from 244d184 to 8713e4f Compare March 20, 2025 22:45
@h2zh h2zh added this to the v7.16 milestone Mar 21, 2025
@h2zh h2zh linked an issue Mar 21, 2025 that may be closed by this pull request
7 tasks
@h2zh h2zh force-pushed the self-monitor-xrdhttp-pelican branch from 8713e4f to 047b3b2 Compare April 8, 2025 13:34
@jhiemstrawisc jhiemstrawisc modified the milestones: v7.16, v7.17 Apr 22, 2025
@h2zh h2zh force-pushed the self-monitor-xrdhttp-pelican branch from 047b3b2 to afb20f6 Compare April 25, 2025 19:52
@h2zh h2zh requested a review from jhiemstrawisc May 7, 2025 14:26
Comment thread self_monitor/self_monitor.go Outdated
Comment thread self_monitor/self_monitor.go Outdated
Comment thread self_monitor/self_monitor.go Outdated
Comment thread self_monitor/self_monitor.go Outdated
Comment thread xrootd/xrootd_config.go
Comment thread xrootd/launch.go Outdated
Comment thread self_monitor/self_monitor.go Outdated
Comment thread self_monitor/self_monitor.go Outdated
Comment thread self_monitor/self_monitor.go Outdated
Comment thread xrootd/launch.go Outdated
@h2zh h2zh requested a review from jhiemstrawisc May 13, 2025 14:45
@h2zh h2zh force-pushed the self-monitor-xrdhttp-pelican branch from bb458c8 to 3d1c5d3 Compare May 13, 2025 15:04
h2zh added 4 commits May 13, 2025 17:28
This new package also combines the self_monitor logics in origin and cache
…ir when drop privs is enabled

- Modified the functions in self test package impacted by the drop privs mode
- Tell the plugin what are the test file transplant destinations by setting the environment variables
(hardcoded the destination paths)
- The self-test file is named "self-test-*.txt" (* is a random string decided by os.CreateTemp)
when it gets created in its birthplace, but it is renamed to "self-test-cache-server.txt" and
overwrite the previous file when it is transplanted to the selfTest dir. The original file in
the birthplace will be deleted after the transplant.
- Set the default values of Origin/Cache_SelfTestInterval
- Move self-monitoring.go to xrootd pkg, avoid creating a new pkg
- Use filepath.Join to construct the path
- Other minor improvements
@h2zh h2zh force-pushed the self-monitor-xrdhttp-pelican branch from 3d1c5d3 to f2385ee Compare May 13, 2025 17:28
@h2zh h2zh assigned jhiemstrawisc and unassigned bbockelm May 13, 2025
Comment thread xrootd/self_monitor.go Outdated
Comment thread xrootd/self_monitor.go Outdated
- Move Origin/Cache's SelfTestInterval fallback code to InitServer func so that the fallback value could be picked up by `pelican config summary`
@jhiemstrawisc jhiemstrawisc force-pushed the self-monitor-xrdhttp-pelican branch from 299876e to 7b954e2 Compare May 22, 2025 14:46

@jhiemstrawisc jhiemstrawisc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!!

@jhiemstrawisc jhiemstrawisc merged commit ff009a1 into PelicanPlatform:main May 22, 2025
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cache Issue relating to the cache component

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache self test fails when drop privs is enabled

3 participants