Skip to content

Caches with default configuration won't start after both #2869 and #3042 compounded bugs #3080

@jhiemstrawisc

Description

@jhiemstrawisc

This bug is a bit nuanced:

  1. Notice this comment for SetServerDefaults(), which warns about a thing you should never do. Despite that warning, it appears we've been doing "The Bad Thing" in the function for awhile, and admittedly I'm guilty of this too.
  2. However, doing "The Bad Thing" wasn't actually a big problem until commit 8361df1 from mega-sized AI PR #2869 was merged. At that point, "The Bad Thing" almost became bad enough to break stuff. I assume the AI noticed this, but rather than understand the bug by reading the docstring, that commit worked around "The Bad Thing" by running SetServerDefaults() multiple times, which somehow hid the issue (see line 1930 at that commit). I haven't gotten into how the issue was effectively hidden by running SetServerDefaults() twice because I don't want to dig much more into a 2600 line PR when the fix is more clear to me now.
  3. Separately, @h2zh in #3042 was trying to fix a different bug stemming from my having done "The Bad Thing" before it was consequential, and he noticed the SetServerDefaults() duplication hack from Allow configuration to be mutable at runtime #2869. When he removed that duplication, suddenly all the instances of the "Bad Thing" became... bad.
  4. These things all stacked and as a result, multiple chunks of configuration, especially around derivation of default parameters for caches, are broken. The symptom from this result is that Pelican constructs bad data/meta directories to hand to XRootD, which then chokes on startup with error messages like:
INFO[2026-02-05T17:50:18Z] =====> oss.localroot namespace                daemon=xrootd.cache
INFO[2026-02-05T17:50:18Z] 260205 17:50:18 041 oss_Config: invalid space path -  data  daemon=xrootd.cache
INFO[2026-02-05T17:50:18Z] =====> oss.space data data                    daemon=xrootd.cache
INFO[2026-02-05T17:50:18Z] 260205 17:50:18 041 oss_Config: invalid space path -  meta  daemon=xrootd.cache
INFO[2026-02-05T17:50:18Z] =====> oss.space meta meta                    daemon=xrootd.cache
INFO[2026-02-05T17:50:18Z] ++++++ Configuring pfc mode . . .             daemon=xrootd.cache
INFO[2026-02-05T17:50:18Z] ------ Storage system initialization failed.  daemon=xrootd.cache
INFO[2026-02-05T17:50:18Z] 260205 17:50:18 041 XrdPfc_Cache: error Config() Unable to create an OSS object  daemon=xrootd.cache

I'm hoping the fix is as simple as not doing "The Bad Thing".

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingcacheIssue relating to the cache componentconfigurationcriticalHigh priority for next release

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions