Add built-in POSIXv2 origin backend with native HTTP server by Copilot · Pull Request #2960 · PelicanPlatform/pelican

Copilot · 2026-01-08T14:41:11Z

Implementation Plan: Built-in POSIX Origin Backend (POSIXv2)

Progress

Addressing second round of feedback. Initial refactoring and visibility changes complete.

Original prompt

This section details on the original issue you should resolve

<issue_title>Create built-in POSIX origin backend</issue_title>
<issue_description>1. Create a new "POSIXv2" storage backend in Pelican
2. Create a new "origin_serve" module that replaces the use of XRootD in the origin. When POSIXv2 is utilized as a storage backend, the launcher will initialize this module instead of xrootd.
3. Have the "origin_serve" module be based on FS-like interfaces from github.com/spf13/afero; integrate that interface with HTTP handlers from golang.org/x/net/webdav. It should implement GET, PUT, HEAD, and PROPFIND at the minimum.
4. SciToken / WLCG Token-based auth should be used; local_cache/cache_authz.go contains an implementation of authorization handlers with appropriate caching. Once the token is decoded and authorization decision made, info about the user and groups (if any) should be added to the HTTP context prior to invoking the filesystem code.

The ultimate goal is to have an end-to-end test with XRootD-based cache demonstrating upload, download, and directory listing with the POSIXv2 backend.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes Create built-in POSIX origin backend #2959

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

bbockelm

Reasonable first attempt. Please address the items I noted.

One important piece: checksums! These are not supported by afero so we'll need to do it on our own. Define an interface for fetching checksums given a filename and an implementation that uses extended attributes to store computed checksums (retrieving them by default; otherwise, recompute & store). Later, we'll develop additional methods for filesystems that don't support extended attributes.

Copilot · 2026-01-08T17:42:02Z

Reasonable first attempt. Please address the items I noted.

One important piece: checksums! These are not supported by afero so we'll need to do it on our own. Define an interface for fetching ch...

All feedback addressed. Added checksum support with:

Checksummer interface for fetching/computing checksums
XattrChecksummer implementation using extended attributes
Support for MD5, SHA1, CRC32
Checksum headers in HEAD responses (Digest, X-Checksum-Sha1, X-Checksum-Crc32)
Future: Add implementations for filesystems without xattr support

Commit 68c0ac6

bbockelm

Another round of change requests. Hopefully the last before my final touchups.

Copilot · 2026-01-09T00:44:35Z

Another round of change requests. Hopefully the last before my final touchups.

Addressed 8 of 16 comments in commit 0a53b53 (visibility, copyright, refactoring, OA4MP placement). Working on remaining comments which involve more complex changes to authorization, checksums, and token handling. Will complete in subsequent commits.

bbockelm

Many things to do. Will have VS Code agent work on these items.

bbockelm · 2026-01-13T01:52:43Z

+	}
+
+	// Extract groups from token using configured claim
+	if groups, ok := tokenClaims[um.groupsClaim]; ok {


Also accept a string of space-separated group names.

- Added checksum support using extended attributes (xattr) - Implemented Checksummer interface with MD5, SHA1, CRC32 support - Added checksum headers to HEAD responses - Updated token extraction to support multiple tokens from headers and query params - Moved user/group extraction into authorize function - Refactored launcher to use boolean flag instead of duplicate function - Added comprehensive authorization tests (SciToken, WLCG, caching, user extraction) - Added request logging at info level - Added test for checksum retrieval in e2e tests Co-authored-by: bbockelm <1093447+bbockelm@users.noreply.github.com>

- Fixed token extraction to use single case-insensitive operation - Fixed exportPrefixMap initialization to use global variable - Fixed typo: initializting -> initializing - Optimized checksum validation to only check mtime when cache exists Co-authored-by: bbockelm <1093447+bbockelm@users.noreply.github.com>

- Updated copyright year to 2026 in filesystem.go - Made structs/functions private (userInfo, getUserInfo, setUserInfo, etc.) - Added logger parameter to filesystem operations and removed log.Debugf calls - Fixed Readdir pagination with state management (dirEntries, dirOffset, mutex) - Moved OA4MP and test file maintenance to non-XRootD section in launcher - Code compiles successfully Co-authored-by: bbockelm <1093447+bbockelm@users.noreply.github.com>

We were going back and forth between breaking and fixing various use cases. This isolates the path manipulation logic for uploads and prevents regressions by adding standalone unnit tests.

bbockelm · 2026-01-15T17:43:01Z

Alright, here goes nothing...

Copilot AI assigned Copilot and bbockelm Jan 8, 2026

Copilot started work on behalf of bbockelm January 8, 2026 14:41 View session

Copilot AI changed the title ~~[WIP] Create built-in POSIX origin backend~~ Add built-in POSIXv2 origin backend with native HTTP server Jan 8, 2026

Copilot AI requested a review from bbockelm January 8, 2026 15:05

Copilot finished work on behalf of bbockelm January 8, 2026 15:05

bbockelm requested changes Jan 8, 2026

View reviewed changes

Comment thread e2e_fed_tests/posixv2_test.go

Comment thread launchers/origin_serve.go Outdated

Comment thread origin_serve/authz.go

Comment thread origin_serve/handlers.go

Comment thread origin_serve/handlers.go Outdated

Comment thread origin_serve/handlers.go Outdated

Copilot started work on behalf of bbockelm January 8, 2026 17:32 View session

Copilot AI requested a review from bbockelm January 8, 2026 17:50

Copilot finished work on behalf of bbockelm January 8, 2026 17:50

bbockelm added enhancement New feature or request origin Issue relating to the origin component labels Jan 8, 2026

bbockelm requested changes Jan 9, 2026

View reviewed changes

Copilot started work on behalf of bbockelm January 9, 2026 00:39 View session

Copilot AI requested a review from bbockelm January 9, 2026 00:45

Copilot finished work on behalf of bbockelm January 9, 2026 00:45

github-advanced-security AI found potential problems Jan 9, 2026

View reviewed changes

bbockelm force-pushed the copilot/add-posix-origin-backend branch from de1be45 to 3b1cdb4 Compare January 10, 2026 18:33

github-advanced-security AI found potential problems Jan 10, 2026

View reviewed changes

Comment thread launchers/origin_serve.go Dismissed

Comment thread launchers/origin_serve.go Dismissed

Comment thread origin_serve/handlers.go Dismissed

Comment thread origin_serve/handlers.go Dismissed

bbockelm force-pushed the copilot/add-posix-origin-backend branch from 1614ac8 to 5dae320 Compare January 11, 2026 17:01

bbockelm reviewed Jan 13, 2026

View reviewed changes

bbockelm self-requested a review January 13, 2026 01:59

Copilot AI and others added 6 commits January 14, 2026 14:55

Initial plan

cfec4be

WIP - checkpoint of the draft server handlers

e4f98d0

Add unit test coverage for the directory listing

b5ef1ad

bbockelm added 10 commits January 14, 2026 14:55

Have director advertise support for PROPFIND

db9b20c

Have XrdCl debug settings match ours

3cc1c61

Fix linter errors

6b1ff77

Factor out the path manipulation logic

728878e

We were going back and forth between breaking and fixing various use cases. This isolates the path manipulation logic for uploads and prevents regressions by adding standalone unnit tests.

Fix some golangci-lint issues

e3dbb0b

Fix use of PROPFIND against director for stat

cf3dbcd

Skip test of directory listings at cache - hit an xrdcl-pelican bug

ef12d80

Assume missing Depth header indicates a stat

139ab9a

Next round of improvements for POSIXv2

acf33bf

Add support for disabling direct reads

2f798fa

bbockelm force-pushed the copilot/add-posix-origin-backend branch from cfa3265 to 2f798fa Compare January 14, 2026 20:58

github-advanced-security AI found potential problems Jan 14, 2026

View reviewed changes

Comment thread origin_serve/checksum.go Fixed

Comment thread origin_serve/checksum.go Fixed

Comment thread origin_serve/checksum.go Fixed

Comment thread origin_serve/checksum.go Fixed

Comment thread origin_serve/checksum.go Fixed

bbockelm added 4 commits January 14, 2026 17:30

Update golang to 1.25 to get os.Root semantics

82bf01c

Fix calculation of upload URLs; make checksums respect os.Root

d59fc5a

Bump golanci-lint version to try for golang v1.25 support.

ab222af

Update container to 1.25 as well

075f197

This was referenced Jan 15, 2026

Switch to v1.25 #2995

Merged

Switch to golang v1.25 #2996

Closed

bbockelm marked this pull request as ready for review January 15, 2026 17:42

bbockelm merged commit 7159a71 into main Jan 15, 2026
30 of 43 checks passed

Conversation

Copilot AI commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implementation Plan: Built-in POSIX Origin Backend (POSIXv2)

Progress

Comments on the Issue (you are @copilot in this section)

Uh oh!

bbockelm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Jan 8, 2026

Uh oh!

bbockelm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Jan 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bbockelm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bbockelm Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bbockelm commented Jan 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jan 8, 2026 •

edited

Loading