Add checksum support to pelican object put#2415
Conversation
patrickbrophy
added
enhancement
bbockelm
left a comment
bbockelm
left a comment
There was a problem hiding this comment.
Fairly minor changes requested; the overall approach is sound.
| requireChecksum: true, | ||
| requestedChecksums: []ChecksumType{AlgCRC32C}, | ||
| } | ||
| transferResult, err := uploadObject(transfer) |
There was a problem hiding this comment.
Interesting - how does this work? It looks like the mock server only responds to HEAD requests ... I would have assumed there was a failure prior to getting to the checksum code?
There was a problem hiding this comment.
After examining the internals of Go’s HTTP server, I found that if no header has been explicitly written to the response, it defaults to sending an HTTP 200 status. This behavior occurs because the test server only sets a header for HEAD requests, so other HTTP methods automatically receive a 200 response by default.
turetske
left a comment
turetske
left a comment
There was a problem hiding this comment.
A few requests, mostly for clarification comments and expanded test cases.
The reviewer requested some comments and changes to the help text. The reviwer also requested test coverage for when there is a mismatch in the checksums.
|
Using super powers to merge since review was handed off from BrianB to Emma and she's given the green light. |
4 participants
This PR addresses issue #2231. This PR introduces the
--checksumflag to thepelican object putcommand. This PR efficiently calculates the checksum on the fly as the object is being uploaded. Once the object is uploaded we will fetch the checksum from the Origin and then compare the locally calculated checksum and the server provided checksum. By default we use the crc32c algorithm to calculate the checksum.