Impact of the new feature
MSUnmerged will be able to cleanup RSEs which do not support directory listing (ls) functionality
Is your feature request related to a problem? Please describe.
Currently there's a lot of historical unmerged data left over at the RAL RSE. These are not cleaned up by the MSUnmerged script as we're an object store and do support directory listings.
Describe the solution you'd like
Could you create a fallback function for
|
def _listFilesForDeletion(self, ctx, dirPfn, batchSize=1000): |
where the script could get a list of files to be cleaned up from the RSE, possibly by using the existing filedumps.
Describe alternatives you've considered
we've considered adding live directory listing on the RSE side, but this would be unfeasible and very inefficient as we're a large object store and this would involve creating an external metadata management layer.
Additional context
Add any other context or screenshots about the feature request here.
Impact of the new feature
MSUnmerged will be able to cleanup RSEs which do not support directory listing (ls) functionality
Is your feature request related to a problem? Please describe.
Currently there's a lot of historical unmerged data left over at the RAL RSE. These are not cleaned up by the MSUnmerged script as we're an object store and do support directory listings.
Describe the solution you'd like
Could you create a fallback function for
WMCore/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py
Line 999 in 374cf8e
where the script could get a list of files to be cleaned up from the RSE, possibly by using the existing filedumps.
Describe alternatives you've considered
we've considered adding live directory listing on the RSE side, but this would be unfeasible and very inefficient as we're a large object store and this would involve creating an external metadata management layer.
Additional context
Add any other context or screenshots about the feature request here.