RetryManager crashes for every random pkl load failure

[LinaresToine]

Impact of the bug
WMAgent --> RetryManager Component of the agent

Describe the bug
Every once in a while we see that the retry manager crashes while loading the job report for a job in cooloff. The error message narrows down to a pickle data was truncated.

How to reproduce it
Steps to reproduce the behavior:
No idea

Expected behavior
The job should either retry a maximum of three times in the case the job report is not accessible.

Additional context and error message
Every job that gets retried needs to unpickle the job report to check the exit code. In the case the unpickling of the report fails, then the retry manager crashes, which is a blocker for other jobs that require a retry. The solution to this should be to allow to retry without checking the exit code in case that unpickling the Report.pkl fails.

[LinaresToine]

The error message is:

2025-07-24 12:02:59,138:140364071564864:ERROR:RetryManagerPoller:Exception while checking for cooloff timeout for job 2417168
pickle data was truncated
2025-07-24 12:02:59,170:140364071564864:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMComponent.RetryManager.RetryManagerPoller.RetryManagerPoller object at 0x7fa91febf920> <@========== WMException Start ==========@>
Exception Class: RetryManagerException
Message: Exception while checking for cooloff timeout for job 2417168
pickle data was truncated
        ClassName : None
        ModuleName : WMComponent.RetryManager.RetryManagerPoller
        MethodName : selectRetryAlgo
        ClassInstance : None
        FileName : /data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/WMComponent/RetryManager/RetryManagerPoller.py
        LineNumber : 274
        ErrorNr : 0

Traceback: 
  File "/data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/WMComponent/RetryManager/RetryManagerPoller.py", line 266, in selectRetryAlgo
    if plugin.isReady(job=job, cooloffType=cooloffType):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/WMComponent/RetryManager/PlugIns/PauseAlgo.py", line 47, in isReady
    report.load(reportPath)

  File "/data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/WMCore/FwkJobReport/Report.py", line 769, in load
    self.unpersist(filename)

  File "/data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/WMCore/FwkJobReport/Report.py", line 411, in unpersist
    self.data = decodeBytesToUnicode(pickle.load(handle))
                                     ^^^^^^^^^^^^^^^^^^^

<@---------- WMException End ----------@>  File "/data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 183, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/WMCore/Database/DBExceptionHandler.py", line 43, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/Utils/Timers.py", line 57, in wrapper
    res = func(*arg, **kw)
          ^^^^^^^^^^^^^^^^
  File "/data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/WMComponent/RetryManager/RetryManagerPoller.py", line 161, in algorithm
    self.doRetries()
  File "/data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/WMComponent/RetryManager/RetryManagerPoller.py", line 296, in doRetries
    self.processRetries(jobs, 'job')
  File "/data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/WMComponent/RetryManager/RetryManagerPoller.py", line 205, in processRetries
    propList = self.selectRetryAlgo(jobList, cooloffType)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/tier0/WMAgent.venv3/lib64/python3.12/site-packages/WMComponent/RetryManager/RetryManagerPoller.py", line 274, in selectRetryAlgo
    raise RetryManagerException(msg)

2025-07-24 12:02:59,170:140364071564864:INFO:Harness:>>>Terminating worker threads

[LinaresToine]

For illustration purposes, this is how I am currently handling the problem: LinaresToine#15

I recon this was the obvious way, but still requires manual input of each problematic job. I could add a similar procedure in the exception block.

[anpicci]

Hi @LinaresToine , I would say if your fix works, feel free to make a PR against the WMCore repo and call for a review. I don't think we can dedicate some effort to this issue

[LinaresToine]

Thanks @anpicci. My fix is rather temporary, and definitely non sustainable. For now it helps us as operators fix fires quickly, but a more thorough solution needs to be implemented. Once I have a clearer idea of such solution, Ill make the PR to WMCore repo.