Runners not picking up workflows for Kata Containers

[BbolroC]

Hi, it seems workflows for Kata Containers have been stuck in the queue for 7 hours because a runner is not picking them up.

Here is an example workflow:

https://github.com/kata-containers/kata-containers/actions/runs/21653115636/job/62427962001

I look forward to any feedback from you. Thanks!

[BbolroC]

FYI: @stevenhorsman @Amulyam24

[stevenhorsman]

Some screenshots added for this in #77 in case it helps

[rahulssv-ibm]

@BbolroC @stevenhorsman, thank you for reporting this issue.

This was an edge-case scenario that occurred due to a mismatch in the number of containers we created per job. The issue happened during a transition where the “Allow all public repositories” permission was enabled in the default runner group.

At that time, some jobs had already been requested and containers were created for them. However, because of the permission change, those running jobs remained idle. Our system is designed to automatically clean up containers if a job is not picked up within 2 minutes, in order to free up capacity (since we operate with limited resources). During the transition, newer jobs were picked up successfully, while the earlier idle jobs were cleaned up — resulting in the container mismatch you observed.

Temporary workaround:
Please trigger a few new jobs on our runners and cancel them within 5 seconds, ensuring they are cancelled before execution begins. This will create additional containers on our end and rebalance the container count.

We are actively working on improving our handling of such edge cases to prevent this from happening again. You can expect improvements in upcoming updates.

Thank you for your understanding, and please let us know if you notice anything further.

[stevenhorsman]

Hi @rahulssv-ibm - thanks for the response. I tried your workaround and we seem to be able to use multiple runners now, so thank you!

[rahulssv-ibm]

Update: Permanent Fix Implemented

We have implemented a permanent fix for the issue where runners were not picking up workflows.

The Fix: Instead of immediately cleaning up jobs that encounter edge-case scenarios (such as permission transitions), we now retain them in a wait queue for up to 24 hours. This approach effectively resolves the container count mismatch that was causing the issue.

Deployment Plan: These changes have been successfully tested. We will be deploying this as a hotfix during our scheduled February maintenance next week.