Feature Request: Automatically switch to Github runner if self hosted runner is not available

[aditi-shah-st1]

Describe the enhancement
Add support in Github action which will pick Github runner if self hosted runner is not available automatically.
e.g. Setup single self hosted runner. If it is available then pipeline should run using self hosted runner and if it is not then pipeline should use Github Runner and complete its job.

[CoreCypator]

I was looking for something like this as well..

Something like:
runs-on: [self-hosted, linux, MyLabel]

And GH does not need to differentiate between Self Hosted or GH Runmers... Use the priority in the list above to whatever labeled Runner is available

[Christopher-C-Robinson]

What?

[aditi-shah-st1]

@CoreCypator Hey can you please tell us how to give priority in the list above?
Routing precedence of self hosted runner see this list as a combination of labels.

In this example, a self-hosted runner that matches all three labels will be eligible to run the job:
runs-on: [self-hosted, linux, ARM64]

[josiahwitheford]

There could be a fallback-runs-on parameter that looks the same as the runs-on property, but runs-on will be tried first and if no runners are online then the fallback-runs-on runners would be tried. Then the 60 minute timeout could kick in after that?

This would be quite a cool feature that I could see being used as a way to spin up local runners for offsetting build costs etc.

[cocoa-xu]

I second this! In my case, I want this feature because my self-hosted runner may be offline (for maintenance or whatever reasons), and it would be really nice if these queued CI jobs could automatically switch to GitHub runners after some time.

[murphman300]

Hey guys ... so.. has this been picked up yet or considered?

[ChristopherHX]

Did you ever try to assign your self-hosted runner to GitHub hosted runner labels like ubuntu-latest, macos-latest and windows-latest?

I observerd these priorities if you use ubuntu-latest as self-hosted runner label (This may not work for autoscaler)

• A self-hosted runner with custom label ubuntu-latest is available => send job to your self-hosted runner
• No self-hosted runner is connected with custom label ubuntu-latest => start your job in a GitHub Hosted Runner

See https://docs.github.com/en/actions/hosting-your-own-runners/using-labels-with-self-hosted-runners for how to label self-hosted runners.

There are no fancy timeout or other mechanism, GitHub Hosted Runners are also used if you have more parallel jobs than self-hosted runners

EDIT
An example workflow to test this.

on: push
jobs:
  _:
    runs-on: ubuntu-latest # This might run on your self-hosted runner if it is online and idle
    steps:
    - run: uname -a

[mmospanenko]

It works! Thank you! =)

[anthonyalayo]

does this still work? I tried this out today and it picked the shared runner over mine.

[jbcaillau]

@anthonyalayo apparently not. seems to previously there was a fallback to github shared runners, not anymore. take a look at
https://github.com/jimmygchen/runner-fallback-action

[phraemer]

For us, the important point of such a feature would be that it wouldn't switch to a Github runner if the self-hosted runners are merely busy. Only if they are not online at all.

[ChristianVermeulen]

Exactly this, we have limited capacity on our local runners which we want to fully take advantage of. If our runners are at full capacity it should not keep queueing up but fall back on the github runners so jobs can keep running.

[jbcaillau]

have a look above. should be OK to update this fallback action to indeed fall back to github shared runners whenever self-hosted ones are not only offline but also active, etc.

[jsoref]

While I'd love to have this, there is something that repository owners can do while they wait for something better:
https://github.com/commercialhaskell/stack/blob/8fd916a3941243ca3ac559390e5ea84af5aef037/.github/workflows/integration-tests.yml#L150-L181

I'm slowly offering variations of this pattern to public repositories as I trip on workflows that require self-hosted runners.

[ianpurton]

I've had some success using this pattern...

jobs:
  # Select a self hosted runner if the username matches else fallback 
  # to ubuntu-latest
  choose-runner:
    runs-on: ubuntu-latest
    outputs:
      runner: ${{ steps.runner.outputs.runner }}

    steps:
    - id: runner
      run: |
        if [ "${{ github.actor }}" == "ianpurton" ]; then
          echo "::set-output name=runner::self-hosted"    
        else
          echo "::set-output name=runner::ubuntu-latest"
        fi

  build:
    needs: [choose-runner]
    # Will run on 'ubuntu-latest' or 'self-runner' based on the results of job `choose-runner`
    runs-on: ${{ needs.choose-runner.outputs.runner }}
    ...
    ...

Replace your-user-name with your github username.

This will run self-hosted for you and ubuntu-latest for everyone else.

Ideally, I'd like a way to detect that my runner is online, but I couldn't figure out how to do that.

Is there a way to get a list of online runners?

[jsoref]

Fwiw, I looked and didn't find a way in the v3 API. (I don't have the energy to check the graphql API.)

Assuming you have some infrastructure hosting this, you could probably expose an endpoint of your own to work around this deficiency.

But, it'd definitely be nicer if someone (it seems like you're on the road to this) could wrap what you've done into a GitHub action, and thus it'd be better if the answer could be retrieved from a GitHub API.

You might try asking support.github.com, occasionally they're helpful.

[jimmygchen]

Thanks for the idea, looks like there's now an API to do this.

I've just created a basic GitHub action based on the above pattern, still in super early stage but I think it should work!
https://github.com/marketplace/actions/check-runner-availability

[advissor]

Thanks @9876691 for this :) It works

[IcebergRage]

Hello ! Any news on this ? :)

Like @CoreCypator said, this would be perfect
runs-on: [self-hosted, linux, MyLabel]

[raghavTinker]

Hi!
Any update on this thread?

[nftushar]

The enhancement proposes the addition of automatic support in GitHub Actions to utilize a GitHub runner if a self-hosted runner is unavailable. Specifically, it suggests setting up a scenario where a single self-hosted runner is configured. If this runner is accessible, the pipeline will utilize it; however, if the self-hosted runner is not available, the pipeline will seamlessly switch to using a GitHub runner to complete its tasks. This approach ensures smooth workflow execution even in cases where the preferred self-hosted runner is not accessible.

[pkeogan]

Answer from a github staff member would be nice!

[AsparagusEduardo]

Any news on this?

[StefaniOSApps]

Dynamically Selecting GitHub Action Runners Based on Availability 🚀

Hello everyone,

I wanted to share a solution for dynamically selecting GitHub Action runners based on their availability and status. This can be particularly useful if you have self-hosted runners and want to fall back to GitHub-hosted runners when none of your self-hosted runners are available.

Here’s a simplified example of how you can achieve this:

jobs:
  check-runner:
    runs-on: ubuntu-latest
    outputs:
      runner-label: ${{ steps.set-runner.outputs.runner-label }}

    steps:
      - name: Set runner
        id: set-runner
        run: |
          runners=$(curl -s -H "Accept: application/vnd.github+json" -H "Authorization: token ${{ secrets.REPO_ACCESS_TOKEN }}" "https://api.github.com/repos/${{ github.repository }}/actions/runners")
          available=$(echo "$runners" | jq '.runners[] | select(.status == "online" and .busy == false and .labels[] .name == "self-hosted")')
          if [ -n "$available" ]; then
            echo "runner-label=self-hosted" >> $GITHUB_OUTPUT
          else
            echo "runner-label=ubuntu-latest" >> $GITHUB_OUTPUT
          fi

  build:
    needs: check-runner
    runs-on: ${{ needs.check-runner.outputs.runner-label }}

    steps:
    - name: Checkout code
      uses: actions/checkout@v3

    - name: Build code
      run: echo "Building on runner ${{ needs.check-runner.outputs.runner-label }}"

Key Points:

1. Special Access Token: Ensure you have a special access token (REPO_ACCESS_TOKEN) with the appropriate permissions set in your repository secrets to access the GitHub API.

2. Debugging: You can add -w "%{http_code}" to the curl command to print out the HTTP status code for debugging purposes.

3. Dynamic Runner Selection: The runner is dynamically selected based on the availability and status of the self-hosted runners. If none are available, it automatically falls back to the GitHub-hosted runner.

Stay productive and healthy, everyone! Ensure your codes are as clean as your hands, and may your builds be as robust as your immune system. Happy coding!

Best regards,
Stefan ✌️

[jsoref]

I expect you can (should) use a fine-grained taken token: https://docs.github.com/en/rest/overview/permissions-required-for-fine-grained-personal-access-tokens?apiVersion=2022-11-28#actions

1. Visit https://github.com/organizations/{your-org}/settings/personal-access-tokens
2. If you haven't run through the wizard to enable fine-grained tokens, do so
3. If you have, make sure they're enabled
4. Visit https://github.com/settings/personal-access-tokens/new
5. Fill out the Token name, Expiration, and Description
6. In Resource owner, select {your-org}
7. Expand Organization permissions
8. For Self-hosted runners, select Read-only
9. Select Generate token

[seriousm4x]

That's the link for your account settings, not your org. That's where I did it.
https://github.com/settings/tokens?type=beta

[jsoref]

For me clicking the generate new token link on https://github.com/settings/tokens?type=beta took me to https://github.com/settings/personal-access-tokens/new

But some of my organizations didn't have new tokens enabled so I had to enable them first...

[rekha-prakash-maersk]

Hello @StefaniOSApps , just curious about GitHub API rate limiting here. if we use enterprise level PAT token created from GitHub App, is there any rate limiting we might hit for GitHub API requests ? we have ~ 300 jobs running at any given minute.

[jsoref]

@StefaniOSApps I should have flagged run: ... ${{ ... }} earlier...

Best practices would be something closer to this for that first step:

      - name: Set runner
        id: set-runner
        env:
          REPO_ACCESS_TOKEN: ${{ secrets.REPO_ACCESS_TOKEN }}
        run: |
          runners=$(curl -s -H "Accept: application/vnd.github+json" -H "Authorization: token $REPO_ACCESS_TOKEN" "$GITHUB_API_URL/repos/$GITHUB_REPOSITORY/actions/runners")
          available=$(echo "$runners" | jq '.runners[] | select(.status == "online" and .busy == false and .labels[] .name == "self-hosted")')
          if [ -n "$available" ]; then
            echo "runner-label=self-hosted" >> "$GITHUB_OUTPUT"
          else
            echo "runner-label=ubuntu-latest" >> "$GITHUB_OUTPUT"
          fi

$GITHUB_API_URL and $GITHUB_REPOSITORY are documented in https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/store-information-in-variables#default-environment-variables

[albertotb]

What about the other way arround? Switching to a self-hosted runner if the spending limit has been reached? Would this work the same way?

[coopbri]

Curious about this as well! There might be a nice workaround to do this right now but I'm not sure. Worst case, can commit changes to the workflow files to toggle it (could automate this process of course, e.g. based on querying spending limit from GitHub API).

Probably could tweak @StefaniOSApps' solution above to make this work as well, just switch the conditions/redundancy constraints.

[jupe]

This is relevant also for me. What about if runner selection could be even more generic, something like;

runs-on: 
   - labels: [self-hosted, linux-fast-machine]     # try to pick this runner first
     timeout-minutes: 0                                        # timeout for such runner
   - labels: [self-hosted, linux-low-machine]      # if not available, pick this runner then
     timeout-minutes: 0
   - labels: [ubuntu-latest]        # if above not available, pick this runner last
     timeout-minutes: 10 

In above example we would have 3 different kind of runners, 2x self-hosted ones, one more powerful and one slower, and for backup plan we can use GitHub hosted runners. More powerful would be most efficient from job point of view, but if it's busy we can still go forward with slower options.

This would allow to kind of priorities runners more generic way and configure how long time job can wait runner for each group.

Propose here could be option when number of affected jobs and runners are rather small, but when workflows spread to several teams and hundreds of files, it might be too hard to maintain.

[lkfortuna]

Hey there from Actions 👋 To provide an update here, we do not have this on our prioritized list for this year. It seems like there are some workarounds proposed here in the thread that work. Just to set expectations, we do not currently have plans to support this as a first party feature.

[mmospanenko]

Yeah, marking as 'ubuntu-latest' is works, and it looks as good solution, but how "legal" it is relatively to product's politics/visions? How long it will be working, in other words :)

[malmi]

It's not working anymore

[Christopher-C-Robinson]

What @malmi said... Not working anymore for me either... Anyone know of another way?

[malmi]

The extra step on the github action to set the runs-on, depending on the github api call, still works for me. See this answer: https://github.com/orgs/community/discussions/20019#discussioncomment-7173595.

[Christopher-C-Robinson]

@malmi this requires at least 1 min of GitHub hosted agent every time right? Or am I understanding wrong?

[ptagl]

@StefaniOSApps answer (here) is amazing.

I used a slightly different approach to better suit my case.

First of all, I wanted to avoid relying on a personal token, so went through the "organization app" path:

• Organization Settings => Developer settings => New GitHub App
• Created a new app by enabling only Self-hosted runners (read-only) under Organization permissions
• After the creation, I created a new private key for the application and stored the pem file locally
• Also, I took note of the App ID

Then I went to the repository that I wanted to configure with the conditional CI workflow

• Settings => Secrets and variables => Actions
• Added a new secret called RUNNERS_PRIVATE_KEY and stored the content of the PEM file
• Added a new secret called RUNNERS_APP_ID and stored the App ID I mentioned above

The final step is to update the GitHub actions workflow with something like this:

(edited after @jsoref feedback)

jobs:
  check-available-runners:
    runs-on: ubuntu-latest
    outputs:
      runner-type: ${{ steps.set-runner.outputs.runner-type }}

    steps:
      - name: Generate a token for reading available runners
        id: generate-token
        uses: actions/create-github-app-token@v2
        with:
          app-id: ${{ secrets.MY_APP_ID }}
          private-key: ${{ secrets.MY_PRIVATE_KEY }}
          owner: ${{ github.repository_owner }}

      - name: Check available runners
        id: set-runner
        env:
            AUTH_TOKEN: ${{ steps.generate-token.outputs.token }}
            ORGANIZATION: ${{ github.repository_owner }}
        run: |
          runners=$(curl -s -H "Accept: application/vnd.github+json" -H "Authorization: token $AUTH_TOKEN" "https://api.github.com/orgs/$ORGANIZATION/actions/runners")
          available=$(echo "$runners" | jq '.runners[] | select(.status == "online" and .busy == false and .labels[] .name == "self-hosted")')
          if [ -n "$available" ]; then
            echo "runner-type=self-hosted" >> $GITHUB_OUTPUT
          else
            echo "runner-type=ubuntu-latest" >> $GITHUB_OUTPUT
          fi

  main-job:
    needs: check-available-runners
    runs-on: ${{ needs.check-available-runners.outputs.runner-type }}

[jsoref]

as in https://github.com/orgs/community/discussions/20019#discussioncomment-13391493, run:...${{...}} isn't aligned with best practices. The step should be written something like this:

      - name: Check available runners
        id: set-runner
        env:
           AUTH_TOKEN: ${{ steps.generate-token.outputs.token }}
        run: |
          runners=$(curl -s -H "Accept: application/vnd.github+json" -H "Authorization: token $AUTH_TOKEN" "$GITHUB_API_URL/orgs/$GITHUB_REPOSITORY_OWNER/actions/runners")
          available=$(echo "$runners" | jq '.runners[] | select(.status == "online" and .busy == false and .labels[] .name == "self-hosted")')
          if [ -n "$available" ]; then
            echo "runner-type=self-hosted" >> "$GITHUB_OUTPUT"
          else
            echo "runner-type=ubuntu-latest" >> "$GITHUB_OUTPUT"
          fi

[ste26054]

It really is a bummer that there is no easy way do define priorities between runners. Per doc

If GitHub finds an online and idle runner that matches the job's runs-on labels and groups, the job is then assigned and sent to the runner.

It used to route primarily jobs to a self hosted runner, then to Github hosted runners (this is what worked for us for the past year) but it does not work anymore 😕. So we have to pay extra $$$ to Github to have a runner that has decent specs (but still with poor cache performance).

[Shubham-cyber-prog]

This would be a very practical and much-needed improvement to Github Action
Currently , When you specify

runs-on:[self-hosted 

[JerryVDP]

3.5 years later, still no easy way to prioritize runners… bummer

[mikehardy]

@jimmygchen made a great "runner fallback action" which I've personally adopted in my CI.

Jimmy has archived his action so I'm considering my fork to be the primary now if people want a packaged up action that allows you to:

• check for availability primary runners in your repo, org, or enterprise
• use those if you can, or fallback to a different runner tag if your primary runner(s) aren't available

https://github.com/marketplace/actions/runner-availability-checker

Would be great if github supported it natively, but until then there is just enough API availability for a solution that works for me.

Cheers

[jbcaillau]

cc @srcansiz

[robtaylor]

Ah, it's gone, and your fork also @mikehardy . What happened?

[archon810]

The updated url is https://github.com/mikehardy/runner-fallback-action but for some reason the action is a 404 in the marketplace.