Feature request: concurrency to queue all jobs waiting on a group

[Shereef]

Current behavior

Job A arrives, finds nothing in its concurrency group and runs
Job B arrives, finds job A running in its concurrency group, looks for anyone else waiting (with the intent to kill them), finds nothing, and waits.
Job C arrives, finds job A running in its concurrency group looks for anyone else waiting (with the intent to kill them), finds job B, kills it, and waits.
Job A finishes, job C is started.

Desired behavior

• Job A arrives, finds nothing in its concurrency group and runs
• Job B arrives, finds job A running in its concurrency group, and waits.
• Job C arrives, finds job A running in its concurrency group, and waits.
• ....etc all jobs just wait in line

Sorry if this was discussed before I tried to search but didn't find anything.

Our current implementation

To give it more context:

• We have 5 team environments
• PRs push to the author's team env
• After merge main has a github action to push to all team environments
• the groups are unique to the team env it pushes to
  • We can't deploy to the same env from 2 github actions that's why concurrency is based on envs

Imagine this scenario:
We have 5 PRs for Team A
We merge to main and update the 5 PRs
All 5 will want to deploy to the env + the main branch deploy that is ongoing
the last PR to merge is going to be queued
all other PRs will be cancelled and have to be retried manually

Suggested solutions

one of the below would work

• cancel-in-queue: false a new command to disable cancelling queuing jobs
• cancel-in-progress: false would not cancel anything and leave everything in queue

[ethomson]

Hi @Shereef - thanks much for the feedback. Our initial design was focused on deployment scenarios, but we want to address the use case that you've outlined. I appreciate you outlining this so nicely and providing these suggestions. I hope that this is an area that we can address in the future.

Thanks again.

[Tarasovych]

Any updates on this?

[miguelb-gk]

Any updates on this?

[finolacahill]

@chrispat @tuves
Is anyone following this highly desirable feature since ethomson left Github?

[finolacahill]

@chrispat @tuves Once again asking if anyone is following up on this feature since ethomson left Github?

[finolacahill]

@ankneis could you see if anyone has anyone taken this on since ethomson left Github please?

[JimiC]

The desired behavior here, I believe, would be for jobs with the same run_id to be canceled (either in_progres or in_queue), whenever a new incremental run_id is present on the same ref.

[justin-pierce]

The OP is asking for jobs/workflows to wait and execute sequentially instead of canceling. Canceling existing workflows when a new workflow is triggered is already supported:

https://docs.github.com/en/[email protected]/actions/using-jobs/using-concurrency

When a concurrent job or workflow is queued, if another job or workflow using the same concurrency group in the repository is in progress, the queued job or workflow will be pending. Any previously pending job or workflow in the concurrency group will be canceled. To also cancel any currently running job or workflow in the same concurrency group, specify cancel-in-progress: true.

If your issue is specific to manually rerunning the same workflow, seems like a separate issue.

[JimiC]

The said behavior is problematic as it's canceling it's own jobs in the same run. I have 6 jobs to run in each commit and they cancel each other in the same run.

[justin-pierce]

Can your issue be solved by using concurrency on the workflow level instead of the job level? Afaik concurrency on the job level is meant more for branching paths (like their fallback value example). Sorry if I'm misunderstanding your usecase though

[JimiC]

Any previously pending **job** or workflow in the concurrency group will be canceled this is what's problematic.
And because jobs of the same commit share the same group they get canceled.

[nelson6e65]

I think I just need the same, but add to a queue instead of cancel the workflow. I meant, don't run un parallel but in serial instead

This will help, for example, use the same build result or cache in multiple runs for the same commit. The first one will create the artifacts/cache and the next one will be faster

[JimiC]

That will complicate things even further.

[Anonymous-Coward]

Cache management is orthogonal to concurrency. Cache management across multiple jobs is something the workflow writer has to think about. Controlled concurrency among jobs in the same concurrency group, which is what this feature request is about, is not possible with what GHA provides out of the box.

[justin-pierce]

Was surprised to see that canceling previous runs was the only option officially supported -- waiting and executing sequentially is what I need and seems to me like a clear usecase (canceling a deployment could result in a broken state)

So far I've found this Action: https://github.com/ahmadnassri/action-workflow-queue

But ideally, workflows sitting waiting for previous workflows to finish wouldn't consume action minutes in an official feature.

[JimiC]

What I needed was exactly what https://github.com/styfle/cancel-workflow-action provided

[mro-pbraswell]

We implemented something similar with a run step, but that means an entire self-hosted runner is occupied while it's waiting for earlier workflow runs to finish.

It's disappointing that the choice is between scaling up our self-hosted runners or surpassing our build minutes quota for github-hosted runners or building a separate queue outside of github actions to handle this kind of functionality.

[duro]

We have a long running workflow that goes through several jobs to deploy AWS infrastructure. If one of them was cancelled mid-way, or deploys happened on to of one another, this would cause some pretty serious issues.

We definitely need a way to queue jobs in sequence without consuming action minutes

[koreyGambill]

Brainstorming - Eventually I will probably have to implement a specific self-hosted group. Maybe like 25 runners on a t3.small. Their job would be to handle queues. So I'd have a workflow or job dedicated to queing, and my larger runners (c7a.xlarge) wouldn't have to sit and wait for 15 mins at a time - instead I can have a single t3.small for all repos.

[liry]

@justin-pierce the action-workflow-queue has an issue when there are multiple workflows started in the same second. See #45.

@koreyGambill exactly, but do you know about some simple yet reliable queuing solution for that?

[kmorkans-lumoa]

Would be great to be able to queue workflow runs (ideally without consuming action minutes). Was surprised that new runs cancel existing ones by default.

[ashwinv11]

We hit this limitation while testing out the Merge Queue feature that is currently in limited beta. Our use case is ensuring that only one workflow is running for the PR at the top of the merge queue. When concurrency is set at the workflow level (for the merge_group event type) and multiple PRs are queued within the merge queue, the system cancels queued check suites once a new PR is added to the queue. It would be fantastic if we had the ability to disable canceling queued jobs!

[akagr]

If you're using self-hosted runners, a workaround/hack we used was to label one of these runners specifically, and then make all the jobs that need to be in a concurrency group just run on this labelled runner instead. No concurrency shenanigans. That causes them to queue up globally across all workflows without any caps on number of waiting jobs.

The only major limitation is that if this runner fails, these jobs won't use any other runner. That was considered an acceptable risk to unblock us until we had a better solution.

[liry]

Well, this is possible only for cases, where you need to serialize only a single job.
Once you have a workflow where multiple jobs need to finish in the given order before new workflow is started (e.g. job-A: deploy to test env, job-B: run tests against the env), then it's not very well usable.

[AvivPeledTalon]

My team also needs this feature.

[clamattia]

My team also needs this.

[MitchellPaff]

My team also need this feature - mainly for queueing mulitple actions each with Terraform plan/deploy steps which cannot be run at same time due to a lock on terraform state.

[galz-cyera]

A must have for my team

[yershalom]

Really hard to run terrafrom/terragrunt at scale without it

[badbort]

Really needing this to be implemented.

I am currently using https://github.com/ben-z/gh-action-mutex for now, but this does pollute the git history.

[dumganhar]

Need this feature too.

[bob-ashforth-sp]

GitHub, I'll add my team's need for this feature as well. I see a LONG list of replies like this, with no response from GitHub. (I'm in DevOps.)

I'm going to put in a GitHub support ticket referencing this request to see if there are any plans to implement it.

Our use cases is that I have created a highly parallelized building and deploying four distinct services in a monorepo. In this workflow, the update_service action is called by the jobs for each of those services, AND we have a very active CI/CD environment which can be processing several merges almost simultaneously.

While I am discussing the use of the current concurrency model (cancel all pending workflow runs except the latest), until and unless that gets the green light our workflow runs will continue to risk colliding and corrupting the shared deployment environment with competing service updates.

[gnuman]

Thanks for the feature request, we need this feature

[davegraham-sage]

+1

Our use case:

• Pipeline builds large legacy desktop application
• It can optionally build several subcomponents in separate reusable workflows dependent on what changes there are to those components
• Each component may commit version number increments and amended library files back to the repo
• Sometimes those components reach their commit jobs at the same time and interfere with each other, causing git commit issues even though they each Pull > Commit > Push due to race conditions
• Concurrency locks could potentially stop one of the commits occurring if three run at the same time

Controlling the queue and allowing more pending jobs would be one way to get around this. We're investigating other options

[nicola-lunghi]

Any news???
Our use case:

• we have one runner that runs tests on an hardware board. we have one step where we program the board and another one where we run the tests. those jobs runs on different runner (programming has to be done from another host)
• the issue is that we need to lock the job so if I'm running the testing phase I don't run another job with the programming (destroying my tests)

the solutions using concurrency would be:

concurrency:
my hw concurrency
cancel-in-progress: false
but this doesn't work as concurrency always act as is it was "cancel-in-progress: true"

[cmyllis]

but this doesn't work as concurrency always act as is it was "cancel-in-progress: true"

It doesnt exactly works that way. You can trigger 2 jobs, but when you trigger a third, the second will be cancelled. If you have it as true, the first will be cancelled when you trigger a second.

[pchico83]

This is not what I am seeing. The in progress job is always cancelled even if I set ""cancel-in-progress: false".
Note that I am setting "cancel-in-progress: false" at the workflow level, not at the job level

[cmyllis]

@pchico83 This is how it is set in mine.

`concurrency:
group: load_tests
cancel-in-progress: false

run-name: Running Test
jobs:
setup-test:`

[dannysauer]

I voted on the issue, but the flood of plus one comments seems pretty impressive too. So, "hey, I want multiple overlapping jobs to queue and run serially instead of terminating" as well. I'm pretty sure this is possible, because one job will queue and multiple jobs waiting on runners will queue. So all the necessary logic seems like it's essentially present somewhere. 😅

[fetwar]

The more I try use GitHub actions for advanced use cases the more I stumble onto community conversations like this.

• seemingly simple to implement
• large number of users requesting the feature
• support ticket/s opened
• users frequently trying to get any word from GitHub staff through multiple different methods

Yet, years later and still... nothing... Not so much as a reply with whether the feature is planned, on the roadmap at all or if this is something we should just learn to live with.

[Felixoid]

Actually, in this case, the silence is a very clear answer.

It looks to me like Actions isn't the first class citizen, and it's not planned to change the state any soon.

[oferqm]

+1

[jeliasson]

Stop it. This thread don't need the spam that +1 invokes.

[LiteBrite82]

@jeliasson is correct, please remember to use the upvote button or an emoji instead of creating a new thread to agree with a comment. It keeps the discussion clean.

Thank You community!!!

[nebuk89]

Hey, sorry the Actions team has been quiet on this. We have added this to the list of paper cuts we are hoping to address in the next 12 months :) I don't have a timeframe for this one, but I want to say we know it's a pain point and do have it on our list <3

[bob-ashforth-sp]

Good to see a reply, @nebuk89 . This is really a LOT more than a paper cut; it's a fact that many workflows are incrementally changing/updating things like databases, and MUST run in sequence. The current design presumes that later workflows are just an addition to the same codebase, and so are expected to produce the same result whether or not an intervening run is cancelled.
I hope that you can spread this perspective to everyone involved in prioritizing work.
I and probably everyone who has chimed in here have been some combination of shocked, annoyed, and even angry that our requests have not resulted in getting this problem fixed. I really hope that you can help!

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[hschimpf]

Hi all! I have a more complex desired behavior:

We have a step that deploys with terraform. We want that if a running job already reached that step, the other ones should wait instead of killing it. Ex:

• Job A reaches, nothing in-progress, starts
• Job B reaches, finds A in-progress but not at deployment step, kills A and starts
• Job C reaches, finds B in-progress and already at deployment step, waits for B to finish
• Job D reaches, finds B in-progress, but also C in-progress (waiting), should kill C since it not reached deployment step, waits for B to finish
• Job B finishes
• Job D continues
• Job D finishes

Deployment succeeded, version B deployed and then replaced with version D

[iusildra]

@hschimpf what about merge queues ? For a TF deploy it seems better to me to run the full deploy for each commits

[ryeleo]

Per original request: my team needs something like cancel-in-queue: false directive to enable queuing workflow runs that cannot be concurrent.

cancel-in-queue: false a new command to disable cancelling queuing jobs

Motivating Example

I have a 'deploy' GitHub Workflow that takes 600 seconds to run a deployment.
I have set the concurrency directive up with group: prod-deployment and cancel-in-progress: false
I have 3 pull requests (PRs: PR1, PR2, PR3).
All 3 of these PRs are merged within a 30 second time window:

1. PR1 merges at 12:00:00
2. PR2 merges at 12:00:15
3. PR3 merges at 12:00:30

Requirements

1. My GitHub Workflow must run for each PR.
2. My GitHub Workflow cannot be run concurrently.

Expected Behavior

1. PR1 is deployed,
2. PR2 is deployed, then
3. PR3 is deployed.

Documented Behavior

1. PR1 is deployed,
2. PR2 is cancelled, then
3. PR3 is deployed