Concurrency Gates

In my work as a support engineer at Buildkite, I often field questions about concurrency. Many of our customers — often larger enterprises with really large builds — want to run multiple pipelines with heavy parallelism in some sections, and enforced ordering in others.

One common issue I encounter is when the customer wants to lock deployment either to an environment, or to end-to-end tests, depending on whether the other is currently executing. One workaround is setting the same concurrency group for both steps, however, doing so removes parallelism from end-to-end tests, and takes away the ability to run multiple build agents.

A customer recently framed the issue this way:

• I want to run Buildkite in a pipeline
• Per branch/build I want to run X number of tests in parallel
• Globally, I want the tests to run in a specific order, to preserve the order of deployments

Buildkite is well suited to addressing concurrency issues as it has complete job isolation and makes it easy to run multiple agents.

The way I recommend resolving this issue is having a single build running using concurrency groups like a gate, and running those deployments and tests in between the gates. This involves using a command step to demark the start and end of the gate.

Visually, it looks like this:

Here’s how to create a concurrency gate:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
steps:
  - command: echo "Running unit tests"
    key: unit-tests

  - command: echo "--> Start of concurrency gate"
    concurrency_group: gate
    concurrency: 1
    key: start-gate
    depends_on: unit-tests
	
  - wait
	
  - command: echo "Running deployment to staging environment"
    key: stage-deploy
    depends_on: start-gate

  - command: echo "Running e2e tests after the deployment"
    parallelism: 3
    depends_on: [stage-deploy]
    key: e2e

  - wait

  - command: echo "End of concurrency gate <--"
    concurrency_group: gate
    concurrency: 1
    key: end-gate

  - command: echo "This and subsequent steps run independently"
    depends_on: end-gate

Here it is in action:

Here, you can see three builds that run in parallel. However, when the left build starts running e2e tests (inside the concurrency gate), the next build will wait for the current one to finish those tests before running its own e2e tests. It's a way to run things in parallel but have the concurrency gate impose an order.

Simplified, the jobs which are part of a concurrency group within a single build which are eligible to run must all finish before another build’s jobs in that concurrency group are permitted to run.

Your end state is that builds have completed in the order they were created, but the time for each build has been reduced, leveraging the ability to run those tests in parallel.