You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/guides/modules/orchestrate/pages/automatic-reruns.adoc
+98-23Lines changed: 98 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,9 +1,9 @@
1
-
= Automatic workflow reruns
1
+
= Automatic reruns
2
2
:page-platform: Cloud
3
3
:page-description: Configure automatic reruns for failed workflows to reduce manual intervention and improve pipeline reliability
4
4
:experimental:
5
5
6
-
Automatic reruns help reduce the impact of temporary failures in your CI/CD pipeline. When a workflow fails due to transient issues, CircleCI can automatically restart the failed workflow without manual intervention.
6
+
Automatic reruns help reduce the impact of temporary failures in your CI/CD pipelines. When a workflow fails due to transient issues, CircleCI can automatically restart the failed step or workflow without manual intervention.
7
7
8
8
Common use cases include:
9
9
@@ -12,23 +12,64 @@ Common use cases include:
12
12
* Working with spot instances that may be terminated unexpectedly.
13
13
* Preventing merge queue delays caused by transient failures.
14
14
15
-
[#introduction]
16
15
== Introduction
17
16
18
-
Automatic reruns provide a safety net for your CI/CD pipelines by automatically retrying failed workflows. This feature helps teams maintain productivity by reducing the need for manual intervention when workflows fail due to temporary issues.
17
+
Automatic reruns provide a safety net for your CI/CD pipelines by automatically retrying failed steps and/or workflows. Automatic reruns help teams maintain productivity by reducing the need for manual intervention when steps and workflows fail due to temporary issues.
19
18
20
-
The feature works by monitoring workflow completion status and automatically triggering a "rerun from failed" operation when <<rerun-criteria,specific conditions are met>>. This ensures that only failed jobs and their dependencies are retried, while successful jobs from the original workflow are not repeated.
19
+
When you configure automatic reruns, CircleCI monitors the status of your pipelines and reruns work as required.
21
20
22
-
Benefits of automatic workflow reruns include:
21
+
Benefits of automatic reruns include:
23
22
24
23
* Reduced manual intervention for transient failures.
25
24
* Improved pipeline reliability and developer productivity.
26
25
* Cost-effective retry strategy that only reruns failed jobs.
27
26
* Configurable retry limits to prevent infinite loops.
28
27
29
-
[#quickstart]
30
28
== Quickstart
31
29
30
+
This section provides config examples to help you set up automatic reruns. The content following this quickstart section details how the feature works, along with some troubleshooting guidance and frequently asked questions.
31
+
32
+
=== Automatic step reruns
33
+
34
+
To enable automatic reruns for a run step in a job, add the `max_auto_reruns` key to your step with a value between 1 and 5:
35
+
36
+
NOTE: Automatic reruns are only supported for `run` steps, not special steps like `checkout` or `setup_remote_docker`. Also you must configure the `command` key for the step, you cannot use the short form run step configuration.
37
+
38
+
.CircleCI config to automatically retry a step up to 3 times if it fails
39
+
[source,yaml]
40
+
----
41
+
version: 2.1
42
+
43
+
jobs:
44
+
my-job:
45
+
steps:
46
+
- run: echo "Hello, world!"
47
+
- run:
48
+
command: echo "This step will automatically rerun up to 3 times if it fails"
49
+
max_auto_reruns: 3
50
+
----
51
+
52
+
You can add a delay between the step failing and the automatic rerun starting, as follows:
53
+
54
+
NOTE: If you configure a delay for your automatic step rerun, the delay must be less than or equal to 10 minutes. The delay can be configured as a string with a unit suffix, using `m` (minutes) or `s` (seconds), for example, `auto_rerun_delay: 10s` or `auto_rerun_delay: 3m`. If you do not supply a delay, the rerun will start immediately after the step fails.
55
+
56
+
.CircleCI config to automatically retry a step up to 3 times if it fails with a 10 second delay between attempts
57
+
[source,yaml]
58
+
----
59
+
version: 2.1
60
+
61
+
jobs:
62
+
my-job:
63
+
steps:
64
+
- run: echo "Hello, world!"
65
+
- run:
66
+
command: echo "This step will automatically rerun up to 3 times if it fails with a 10 second delay between attempts"
67
+
max_auto_reruns: 3
68
+
auto_rerun_delay: 10s
69
+
----
70
+
71
+
=== Automatic workflow reruns
72
+
32
73
To enable automatic reruns for a workflow, add the `max_auto_reruns` key to your workflow with a value between 1 and 5:
33
74
34
75
.CircleCI config to automatically retry my-workflow up to 3 times if it fails
@@ -69,23 +110,33 @@ workflows:
69
110
[#how-automatic-reruns-work]
70
111
== How automatic reruns work
71
112
72
-
Automatic workflow reruns function similarly to manually clicking "Rerun workflow from failed" in the CircleCI interface. When a workflow fails, CircleCI evaluates whether the workflow qualifies for automatic rerun based on specific criteria, as described in the following section.
113
+
Automatic step reruns rerun a full failed step.
114
+
115
+
Automatic workflow reruns function similarly to manually selecting btn:[Rerun workflow from failed] in the CircleCI web app. When a workflow fails, CircleCI evaluates whether the workflow qualifies for automatic rerun based on specific criteria, as described in the following section.
116
+
117
+
When both automatic step reruns and automatic workflow reruns are configured together for the same job, the following rules apply:
73
118
74
-
[#rerun-criteria]
75
-
=== Rerun criteria
119
+
* The automatic step rerun triggers first.
120
+
* If the workflow still fails after all step rerun attempts, the automatic workflow rerun triggers.
76
121
77
-
For automatic reruns to trigger, all of the following conditions must be met:
122
+
=== Automatic rerun criteria
78
123
124
+
CircleCI monitors *workflow* completion status and automatically triggers a "rerun from failed" operation when the following criteria are met. This ensures that only failed jobs and their dependencies are rerun, while successful jobs from the original workflow are not repeated.
79
125
* The workflow status is "failed".
80
126
* The `max_auto_reruns` value is specified in the configuration.
81
127
* The number of remaining rerun attempts is greater than zero, and less than or equal to the specified `max_auto_reruns` value.
82
128
* The workflow is not a manual rerun.
83
129
* The pipeline is not older than 90 days.
84
130
131
+
CircleCI monitors *step* completion status and automatically reruns a failed step when the following criteria are met:
132
+
* The step status is "failed".
133
+
* The `max_auto_reruns` value is specified for the step in the configuration.
134
+
* The number of remaining rerun attempts is greater than zero, and less than or equal to the specified `max_auto_reruns` value.
135
+
85
136
[#rerun-behavior]
86
-
=== Rerun behavior
137
+
=== Automatic workflow rerun behavior
87
138
88
-
When an automatic rerun is triggered:
139
+
When an automatic *workflow* rerun is triggered:
89
140
90
141
* Only failed jobs from the original workflow are retried. If the previous failure blocked dependent jobs from running, these jobs are also run.
91
142
* Successfully completed jobs are not rerun.
@@ -99,23 +150,48 @@ CircleCI provides several ways to monitor and track automatic rerun activity.
99
150
[#ui-indicators]
100
151
=== UI indicators
101
152
102
-
Automatic reruns are indicated on the pipelines page in the CircleCI web app. In the Trigger event column you sill see *Auto-rerun* followed by the rerun attempt number, as shown in the following screenshot.
153
+
Automatic workflow reruns are indicated on the pipelines page in the CircleCI web app. In the Trigger event column you sill see *Auto-rerun* followed by the rerun attempt number, as shown in the following screenshot.
103
154
104
155
In this example the workflow is rerun twice out of a possible five attempts before it succeeds.
105
156
157
+
.Automatic workflow reruns in the CircleCI web app
You can get information about automatic reruns via the CircleCI APIs.
109
169
110
-
You can retrieve information about automatic reruns using the link:https://circleci.com/docs/api/v2/index.html#tag/Workflow/operation/getWorkflowById[CircleCI API]:
170
+
==== Automatic workflow reruns
171
+
172
+
You can retrieve information about *automatic workflow reruns* using the link:https://circleci.com/docs/api/v2/index.html#tag/Workflow/operation/getWorkflowById[CircleCI API v2]:
111
173
112
174
[source,bash]
113
175
----
114
176
curl -X GET "https://circleci.com/api/v2/workflow/{workflow-id}" \
115
177
-H "Circle-Token: YOUR_TOKEN"
116
178
----
117
179
118
-
The API response includes additional fields for automatic reruns:
180
+
The API response includes the following fields for automatic workflow reruns:
181
+
182
+
* `auto_rerun_number`: The current rerun attempt number.
183
+
* `max_auto_reruns`: The maximum number of reruns configured.
184
+
185
+
==== Automatic step reruns
186
+
187
+
You can retrieve information about automatic step reruns using the link:https://circleci.com/docs/api/v1/index.html#single-job[job information API endpoint from the CircleCI API v1.1].
Copy file name to clipboardExpand all lines: docs/reference/modules/ROOT/pages/configuration-reference.adoc
+54Lines changed: 54 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1381,6 +1381,17 @@ NOTE: the `run` step replaces the deprecated `deploy` step. If your job has a pa
1381
1381
| N
1382
1382
| String
1383
1383
| <<the-when-attribute,Specify when to enable or disable the step>>. Takes the following values: `always`, `on_success`, `on_fail` (default: `on_success`)
1384
+
1385
+
| `max_auto_reruns`
1386
+
| N
1387
+
| Integer
1388
+
| The maximum number of times to automatically rerun the step if it fails. Must be between `1` and `5`.
1389
+
1390
+
| `auto_rerun_delay`
1391
+
| N
1392
+
| String
1393
+
| The delay between reruns of the step if it fails. This delay can only be set along with `max_auto_reruns`. The string is a decimal with unit suffix using either seconds `s` or minutes `m` up to a maximum of 10 minutes, such as "10s", "2m".
1394
+
1384
1395
|===
1385
1396
1386
1397
Each `run` declaration represents a new shell. It is possible to specify a multi-line `command`, each line of which will be run in the same shell.
@@ -1555,6 +1566,49 @@ A job can exit without failing by using `run: circleci-agent step halt`. However
1555
1566
1556
1567
'''
1557
1568
1569
+
==== Automatic step reruns
1570
+
1571
+
The following attributes can be used to automatically rerun a step if it fails, and delay that rerun if required:
1572
+
1573
+
[cols="2,1,1,2"]
1574
+
|===
1575
+
| Key | Required | Type | Description
1576
+
1577
+
|`max_auto_reruns`
1578
+
| N
1579
+
| Integer
1580
+
| The maximum number of times to automatically rerun the step if it fails. Must be between `1` and `5`.
1581
+
1582
+
| `auto_rerun_delay`
1583
+
| N
1584
+
| String
1585
+
| The delay between reruns of the step if it fails. This delay can only be set along with `max_auto_reruns`. The string is a decimal with unit suffix using either seconds `s` or minutes `m` up to a maximum of 10 minutes, such as "10s", "2m". If you do not supply a delay, the rerun will start immediately after the step fails.
1586
+
|===
1587
+
1588
+
Automatic reruns are only supported for `run` steps, not special steps like `checkout` or `setup_remote_docker`.
1589
+
1590
+
You must configure the `command` key for the step, you cannot use the short form run step configuration, for example, the following is not supported for use with automatic reruns: `- run: echo "Hello, world!"`
1591
+
1592
+
*Example:*
1593
+
1594
+
.CircleCI job configured to automatically rerun a step up to 3 times if it fails with a 10 second delay between attempts
1595
+
[,yml]
1596
+
----
1597
+
version: 2.1
1598
+
1599
+
jobs:
1600
+
my-job:
1601
+
steps:
1602
+
- run:
1603
+
command: echo "Hello, world!"
1604
+
max_auto_reruns: 3
1605
+
auto_rerun_delay: 10s
1606
+
----
1607
+
1608
+
For more information, see the xref:guides:orchestrate:automatic-reruns.adoc#[Automatic reruns] page.
0 commit comments