You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<TabItem value="first-replica-failing" label="First replica is failing">
159
159
160
160
:::warning
161
-
Before attempting any recovery procedure, create a backup of the virtual cluster namespace on the host cluster. If using namespace syncing, back up all synced namespaces as well.
161
+
Before attempting any recovery procedure, create a backup of your virtual cluster using `vcluster snapshot create` or manually backup the virtual cluster namespace on the host cluster. If using namespace syncing, back up all synced namespaces as well.
162
162
:::
163
163
164
164
The recovery procedure depends on your StatefulSet `podManagementPolicy` configuration. vCluster version 0.20 and later use `Parallel` by default. Earlier versions used `OrderedReady`.
165
165
166
+
:::info
167
+
If more than one pod is down with `podManagementPolicy: OrderedReady`, you must first [migrate to `Parallel`](#migrate-to-parallel) before attempting recovery.
168
+
:::
169
+
166
170
Check your configuration:
167
171
168
172
<InterpolatedCodeBlock
@@ -175,7 +179,16 @@ Check your configuration:
175
179
176
180
<Flow>
177
181
<Step title="Delete the failed pod and PVC">
178
-
Delete the corrupted pod and PVC for replica-0:
182
+
First, identify the PVC for replica-0:
183
+
184
+
<InterpolatedCodeBlock
185
+
code={`kubectl get pvc -l [[VAR:VCLUSTER LABEL:app=vcluster]] -n [[VAR:NAMESPACE:vcluster-my-team]]`}
186
+
language="bash"
187
+
/>
188
+
189
+
<br />
190
+
191
+
Delete the corrupted pod and its PVC:
179
192
180
193
<InterpolatedCodeBlock
181
194
code={`kubectl delete pod [[VAR:VCLUSTER NAME:my-vcluster]]-0 -n [[VAR:NAMESPACE:vcluster-my-team]]
The pod restarts with a new empty PVC. After 1-3 pod restarts, the automatic recovery adds it back to the etcd cluster.
201
+
The pod restarts with a new empty PVC. The initial attempts fail because the new member tries to join the existing etcd cluster but lacks the required data. After 1-3 pod restarts, vCluster's automatic recovery detects the empty member and properly adds it as a new learner, allowing it to sync data from healthy members and join the cluster.
189
202
</Step>
190
203
191
204
<Step title="Monitor recovery">
@@ -239,29 +252,21 @@ Delete the StatefulSet without deleting the pods:
239
252
</Step>
240
253
241
254
<Step title="Update configuration to Parallel">
255
+
<a id="migrate-to-parallel"></a>
256
+
242
257
Update your virtual cluster configuration to use `Parallel` pod management policy.
The pod restarts with a new empty PVC and automatic recovery adds it back to the cluster after 1-3 pod restarts.
308
+
The pod restarts with a new empty PVC. The initial attempts fail because the new member tries to join the existing etcd cluster but lacks the required data. After 1-3 pod restarts, vCluster's automatic recovery detects the empty member and properly adds it as a new learner, allowing it to sync data from healthy members and join the cluster.
0 commit comments