diff --git a/docs/upgrades/roll-back.md b/docs/upgrades/roll-back.md new file mode 100644 index 0000000000..c9feb5c891 --- /dev/null +++ b/docs/upgrades/roll-back.md @@ -0,0 +1,145 @@ +# Rolling Back K3s + +You can roll back a K3s Kubernetes version after a problematic upgrade, using a combination of K3s binary downgrading and datastore restoration. This is applicable to both clusters with external databases and clusters using an embedded etcd. + +## Important Considerations + +- **Backups:** Before upgrading, ensure you have a valid database or etcd snapshot from your cluster running the older version of K3s. Without a backup, a rollback is impossible. +- **Potential Data Loss:** The `k3s-killall.sh` script forcefully terminates K3s processes and may result in data loss if applications are not properly shut down. +- **Version Specifics:** Always verify K3s and component versions before and after the rollback. + +## Rollback with External Database + +This section applies to K3s clusters using an external database (e.g., PostgreSQL, MySQL). + +1. If the cluster is running and the Kubernetes API is available, gracefully stop workloads by draining all nodes: + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data ... + ``` + + :::note + + This process may disrupt running applications. + + ::: + +1. On each node, stop the K3s service and all running pod processes: + + ```bash + k3s-killall.sh + ``` + +1. Restore a database snapshot taken before upgrading K3s and verify the integrity of the database. For example, if you're using PostgreSQL, run the following command: + + ```bash + pg_restore -U -d + ``` + +1. On each node, roll back the K3s binary to the previous version. + + - Clusters with Internet Access: + - Server nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="server" sh - + ``` + + - Agent nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="agent" sh - + ``` + + - Air-gapped Clusters: + + - Download the artifacts and run the [install script](../installation/airgap.md#install-k3s) locally. Verify the K3s version after install with `k3s --version` and reapply any custom configurations that where used before the upgrade. + +1. Start the K3s service on each node: + + ```bash + systemctl start k3s + ``` + +1. Verify the K3s service status with `systemctl status k3s`. + +## Rollback with Embedded etcd + +This section applies to K3s clusters using an embedded etcd. + +1. If the cluster is running and the Kubernetes API is available, gracefully stop workloads by draining all nodes: + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data ... + ``` + +1. On each node, stop the K3s service and all running pod processes: + + ```bash + k3s-killall.sh + ``` + +1. On each node, roll back the K3s binary to the previous version, but *do not* start K3s. + + - Clusters with Internet Access: + + - Server nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_START="true" sh - + ``` + + - Agent nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="agent" INSTALL_K3S_SKIP_START="true" sh - + ``` + + - Air-gapped Clusters: + + - Download the artifacts and run the [install script](../installation/airgap.md#install-k3s) locally. Add the environment variable `INSTALL_K3S_SKIP_START="true"` when running the install script to prevent K3s from starting. + +1. On the first server node or the node without a `server:` entry in its [K3s config file](../installation/configuration.md), initiate the cluster restore. Refer to the [Snapshot Restore Steps](../cli/etcd-snapshot.md#snapshot-restore-steps) for more information: + + ```bash + k3s server --cluster-reset --cluster-reset-restore-path= + ``` + + :::warning + This will overwrite all data in the etcd datastore. Verify the snapshot's integrity before restoring. Be aware that large snapshots can take a long time to restore. + ::: + +1. Start the K3s service on the first server node: + + ```bash + systemctl start k3s + ``` + +1. On the other server nodes, remove the K3s database directory: + + ```bash + rm -rf /var/lib/rancher/k3s/server/db + ``` + +1. Start the K3s service on the other server nodes: + + ```bash + systemctl start k3s + ``` + +1. Start the K3s service on all agent nodes: + + ```bash + systemctl start k3s + ``` + +1. Verify the K3s service status with `systemctl status k3s`. + +## Verification + +After the rollback, verify the following: + +- K3s version: `k3s --version` +- Kubernetes cluster health: `kubectl get nodes` +- Application functionality. +- Check the K3s logs for errors. diff --git a/i18n/ja/docusaurus-plugin-content-docs/current/upgrades/roll-back.md b/i18n/ja/docusaurus-plugin-content-docs/current/upgrades/roll-back.md new file mode 100644 index 0000000000..c9feb5c891 --- /dev/null +++ b/i18n/ja/docusaurus-plugin-content-docs/current/upgrades/roll-back.md @@ -0,0 +1,145 @@ +# Rolling Back K3s + +You can roll back a K3s Kubernetes version after a problematic upgrade, using a combination of K3s binary downgrading and datastore restoration. This is applicable to both clusters with external databases and clusters using an embedded etcd. + +## Important Considerations + +- **Backups:** Before upgrading, ensure you have a valid database or etcd snapshot from your cluster running the older version of K3s. Without a backup, a rollback is impossible. +- **Potential Data Loss:** The `k3s-killall.sh` script forcefully terminates K3s processes and may result in data loss if applications are not properly shut down. +- **Version Specifics:** Always verify K3s and component versions before and after the rollback. + +## Rollback with External Database + +This section applies to K3s clusters using an external database (e.g., PostgreSQL, MySQL). + +1. If the cluster is running and the Kubernetes API is available, gracefully stop workloads by draining all nodes: + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data ... + ``` + + :::note + + This process may disrupt running applications. + + ::: + +1. On each node, stop the K3s service and all running pod processes: + + ```bash + k3s-killall.sh + ``` + +1. Restore a database snapshot taken before upgrading K3s and verify the integrity of the database. For example, if you're using PostgreSQL, run the following command: + + ```bash + pg_restore -U -d + ``` + +1. On each node, roll back the K3s binary to the previous version. + + - Clusters with Internet Access: + - Server nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="server" sh - + ``` + + - Agent nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="agent" sh - + ``` + + - Air-gapped Clusters: + + - Download the artifacts and run the [install script](../installation/airgap.md#install-k3s) locally. Verify the K3s version after install with `k3s --version` and reapply any custom configurations that where used before the upgrade. + +1. Start the K3s service on each node: + + ```bash + systemctl start k3s + ``` + +1. Verify the K3s service status with `systemctl status k3s`. + +## Rollback with Embedded etcd + +This section applies to K3s clusters using an embedded etcd. + +1. If the cluster is running and the Kubernetes API is available, gracefully stop workloads by draining all nodes: + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data ... + ``` + +1. On each node, stop the K3s service and all running pod processes: + + ```bash + k3s-killall.sh + ``` + +1. On each node, roll back the K3s binary to the previous version, but *do not* start K3s. + + - Clusters with Internet Access: + + - Server nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_START="true" sh - + ``` + + - Agent nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="agent" INSTALL_K3S_SKIP_START="true" sh - + ``` + + - Air-gapped Clusters: + + - Download the artifacts and run the [install script](../installation/airgap.md#install-k3s) locally. Add the environment variable `INSTALL_K3S_SKIP_START="true"` when running the install script to prevent K3s from starting. + +1. On the first server node or the node without a `server:` entry in its [K3s config file](../installation/configuration.md), initiate the cluster restore. Refer to the [Snapshot Restore Steps](../cli/etcd-snapshot.md#snapshot-restore-steps) for more information: + + ```bash + k3s server --cluster-reset --cluster-reset-restore-path= + ``` + + :::warning + This will overwrite all data in the etcd datastore. Verify the snapshot's integrity before restoring. Be aware that large snapshots can take a long time to restore. + ::: + +1. Start the K3s service on the first server node: + + ```bash + systemctl start k3s + ``` + +1. On the other server nodes, remove the K3s database directory: + + ```bash + rm -rf /var/lib/rancher/k3s/server/db + ``` + +1. Start the K3s service on the other server nodes: + + ```bash + systemctl start k3s + ``` + +1. Start the K3s service on all agent nodes: + + ```bash + systemctl start k3s + ``` + +1. Verify the K3s service status with `systemctl status k3s`. + +## Verification + +After the rollback, verify the following: + +- K3s version: `k3s --version` +- Kubernetes cluster health: `kubectl get nodes` +- Application functionality. +- Check the K3s logs for errors. diff --git a/i18n/kr/docusaurus-plugin-content-docs/current/upgrades/roll-back.md b/i18n/kr/docusaurus-plugin-content-docs/current/upgrades/roll-back.md new file mode 100644 index 0000000000..c9feb5c891 --- /dev/null +++ b/i18n/kr/docusaurus-plugin-content-docs/current/upgrades/roll-back.md @@ -0,0 +1,145 @@ +# Rolling Back K3s + +You can roll back a K3s Kubernetes version after a problematic upgrade, using a combination of K3s binary downgrading and datastore restoration. This is applicable to both clusters with external databases and clusters using an embedded etcd. + +## Important Considerations + +- **Backups:** Before upgrading, ensure you have a valid database or etcd snapshot from your cluster running the older version of K3s. Without a backup, a rollback is impossible. +- **Potential Data Loss:** The `k3s-killall.sh` script forcefully terminates K3s processes and may result in data loss if applications are not properly shut down. +- **Version Specifics:** Always verify K3s and component versions before and after the rollback. + +## Rollback with External Database + +This section applies to K3s clusters using an external database (e.g., PostgreSQL, MySQL). + +1. If the cluster is running and the Kubernetes API is available, gracefully stop workloads by draining all nodes: + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data ... + ``` + + :::note + + This process may disrupt running applications. + + ::: + +1. On each node, stop the K3s service and all running pod processes: + + ```bash + k3s-killall.sh + ``` + +1. Restore a database snapshot taken before upgrading K3s and verify the integrity of the database. For example, if you're using PostgreSQL, run the following command: + + ```bash + pg_restore -U -d + ``` + +1. On each node, roll back the K3s binary to the previous version. + + - Clusters with Internet Access: + - Server nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="server" sh - + ``` + + - Agent nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="agent" sh - + ``` + + - Air-gapped Clusters: + + - Download the artifacts and run the [install script](../installation/airgap.md#install-k3s) locally. Verify the K3s version after install with `k3s --version` and reapply any custom configurations that where used before the upgrade. + +1. Start the K3s service on each node: + + ```bash + systemctl start k3s + ``` + +1. Verify the K3s service status with `systemctl status k3s`. + +## Rollback with Embedded etcd + +This section applies to K3s clusters using an embedded etcd. + +1. If the cluster is running and the Kubernetes API is available, gracefully stop workloads by draining all nodes: + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data ... + ``` + +1. On each node, stop the K3s service and all running pod processes: + + ```bash + k3s-killall.sh + ``` + +1. On each node, roll back the K3s binary to the previous version, but *do not* start K3s. + + - Clusters with Internet Access: + + - Server nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_START="true" sh - + ``` + + - Agent nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="agent" INSTALL_K3S_SKIP_START="true" sh - + ``` + + - Air-gapped Clusters: + + - Download the artifacts and run the [install script](../installation/airgap.md#install-k3s) locally. Add the environment variable `INSTALL_K3S_SKIP_START="true"` when running the install script to prevent K3s from starting. + +1. On the first server node or the node without a `server:` entry in its [K3s config file](../installation/configuration.md), initiate the cluster restore. Refer to the [Snapshot Restore Steps](../cli/etcd-snapshot.md#snapshot-restore-steps) for more information: + + ```bash + k3s server --cluster-reset --cluster-reset-restore-path= + ``` + + :::warning + This will overwrite all data in the etcd datastore. Verify the snapshot's integrity before restoring. Be aware that large snapshots can take a long time to restore. + ::: + +1. Start the K3s service on the first server node: + + ```bash + systemctl start k3s + ``` + +1. On the other server nodes, remove the K3s database directory: + + ```bash + rm -rf /var/lib/rancher/k3s/server/db + ``` + +1. Start the K3s service on the other server nodes: + + ```bash + systemctl start k3s + ``` + +1. Start the K3s service on all agent nodes: + + ```bash + systemctl start k3s + ``` + +1. Verify the K3s service status with `systemctl status k3s`. + +## Verification + +After the rollback, verify the following: + +- K3s version: `k3s --version` +- Kubernetes cluster health: `kubectl get nodes` +- Application functionality. +- Check the K3s logs for errors. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/upgrades/roll-back.md b/i18n/zh/docusaurus-plugin-content-docs/current/upgrades/roll-back.md new file mode 100644 index 0000000000..c9feb5c891 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/upgrades/roll-back.md @@ -0,0 +1,145 @@ +# Rolling Back K3s + +You can roll back a K3s Kubernetes version after a problematic upgrade, using a combination of K3s binary downgrading and datastore restoration. This is applicable to both clusters with external databases and clusters using an embedded etcd. + +## Important Considerations + +- **Backups:** Before upgrading, ensure you have a valid database or etcd snapshot from your cluster running the older version of K3s. Without a backup, a rollback is impossible. +- **Potential Data Loss:** The `k3s-killall.sh` script forcefully terminates K3s processes and may result in data loss if applications are not properly shut down. +- **Version Specifics:** Always verify K3s and component versions before and after the rollback. + +## Rollback with External Database + +This section applies to K3s clusters using an external database (e.g., PostgreSQL, MySQL). + +1. If the cluster is running and the Kubernetes API is available, gracefully stop workloads by draining all nodes: + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data ... + ``` + + :::note + + This process may disrupt running applications. + + ::: + +1. On each node, stop the K3s service and all running pod processes: + + ```bash + k3s-killall.sh + ``` + +1. Restore a database snapshot taken before upgrading K3s and verify the integrity of the database. For example, if you're using PostgreSQL, run the following command: + + ```bash + pg_restore -U -d + ``` + +1. On each node, roll back the K3s binary to the previous version. + + - Clusters with Internet Access: + - Server nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="server" sh - + ``` + + - Agent nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="agent" sh - + ``` + + - Air-gapped Clusters: + + - Download the artifacts and run the [install script](../installation/airgap.md#install-k3s) locally. Verify the K3s version after install with `k3s --version` and reapply any custom configurations that where used before the upgrade. + +1. Start the K3s service on each node: + + ```bash + systemctl start k3s + ``` + +1. Verify the K3s service status with `systemctl status k3s`. + +## Rollback with Embedded etcd + +This section applies to K3s clusters using an embedded etcd. + +1. If the cluster is running and the Kubernetes API is available, gracefully stop workloads by draining all nodes: + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data ... + ``` + +1. On each node, stop the K3s service and all running pod processes: + + ```bash + k3s-killall.sh + ``` + +1. On each node, roll back the K3s binary to the previous version, but *do not* start K3s. + + - Clusters with Internet Access: + + - Server nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_START="true" sh - + ``` + + - Agent nodes: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=vX.Y.Zk3s1 INSTALL_K3S_EXEC="agent" INSTALL_K3S_SKIP_START="true" sh - + ``` + + - Air-gapped Clusters: + + - Download the artifacts and run the [install script](../installation/airgap.md#install-k3s) locally. Add the environment variable `INSTALL_K3S_SKIP_START="true"` when running the install script to prevent K3s from starting. + +1. On the first server node or the node without a `server:` entry in its [K3s config file](../installation/configuration.md), initiate the cluster restore. Refer to the [Snapshot Restore Steps](../cli/etcd-snapshot.md#snapshot-restore-steps) for more information: + + ```bash + k3s server --cluster-reset --cluster-reset-restore-path= + ``` + + :::warning + This will overwrite all data in the etcd datastore. Verify the snapshot's integrity before restoring. Be aware that large snapshots can take a long time to restore. + ::: + +1. Start the K3s service on the first server node: + + ```bash + systemctl start k3s + ``` + +1. On the other server nodes, remove the K3s database directory: + + ```bash + rm -rf /var/lib/rancher/k3s/server/db + ``` + +1. Start the K3s service on the other server nodes: + + ```bash + systemctl start k3s + ``` + +1. Start the K3s service on all agent nodes: + + ```bash + systemctl start k3s + ``` + +1. Verify the K3s service status with `systemctl status k3s`. + +## Verification + +After the rollback, verify the following: + +- K3s version: `k3s --version` +- Kubernetes cluster health: `kubectl get nodes` +- Application functionality. +- Check the K3s logs for errors. diff --git a/sidebars.js b/sidebars.js index 5d54232ad3..5ae733f357 100644 --- a/sidebars.js +++ b/sidebars.js @@ -36,6 +36,7 @@ module.exports = { 'upgrades/killall', 'upgrades/manual', 'upgrades/automated', + 'upgrades/roll-back', ], }, {