In this article, I am going to give a step-by-step guide to taking backup and restore of the ETCD on the Kubernetes cluster.
For this, I am taking the Kubeadm v1.28.0 version cluster but you can do it in any version.
How To Take ETCD Backup?
It is the responsibility of a Kubernetes admin to take regular backups of the ETCD because all Kubernetes objects are stored on ETCD.
Step 1: Find The ETCD Manifest
Before we perform the backup we need some file path information such as truster-ca-file
, cert-file
and key-file
.
Ensure you are in the Master Node (controlplane) if not, first, SSH into the Master Node.
ETCD manifest will be available in the /etc/kubernetes/manifests
directory.
Let’s open the manifest.
sudo cat /etc/kubernetes/manifests/etcd.yaml
Our required information will be obtained from the following part of the file.
Step 2: ETCD Snapshot
To take a snapshot, use the following command.
ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db
Here, /opt/etcd-backup.db
is the path where I want to store the backup, you can choose your own destination and change the name as you wish but the .db
extension is necessary.
The remaining information is the default for almost every cluster, so you can also use the same information.
If the backup is successfully done, you can also see the same output.
To verify the snapshot, use the following command.
ETCDCTL_API=3 etcdctl --write-out=table snapshot status /opt/etcd-backup.db
How to Restore ETCD?
Step 1: Restore ETCD From Snapshot
First, you need to know on which path the snapshot is, if you are preparing for the CKA exam, you might be asked to restore the ETCD and for that, they will provide the path of the snapshot.
For restoration, we need one information, which is data-dir
, that we can take from the ETCD manifest.
sudo cat /etc/kubernetes/manifests/etcd.yaml
To restore the ETCD, use the following command.
ETCDCTL_API=3 etcdctl --data-dir /var/lib/etcd snapshot restore /opt/etcd-backup.db
If the restoring process is properly done, you will get this same output.
To ensure everything is properly running, check the Pods on the kube-system
namespace.
kubectl get pods -n kube-system
Possible Errors & Troubleshoot
Error: Error: data-dir “/var/lib/etcd” not empty or could not be read
When you try to restore the ETCD, you might probably get this error.
Error: data-dir "/var/lib/etcd" not empty or could not be read
Solution:
Remove the /var/lib/etcd
and perform the restore again.
Conclusion
For testing purposes, if you want to know the changes after the restore, then create a Pod on the cluster before performing the restore so that after the restore if the process is successfully done, you won’t able to see the Pod.
Etcd backup is a very important concept in CKA certification. If you are preparing for CKA certification, make use of the CKA coupon form the Linux Foundation offer page.