At the time the production cluster ran on two small virtual machines: one carried the control plane, the other the web tier. That split dated from September, when the control plane ran short of memory and I relieved it by moving the websites to a separate machine. In October a simpler layout became possible: one dedicated server instead of two VMs. Before touching anything I checked last week’s peaks in Prometheus. Even together, the two machines peaked well below the new server’s memory, so consolidating made sense. The deadline was the end of the contract on the smaller machines.
The simplest route is to build a new cluster and move the apps. I took the harder but cleaner one: the same cluster, the same etcd database, the same certificates, just on different hardware.
Stage 0: move the endpoint to a DNS name before anything else
The cluster knew its API address as the bare IP of the old server, and that address was going away. So the first step, done separately and verified before the rest, had nothing to do with the new server: I switched the cluster endpoint to a DNS name the API certificate already covered. The name pointed at the same old address, so in theory nothing changed.
Talos lets you apply a change like that in trial mode, with automatic rollback, and only then for good. The workers did not even notice, because the kubelet connects through the local KubePrism proxy. On the control plane, the change restarted the static pods and the API was briefly unavailable, which the dry run had not warned about.
Stage 1: a second control plane
I installed the new server from a Talos image taken from the official release, checking its checksum and its cosign signature. The images generated on demand came with no checksums to compare against, and I did not want to install a production system from a file I could not verify. I rendered the config with the same talosctl version the cluster runs, with an etcd section identical to the old node’s.
The new node joined etcd on its own: first as a learner that only catches up on data, then as a full member. From that moment etcd had two members and quorum needed both. Losing either would stop the whole cluster, so I did the next two stages the same night.
Stage 2: apps and volumes
I pin web apps to nodes by label, not by hostname. All it took was labeling the new server, removing the label from the old one and restarting the pods. I moved the volumes with a script that checks sha256 on both sides, with the app stopped. Finally I pointed the sites’ DNS records at the new address.
Here is a trap worth knowing: a PersistentVolume with the Delete policy is wiped by the provisioner, data included, once it is released. While copying, that meant the source copy would vanish the moment the app switched to the new volume. Before releasing each such PV, I switched it to Retain.
Stage 3: etcd moves house
The endpoint name went to the new address. Then a manual etcd snapshot, verified, and the old node left etcd.
Before handing over etcd: a snapshot, always, even when everything looks fine
$ talosctl -n $NEW_NODE etcd snapshot etcd-before-handover.db
$ talosctl -n $NEW_NODE etcd members
# the old node leaves etcd...
$ talosctl -n $OLD_NODE etcd leave
# ...and stops right away; only the system partitions are wiped, the data stays as an emergency copy
$ talosctl -n $OLD_NODE reset --system-labels-to-wipe STATE --system-labels-to-wipe EPHEMERAL --reboot=false
$ kubectl delete node $OLD_NODEThe old machines were not wiped clean. Their data disks stayed untouched as an emergency copy until I canceled those servers.
Two surprises
Changing the endpoint also changes who issues service-account tokens. On Talos the endpoint address goes into the API server’s --service-account-issuer and --api-audiences. Every token minted before the change started getting Unauthorized until the kubelet renewed it. Because of that, the Cilium agent on one node could not see the new node, several operators reported errors and two pods went into CrashLoopBackOff. Restarting the affected pods fixed it at once. Next time I will pass the old issuer as an extra one during the switch, or plan the restarts up front.
An old API server without etcd keeps accepting connections. After etcd leave the old node still answered on the API port, and KubePrism kept sending part of the traffic there. Leader-elected controllers could not renew their leases for tens of seconds, until I stopped the old node. That is why the commands above run reset right after leave, not “in a minute”.
What I learned
- Split the changes that can be split. Moving the endpoint to a DNS name as its own verified step meant the rest of the night had one fewer unknown.
- Take an etcd snapshot before every change to quorum. Not because something will go wrong, but because if it does, without one there is nowhere to go back to.
- Plan for side effects, not only for steps. Both surprises came from behavior that is documented but invisible in a diff. They are now in the repository’s README, so nobody, me included, has to discover them twice.