Talos Linux: a server without SSH, and why that is good

In August my home server running IncusOS had three virtual machines: an older one with the household services (Samba, Jellyfin, Frigate), a separate one for the Matrix chat server, and a fresh k3s machine where I was learning Kubernetes. Each had its own updates, its own logs and its own “I changed something here by hand once”. I decided to replace them with a single Kubernetes machine and to run it on Talos Linux.

An OS you cannot log in to

Talos has no SSH, no shell and no package manager. Not even bash. The whole system is a kernel, a few services and Kubernetes, and you manage it only through an API with the talosctl tool. The machine’s configuration is one YAML file: network, disks, kubelet settings, extensions. You change the file, send it to the node, and the node converges.

For me it solves a specific problem. On an ordinary server the most dangerous changes are the ones someone made by hand and never wrote down. Talos does not allow that. If something is on the node, it is in the config file, and that file lives in a repository. A whole class of attacks goes away too: there is no SSH daemon to guard and no shell for anyone to land in.

Before deciding: two tests on a throwaway machine

I did not want to pick an OS from its marketing page. Two things could have ruled Talos out, so I tested both on a temporary VM before touching anything real.

The first was data. Photos, files and recordings sit on a two-disk ZFS mirror on the host, and the VMs see them through virtiofs. Only recent Talos releases have the ExternalVolumeConfig document that mounts such a volume. The test: a pod wrote a file, and talosctl read saw it on the node side.

The second was the GPU. Jellyfin and Frigate need hardware video decoding. Talos will not let you install a driver by hand, but Image Factory builds a system image with the extensions you choose, in my case the firmware for Intel integrated graphics. Along the way it turned out the old VM had never had the real GPU at all, only a virtual card, so Frigate had been decoding video on the CPU the whole time.

Only after both tests passed did I decide.

The whole machine in one file

Here is part of the config patch (names generalized). Three documents in one file: the hostname, a RAID volume over virtiofs, and a separate disk chosen by an expression rather than by device name.

# Newer Talos releases emit this document by default: the hostname goes here,
# not in machine.network (both at once fail validation)
apiVersion: v1alpha1
kind: HostnameConfig
auto: "off"
hostname: lab-node
---
# a host volume over virtiofs, mounted at /var/mnt/data
apiVersion: v1alpha1
kind: ExternalVolumeConfig
name: data
filesystemType: virtiofs
mount:
  virtiofs:
    tag: incus_data
---
# a separate disk for bulk data: picked by size, not by a /dev/sdX that can change
apiVersion: v1alpha1
kind: UserVolumeConfig
name: bulk
provisioning:
  diskSelector:
    match: '!system_disk && disk.size > 149u * GiB && disk.size < 151u * GiB'
  minSize: 149GiB
  grow: true
filesystem:
  type: xfs

The cluster secrets (the PKI and the API client config) live in the same repository, encrypted with git-crypt. The patches stay in plain text so they can be read in a review.

A typical change looks like this:

$ talosctl -n $NODE apply-config --file rendered/controlplane.yaml --dry-run
$ talosctl -n $NODE apply-config --file rendered/controlplane.yaml --mode=try
$ talosctl -n $NODE apply-config --file rendered/controlplane.yaml --mode=no-reboot
# instead of ssh and less: logs and files go through the API too
$ talosctl -n $NODE dmesg | tail
$ talosctl -n $NODE read /var/mnt/data/test.txt

What changes when there is no host

No shell has consequences worth knowing before a migration. CI that built images on the host through the Docker socket has nothing to use here, so the runner had to move into the cluster, in its own namespace with elevated privileges. Talos enforces the “baseline” PodSecurity level by default, so any pod with a hostPath or access to /dev/dri needs its namespace deliberately labeled as privileged. That is a good thing: the exceptions are visible in git rather than scattered around the system.

I split the migration into phases and did it in two days, 7 and 8 August: platform, apps with data, the services from the old host, internet traffic together with the chat server, and clean-up. I stopped the old machines and kept them as a safety net in case I had to go back.

What I learned

• Test the deal-breakers before you commit. A test on a throwaway machine answered questions no description can, and it uncovered a GPU problem that had been there for a long time.
• The client version has to match the cluster. talosctl writes an installer image of its own version into the config it renders. So a newer client quietly produces a config that upgrades the OS and Kubernetes the next time the node is installed. I always render with the version the cluster runs.
• “No SSH” is not less control, it is control in one place. Everything I could do by hand on a server I now do through a file in a repository, which means history, review and repeatability.