Bogusław Witek

Kubernetes from zero at home: k3s, Helm, Argo CD and Sealed Secrets

Kubernetes was the biggest gap in my skills, so instead of reading about it I ran it on my home server and set up copies of real services on it. A few days later the whole cluster could be rebuilt from one repository, passwords included.

· 5 min read

Written in November 2026 from notes made at the time.

I have been running Linux servers for years, but for a long time Kubernetes was something I read about rather than something I kept alive. In July 2026 I decided to change that. The rule was simple: no tutorial examples that get deleted an hour later. Every step had to leave behind a working service and a commit in a repository.

Why k3s and not “real” Kubernetes straight away

I started with one virtual machine with 4 GiB of RAM on my home server. That is not enough for a comfortable full cluster with a separate etcd and several nodes. k3s fits in a single process and ships what a cluster needs to run: a pod network, Traefik as the ingress, a provisioner for volumes on local disk and a simple load balancer. I wanted to learn Kubernetes objects, not fight the CNI on day one, so the trade-off was an easy call.

The order was: Pod, Deployment, Service and Ingress on simple apps, then volumes and secrets with BookStack and MariaDB, then my own Helm chart, and GitOps last. Each step built on the one before. The same BookStack that started with hand-applied manifests was managed by Argo CD a few days later, with no downtime: Argo adopted the existing objects instead of creating them again.

Helm: my own chart first, other people’s after

I wrote a chart for a speedtest app so I would understand what helm install actually does. There is less magic than it seems: Go templates, a values.yaml file and a versioned set of manifests. That paid off immediately with kube-prometheus-stack, whose values run to hundreds of lines. Without knowing what renders from what, it is easy to get lost in the overrides.

The same chart gave me my first real problem. The Prometheus CRDs are too big for the annotation that a plain client-side kubectl apply writes. The fix is Server-Side Apply, which in Argo CD is a single sync option.

GitOps: the repository is the source of truth

Argo CD reads the repository from my own Gitea instance, with a read-only token. The layout is the app-of-apps pattern: one root application watches the apps/ directory, and every file in it is another application. Adding a service means one new file and one directory of manifests. Within a few days Argo CD was even managing itself.

# apps/root.yaml – one application that looks after all the others
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/me/homelab-k3s.git
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd

I left automatic sync off on purpose. A push shows up in Argo as “OutOfSync” and waits for me to apply it. On a cluster meant for learning I sometimes change things by hand to see what happens, and I did not want self-heal reverting my experiments before I had a chance to look at them. Manual sync also forces me to read the diff before every deployment. I have kept that habit ever since, in production too.

Sealed Secrets: passwords in git without the guilt

One problem was left. If the cluster is rebuilt from the repository, the database password has to live there too. I picked Sealed Secrets because it is native to Kubernetes and Argo CD needs no plugin for it. A secret is encrypted with the controller’s public key and only the ciphertext goes into the repository. Only the controller inside the cluster can decrypt it.

In the default mode the ciphertext is bound to the secret’s name and namespace. That has a consequence: the same password used in two namespaces needs two separately sealed files. It is annoying at first. Later you appreciate that nobody can copy a sealed secret into another app and read it there.

I did not take it on trust. I ran a fire drill:

The passwords sit in a file outside the repository and go straight into kubeseal
$ kubectl create secret generic bookstack-env -n bookstack --from-env-file=/run/user/$UID/bookstack.env --dry-run=client -o yaml | kubeseal --format yaml > bookstack/bookstack-sealedsecret.yaml
$ git add bookstack/bookstack-sealedsecret.yaml && git commit -m "bookstack: secrets as SealedSecrets"
# fire drill: delete the decrypted Secret and see whether the controller brings it back
$ kubectl -n bookstack delete secret bookstack-env
$ kubectl -n bookstack get secret bookstack-env

The controller recreated the Secret from the sealed version. Only after that drill did I decide the approach would stay.

What I learned

  • The Sealed Secrets controller key is the most important secret in the whole cluster. Without it every encrypted file in the repository is useless. I keep a copy in my password manager. Two weeks later I rebuilt this cluster from scratch on a different OS, and that copy is why every secret decrypted without re-sealing: the key goes in before the controller starts.
  • Learn on real services. A tutorial ends when “hello world” works. A real app has a database, a volume, a password and people who use it, and only that teaches you what a deployment without downtime means.
  • Manual sync is not laziness. When starting out, I would rather see every difference between git and the cluster and approve it myself than find out afterwards what an automated process did on my behalf.

All posts