Cilium instead of the default network: network policies that actually work

I had already written NetworkPolicies for several apps: deny everything first, then allow the connections that are needed. Argo CD showed them as synced and kubectl get networkpolicy listed them. The problem was that the cluster ran on Flannel, and Flannel does not implement network policies at all. The Kubernetes API accepts the object, because it is just a resource in the database, but nothing enforces it. Every pod could talk to every other pod.

That is exactly why I moved to Cilium: the production cluster on 18–19 August, the home cluster on 20 August.

What Cilium brings

Cilium is an eBPF-based CNI. Instead of iptables rules it programs the kernel directly, and it can replace kube-proxy entirely. On Talos that means two changes in the machine config: no default CNI and kube-proxy disabled. The rest is set by the Helm chart, which Argo CD manages.

Three things mattered to me. First, network policies are actually enforced. Second, Hubble shows every flow between pods together with its verdict, so you can see what was blocked and by which rule. Third, Cilium has its own CiliumNetworkPolicy, which can express things a plain NetworkPolicy cannot.

Proof from the datapath, not from a status

The most important decision in this migration was to prove enforcement on real traffic, not on a status in Argo. The test was simple: two pods, the connection works, I apply a “deny-all” policy, the connection has to fail and Cilium has to log the drop. Then I delete the policy and the connection comes back.

$ kubectl -n kube-system exec ds/cilium -- cilium-dbg monitor --type drop
# recorded on 20.08.2026; pod addresses replaced
xx drop (Policy denied) flow … file bpf_lxc.c:2412, identity 5825->64145: <pod-a>:32982 -> <pod-b>:8080 tcp SYN
# with the policy deleted the connection returns 200 again; under Flannel the same policy did nothing
$ kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list

cilium-dbg endpoint list shows, for every endpoint, whether ingress and egress policy is enabled. That is what I trust, not a green icon.

Policies: deny by default, allow explicitly

I rolled the baseline out one namespace at a time. Where data lives, the default deny covers both directions. Elsewhere, denying ingress is enough to start with, because, for example, monitoring has to be able to reach everything. Some places deserve a second thought before a default deny: the cluster’s own system components, because a mistake there costs the whole cluster; namespaces with webhooks the API server calls, because a bad rule there looks like an API outage; and wherever you start a debug pod during an incident, because an incident is the wrong time to discover a policy.

Every new app brings its own policies in its own directory and starts in a namespace that denies both directions. The rule people forget most often is DNS:

# deny both directions for the whole namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny, namespace: example}
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# without this nothing in the namespace can resolve a name
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: allow-dns-egress, namespace: example}
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}
          podSelector: {matchLabels: {k8s-app: kube-dns}}
      ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
---
# the API server runs in the host network, so no ipBlock will match it;
# Cilium can name it directly
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata: {name: allow-kube-apiserver-egress, namespace: example}
spec:
  endpointSelector: {}
  egress:
    - toEntities: [kube-apiserver]

The last rule has a story. The database namespace runs a backup Job that talks to the Kubernetes API through the database operator. Under the default deny alone it would die silently, with no error you would notice right away. A CIDR rule would not help, because Cilium classifies traffic to the node address as host, not as an IP from a pool. Only toEntities: [kube-apiserver] does the job. The lesson stuck: I check every policy against periodic jobs, not only against what fails right away.

Hubble: a traffic map I do not expose

Hubble UI draws a live graph of connections between services and is excellent for writing policies: you see who talks to whom before anything gets blocked. It has one drawback: no login of its own. Publishing it at a public address would show the map of the whole cluster to anyone who guesses the name. So it is reachable only over a private VPN, and the proxy pod in front of it has its own default deny.

What I learned

• “Synced” is not “works”. An object in the API is a declaration. Enforcement has to be proven on traffic: a dropped packet in the log, and connectivity back once the rule is gone.
• Exceptions to default deny are decisions to write down, not a backlog. For every namespace without a deny, the repository says why.
• Do not render the network chart by hand. My local Helm, a different version, rendered the Cilium chart with small differences that, once applied, would have changed the DaemonSet template and restarted networking for the whole cluster. Argo CD manages that chart, and only Argo CD.