All writing

Home LabBy 3 min read

Kubernetes on a stack of Raspberry Pis

Seven Raspberry Pi 5 boards running upstream Kubernetes with kubeadm and Argo CD: what the cluster runs, what broke on ARM, and why small hardware makes you a better platform engineer.

Updated

On this page · 4 sections
  1. The cluster
  2. Git is the only way in
  3. What broke
  4. What it taught me

Managed Kubernetes hides a lot from you. That is the point of it, and it is also why I run my own cluster at home: seven Raspberry Pi 5 boards running the services my family and I use every day. It is where I try out ideas without a cloud bill or a production blast radius.

The cluster

PartWhat I run
Control plane1 × Raspberry Pi 5, 8 GB, tainted so workloads stay off it
Workers6 × Raspberry Pi 5, 16 GB each
OS and runtimeUbuntu 24.04 LTS (arm64), containerd
KubernetesUpstream v1.35, installed with kubeadm
Networkingflannel for pod networking, MetalLB in L2 mode for LoadBalancer IPs, ingress-nginx
TLS and secretscert-manager with Let’s Encrypt DNS-01, Sealed Secrets
StorageNFS and iSCSI from a separate NAS (see my NAS post)
DeliveryArgo CD, about 40 applications
Monitoringkube-prometheus-stack (Prometheus, Alertmanager, Grafana), alerts to Telegram

That is 28 cores and a little over 100 GB of RAM. On a normal day the workers sit at 8 to 25% CPU and 32 to 62% memory. On it run Immich for family photos, n8n, Postgres, Redis, Mosquitto for MQTT, Qdrant, SearXNG and a few more.

I chose kubeadm over a lightweight distribution like k3s on purpose. I wanted the same moving parts as a production cluster: etcd, the API server, the controller manager and the scheduler as separate static pods that I have to keep healthy myself.

Git is the only way in

Every application is an Argo CD Application that points at a Git repo. Here is the one for Immich:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: immich
  namespace: argocd
spec:
  project: home-lab
  source:
    repoURL: "git@github.com:<user>/immich.git"
    targetRevision: main
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: immich
  syncPolicy:
    automated:
      prune: true      # delete what was removed from Git
      selfHeal: true   # undo manual changes in the cluster
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

With selfHeal on, a quick kubectl edit is reverted within minutes. That felt annoying for the first week. After that it became the reason I trust the cluster: what is in Git is what is running.

What broke

ARM images. Not every image ships an arm64 build. MongoDB Ops Manager only publishes amd64 images, and on the Pis it crash-looped with exec format error, so I replaced it with the MongoDB Community operator. Immich failed in a quieter way: I had pinned images by @sha256 digest, and those digests pointed at the amd64 manifest, so containerd could not pull them. Pinning by tag fixed it.

Missing resource requests. The Argo CD server had no CPU requests at all. When three ClickHouse replicas landed on the same node, they took the CPU, Argo CD’s gRPC calls slowed to 0.7 to 1.2 seconds against a one-second readiness probe, and nginx started returning 503s. Adding requests fixed it. On small machines, bad sizing shows up within hours.

Storage latency. A three-node Elasticsearch cluster on NFS kept losing its master, because writing cluster state to the NAS took 15 to 60 seconds and Elasticsearch gives up after 30. I cut it to a single node with shorter retention. Some workloads do not belong on network storage, however convenient it is.

Things installed by hand. I had installed ingress-nginx manually at v1.12.0, which is affected by CVE-2025-1974 (“IngressNightmare”). Nothing reminded me to upgrade it because Argo CD did not know it existed. I moved it under Argo CD at v1.15.1 and fixed the forwarded-IP trust settings so the LAN-only allowlists cannot be spoofed. Argo CD itself still comes from the floating stable manifest, and that bit me too: an automatic move to 3.5 changed a default flag and crash-looped the repo server.

What it taught me

A cluster of small ARM boards won’t let you hide sloppy resource management behind a bigger instance type.

Most of these failures would never reach me on GKE. The image would be built for the right architecture, the node would be big enough to absorb a missing request, and the storage would be fast. Running it myself means I have seen each of those failures once, at home, with nothing at stake but my family’s patience.