A Kubernetes Cluster of Mini PCs for 100+ Web Apps: k3s, Redundancy, and Logs on a NAS

The shelf

Mini PCs are the best value in computing I know of. A Beelink with an eight-core Ryzen, 32 GB of RAM and a fast NVMe costs less than two months of the equivalent cloud instance, draws 15 to 30 watts, and makes no noise. The fleet grew one box at a time, each bought for a job, and k3s is what let me stop thinking about which box a job lives on.

The cluster

k3s, because it is Kubernetes in one binary with the batteries I wanted (containerd, Traefik, CoreDNS, embedded etcd) and none I did not. Three servers for an etcd quorum, which is the smallest number that survives losing one; everything else is an agent. The Beelink the Tiny VPS product lives on is tainted so nothing lands there by accident, and the Dell is labelled so that the crawlers and queue processors, which want cores and RAM rather than a public port, schedule onto it.

# control plane, first node (server3)
curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --disable servicelb \
  --node-label role=control \
  --tls-san k3s.marie.lan

# the other two Minisforum boxes join as servers (etcd quorum of three)
curl -sfL https://get.k3s.io | K3S_TOKEN="$(sudo cat /var/lib/rancher/k3s/server/node-token)" \
  sh -s - server --server https://server3:6443 --node-label role=control

# every Beelink and the Dell join as agents
curl -sfL https://get.k3s.io | K3S_URL=https://k3s.marie.lan:6443 K3S_TOKEN=... \
  sh -s - agent --node-label role=apps
kubectl label node server1 role=processors
kubectl taint node server25 reserved=tinyvps:NoSchedule

$ kubectl get nodes -o wide
NAME       STATUS   ROLES                       AGE    VERSION        INTERNAL-IP
server3    Ready    control-plane,etcd,master   212d   v1.33.4+k3s1   192.168.4.49
server10   Ready    control-plane,etcd,master   212d   v1.33.4+k3s1   192.168.4.43
server14   Ready    control-plane,etcd,master   210d   v1.33.4+k3s1   192.168.4.46
server7    Ready    <none>                      212d   v1.33.4+k3s1   192.168.4.56
server9    Ready    <none>                      212d   v1.33.4+k3s1   192.168.4.91
server11   Ready    <none>                      180d   v1.33.4+k3s1   192.168.4.21
server25   Ready    <none>                      14d    v1.33.4+k3s1   192.168.4.28
server1    Ready    <none>                      212d   v1.33.4+k3s1   192.168.4.75

$ kubectl top nodes
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
server7    2890m        18%    19204Mi         68%
server9    610m         5%     9871Mi          35%
server1    4120m        17%    22310Mi         36%
...

Every host still has a GitHub Actions self-hosted runner, because runs-on: server7 remains the easiest way to run a known script on a known machine with secrets injected and a log kept. What changed is what the scripts do: they used to run docker run on that host; now most of them run kubectl against the cluster from wherever they happen to execute, and the host they execute on stopped mattering.

The deploy that every app shares

An app’s CI builds an image, pushes it to ghcr.io and fires a repository_dispatch at the devops repository with the image digest. The deploy workflow is now four lines, and the rollback is the one Kubernetes already knows how to do:

# devops/.github/workflows/deploy-8examples-website.yml
jobs:
  deploy:
    runs-on: [self-hosted, linux, x64, server23]     # anything with a kubeconfig
    steps:
      - run: |
          kubectl -n sites set image deployment/nextjs-8examples \
            web=ghcr.io/8exgh/8examples-website@${{ github.event.client_payload.digest }}
          kubectl -n sites rollout status deployment/nextjs-8examples --timeout=120s \
            || { kubectl -n sites rollout undo deployment/nextjs-8examples; exit 1; }

What the Deployment looks like depends on one question: does the app have a single writer? Nearly all of mine do, a Next.js container with an events.db SQLite file, and the manifest has to respect that. One replica, the Recreate strategy so two pods never hold the same file, and the file on a PersistentVolumeClaim so the pod can come back on a different node. Stateless apps get two replicas, a rolling update, and a spread constraint so the two never share a box:

# apps/8examples/deployment.yaml: one writer, one SQLite file, so one pod
apiVersion: apps/v1
kind: Deployment
metadata: { name: nextjs-8examples, namespace: sites }
spec:
  replicas: 1
  strategy: { type: Recreate }          # never two pods on one events.db
  selector: { matchLabels: { app: nextjs-8examples } }
  template:
    metadata: { labels: { app: nextjs-8examples } }
    spec:
      nodeSelector: { role: apps }
      containers:
        - name: web
          image: ghcr.io/8exgh/8examples-website:latest
          ports: [{ containerPort: 3000 }]
          envFrom: [{ secretRef: { name: nextjs-8examples } }]
          volumeMounts: [{ name: data, mountPath: /app/data }]
          readinessProbe: { httpGet: { path: /openclaw, port: 3000 }, periodSeconds: 5 }
          resources: { requests: { cpu: 100m, memory: 256Mi }, limits: { memory: 1Gi } }
      volumes:
        - name: data
          persistentVolumeClaim: { claimName: nextjs-8examples-data }   # NFS, on the NAS
---
# a stateless site gets two pods on two boxes and a rolling update
apiVersion: apps/v1
kind: Deployment
metadata: { name: flappy-bird, namespace: sites }
spec:
  replicas: 2
  strategy: { type: RollingUpdate, rollingUpdate: { maxUnavailable: 0, maxSurge: 1 } }
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector: { matchLabels: { app: flappy-bird } }

The readiness probe is a real request to a real page, which catches the class of failure where the process starts and the app does not; rollout status waits on it, and rollout undo puts the previous ReplicaSet back when it never comes true. A broken build never replaces a working site.

One front door, two connectors

No machine has a public port. Cloudflare Tunnel is the only way in: a cloudflared Deployment with two replicas, one on each of two control-plane nodes, both serving the same tunnel, pointed at Traefik’s Service. Every public hostname, from 8examples.com to the smallest restaurant site, is a CNAME to the tunnel, and an Ingress object per app tells Traefik which Service gets which host. Losing either connector loses nothing; a tunnel with replicas just stops using the one that went away. TLS, HTTP/3, caching and the WAF are Cloudflare’s problem, and the eero forwards nothing for the web.

Adding a site is a DNS record and an Ingress, which a workflow writes through the Cloudflare API and kubectl apply, so provisioning a customer’s site from a paid checkout to a live hostname is automatic end to end.

The commands I actually type

Most days the cluster needs nothing. When something is wrong, this is the order I ask questions in, from “is anything unhappy” down to “give me its database”:

# what is unhappy, anywhere
kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded
kubectl get events -A --sort-by=.lastTimestamp | tail -30

# one app: why is it restarting, and what did it say before it died
kubectl -n sites describe pod -l app=nextjs-statusnest | sed -n '/Events/,$p'
kubectl -n sites logs deploy/nextjs-statusnest --previous --tail=200

# what changed, and put it back
kubectl -n sites rollout history deployment/nextjs-statusnest
kubectl -n sites rollout undo deployment/nextjs-statusnest --to-revision=14

# a shell in the running container, or a copy of its database
kubectl -n sites exec -it deploy/nextjs-8examples -- sh
kubectl -n sites cp nextjs-8examples-7d9f4c6b8-x2k9p:/app/data/events.db ./events.db

# which node is full, and move things off a box before a reboot
kubectl describe node server7 | sed -n '/Allocated resources/,/Events/p'
kubectl drain server7 --ignore-daemonsets --delete-emptydir-data
sudo reboot            # on server7; single-replica apps come back on another node
kubectl uncordon server7

The drain-reboot-uncordon sequence is the one I am most grateful for. Kernel updates used to mean choosing an hour when a box’s apps could be down; now drain moves the two-replica apps’ traffic to their other pod and reschedules the single-replica ones onto another node, their data following them over NFS, and the reboot is invisible. For anything interactive I use k9s, which is the same information with the keyboard.

Where the redundancy is, and where it is not

Process level: everything

A pod that dies is replaced; a node that reboots gets its pods back or sees them rescheduled. This is the redundancy that has mattered most in practice, because the failures I actually get are a process dying or a box rebooting.

Deploy level: everything with a readiness probe

Every Deployment has one. The rollout waits on it and undoes itself when it fails.

Ingress level: two connectors, three control-plane nodes

Two cloudflared replicas on separate boxes and an etcd quorum that survives losing a Minisforum. The failure I have not engineered away is the home internet connection itself: if it drops, everything on the LAN drops with it, and only the OVH hosts stay up. That is why the things customers pay for monthly, the managed workers and the restaurant sites, live there and not on the cluster.

Host level: the stateless apps, and only them

Load balancing across boxes exists for exactly the apps where it is honest, which means the ones without a single-writer database: the game, the static-ish marketing sites, the relays. They run two replicas spread across nodes and Traefik balances between them. The other forty-odd apps have a SQLite file, and SQLite has one writer. Two pods against two copies of an event-sourced events.db is not redundancy, it is a split brain with a nicer name, so those stay at one replica with Recreate, and what the cluster gives them is not load balancing but mobility: when their node dies they come back on another one in about a minute, with their data, which is the redundancy that actually fits them. The ones whose traffic ever earns it get moved to Postgres and a second replica, one at a time. Most never will: a restaurant site serves a few hundred visits a day from a box that could serve a few hundred a second.

Every pod’s logs on the NAS

Logs used to live in each host’s Docker json-file and die with its disk; reading them meant a workflow per host. Now a Promtail DaemonSet on every node ships every pod’s stdout, labelled with namespace, app and node, to Loki, and Loki keeps its chunks on a PersistentVolume that is an NFS export on a UGREEN NAS on the same shelf. The NAS snapshots the volume nightly and keeps ninety days; the cluster keeps nothing it cannot lose. The same StorageClass backs every app’s data volume, which is what makes a single-replica SQLite app able to follow its pod to another node.

# monitoring/loki-values.yaml (helm: grafana/loki, single-binary mode)
loki:
  commonConfig: { replication_factor: 1 }
  storage: { type: filesystem }
  limits_config: { retention_period: 90d }
singleBinary:
  persistence:
    storageClass: nfs-nas          # the UGREEN export, see below
    size: 200Gi
---
# the StorageClass: one NFS export on the NAS, a subdirectory per claim
# helm install nfs nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
#   --set nfs.server=192.168.4.40 --set nfs.path=/volume1/k3s \
#   --set storageClass.name=nfs-nas --set storageClass.defaultClass=true

One grep on the NAS replaced eight workflows. These are the queries I run most:

# live, across every replica of an app, with stern
stern -n sites nextjs-8examples --since 10m

# the last hour of everything that logged an error, cluster-wide, from the NAS
logcli query --since=1h '{namespace="sites"} |~ "(?i)error|unhandled"' --limit 200

# one customer's site, one request id, however many pods it crossed
logcli query --since=24h '{app="openclaw-website---designsbylux"} |= "req_01J9"'

# what the Tiny VPS host agent has been doing
logcli query --since=6h '{namespace="tinyvps"} | json | line_format "{{.msg}}"'

Two gotchas worth the price of this post. NFS and SQLite only get along if exactly one process ever opens the file, which the Recreate strategy guarantees for the web pod but not for a background processor that opens the same database from another node; those pairs now talk over HTTP instead of sharing a file. And Loki’s retention needs the compactor enabled and a retention_period set, or the NAS fills quietly over a quarter and you find out from the NAS.

The operator is an OpenClaw worker

server23 is the Beelink on my desk, and it is not a node. It runs my own OpenClaw worker, the same product the managed fleet sells, with one addition: a kubeconfig for a ServiceAccount that can read everything in the cluster, including pod logs, events and node metrics, and can patch Deployments in the app namespaces, but cannot read a Secret or touch the control plane.

# the OpenClaw worker's identity: read everything, restart app Deployments,
# never read a Secret, never touch the control plane
apiVersion: v1
kind: ServiceAccount
metadata: { name: openclaw-ops, namespace: ops }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: openclaw-ops }
rules:
  - apiGroups: ["", "apps", "batch", "metrics.k8s.io"]
    resources: ["pods", "pods/log", "events", "nodes", "deployments", "replicasets", "jobs", "services", "endpoints", "persistentvolumeclaims"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["patch"]                # rollout restart / undo, in app namespaces only
    resourceNames: []
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: openclaw-ops, namespace: sites }
roleRef: { kind: ClusterRole, name: openclaw-ops, apiGroup: rbac.authorization.k8s.io }
subjects: [{ kind: ServiceAccount, name: openclaw-ops, namespace: ops }]

That is enough for the operations I used to do by hand. It has kubectl, stern and logcli on its path, it reads the same Loki, and I talk to it in Rocket.Chat from my phone:

me:    statusnest has been flapping since lunch, what's going on?
claw:  nextjs-statusnest restarted 6 times in the last 2h on server7.
       Last exit was OOMKilled at 13:41; memory limit is 512Mi and the
       pod reached 511Mi each time right after the 13:40 crawl job.
       Previous logs end with "sqlite busy" from background-processor-statusnest,
       which runs on server1 against the same NFS volume.
       Suggested: raise the limit to 1Gi, or stop the processor opening the
       web pod's database. I have not changed anything.
me:    bump it to 1Gi and restart it
claw:  Patched deployment/nextjs-statusnest limits.memory 512Mi -> 1Gi
       (revision 23) and rolled it; 1/1 ready on server11 after 14s.
       I will tell you if it restarts again in the next hour.

The RBAC is the whole safety story and it is deliberately boring: it can look at anything and restart or roll back an app, which is what I would do first anyway, and it cannot do the things I would want to be awake for. Scaling a Deployment, changing a limit, or touching a volume are patches too, so they are inside the role; deleting anything, reading credentials, and the control plane are not. When it is unsure it says so and does nothing, which is the behaviour I want from a junior on call.

What the move taught me

State decides the architecture. Every manifest above follows from choosing SQLite-per-app for simplicity and then respecting what that choice forbids: one replica, Recreate, a volume that can move. Kubernetes did not change that; it made the consequences explicit.

Keep the front door off the worker nodes. Every outage I caused myself came from a deploy; none touched the tunnel because the connectors only run on control-plane nodes.

Three small boxes for the control plane beat one big one. A Minisforum costs less than a weekend of debugging a single-node etcd that lost its disk.

Put customers’ things off the LAN. My home connection is the single point of failure I cannot fix, so the things that bill monthly do not depend on it.

Give the on-call agent a kubeconfig, not root. A ServiceAccount with a narrow role turns an AI worker from a risk into the first responder.

If you run a shelf like this and have a better answer for SQLite on NFS, the comments are open. If you would rather someone ran a box for you, that is what the managed workers are, and the operator above is one of them.