System: Kubernetes Cluster Architecture Lab
Category: DevOps · Kubernetes & Orchestration
Reading Time: 6 min read
Deploying local k3s clusters: what I learned about readiness probes and rolling updates
Cluster orchestration, Service discovery, ConfigMaps, and zero-downtime rollouts.
Covered: k3s · Minikube · Ingress · ClusterIP · Readiness/Liveness Probes · ConfigMaps
01. WHY I BUILT IT
The Trigger & Underlying Problem
The Trigger: Moving from Docker Compose to multi-host orchestration required mastering Kubernetes abstractions directly.
The Core Problem: Docker Compose works well on a single host, but has no built-in mechanism for automated self-healing, rolling updates without downtime, or declarative service discovery across nodes.
Why Standard Solutions Were Insufficient: Running local lightweight clusters (k3s / Minikube) provides a realistic environment to experiment with manifests without incurring cloud cluster costs.
02. PRACTICAL IMPACT
What It Solves in Practice
PRACTICAL PROBLEM
Deploys a multi-tier microservice architecture declaratively, handles rolling upgrades automatically, and routes edge traffic through Ingress.
BENEFICIARIES
Provides reproducible deployment manifests ready to port to managed cloud engines like Amazon EKS.
BEFORE THIS SYSTEM
Single-node deployments that experienced downtime during updates.
03. IMPLEMENTATION
Architecture & Key Engineering Decisions
Local k3s cluster running Deployments, ClusterIP Services, Traefik Ingress, and ConfigMaps for application configuration.
Key Decisions:
- Configured rolling update strategy with `maxSurge: 1` and `maxUnavailable: 0` to prevent drop-offs during image rollouts.
- Separated configuration into ConfigMaps and sensitive credentials into Kubernetes Secrets.
- Implemented dedicated liveness probes on `/health/live` and readiness probes on `/health/ready`.
04. CHALLENGES & SOLUTIONS
What Failed in Practice & How It Was Addressed
WHAT WENT WRONG
- Initial rolling updates still dropped HTTP requests because the service routed traffic to newly started pods before their Python runtime had initialized database connections.
- Pods entered `CrashLoopBackOff` when a missing ConfigMap environment variable caused the application to exit immediately on boot.
HOW I SOLVED IT
- Configured explicit readiness probes with `initialDelaySeconds: 5` and `periodSeconds: 10`. Kubernetes now withholds traffic until the readiness probe confirms database connectivity.
- Added a preStop lifecycle hook (`sleep 5`) to allow in-flight connections to drain before the container receives SIGTERM.
05. REFLECTION & LESSONS
Takeaways & Future Iterations
What I Learned:
- The critical difference between liveness probes (restarts unhealthy pods) and readiness probes (removes pods from Service endpoints during startup or saturation).
- How kube-proxy updates iptables rules to distribute traffic across pod endpoints.
WHAT I WOULD CHANGE NOW:
I would package the manifests into Helm charts with `values.yaml` to manage configuration differences across environments cleanly.