SIOPS — production Kubernetes platform
An RKE2 cluster on OpenStack run by a team of five, four tenants in production, and the incidents that come with it.
The platform
SIOPS is the in-house DevOps/SRE team of the SIGL major. It runs a Kubernetes cluster that is genuinely in production: other teams host their applications on it, connect to it daily, and complain when it breaks. This is not a lab we rebuild every Monday.
RKE2 v1.32 · 6 nodes on OpenStack · Flannel CNI · Cinder CSI storage
~60 applications reconciled by ArgoCD · 4 tenants in production + staging
Observability: Prometheus · Grafana · Loki · Alloy · Gatus
~440 commits over the semester, team of 5, Kanban on Linear
My role — Product Owner
I own the team backlog and decide what ships. In practice: turning a tenant request into a deliverable increment, deciding what we will not do this sprint, and defending that call to tenants and to the team alike.
It is not a figurehead role. I am in the logs and on the post-mortems — which is exactly why the trade-offs hold up. I know what a request that looks trivial actually costs.
Cassiopée — GitOps self-service portal
Before Cassiopée, every environment request went through a member of the platform team. The portal turns that path into Git-driven self-service.
A user requests an application, a database, an S3 bucket or an ingress. The portal
commits the matching values.yaml to the GitOps repository; ArgoCD
reconciles. Credentials are generated on the fly and stored in Vault — they never touch
the repository. Quotas are enforced per group.
FastAPI · ArgoCD · Helm · HashiCorp Vault · Swift / RGW
Outcome: a complete environment deployed by commit, with no platform team involvement. [TODO: metric] number of environments provisioned through the portal.
Incidents & post-mortems
INC-01 — The cluster deadlocks itself on restart
Symptom. master-1 unreachable, unstable API server, cluster running degraded.
Diagnosis. The node was undersized. Under memory pressure the OOM
killer takes out etcd and quorum drops to 2/3. Worse, recovery blocks itself:
rke2-server waits on kubelet, kubelet waits on the etcd static pod, and etcd
no longer starts. A three-way deadlock.
Fix. rke2-killall to break the loop, then a clean node
restart — and resizing the master so the cause does not come back.
Lesson. A control plane is not sized like a worker. etcd's resources are an availability constraint, not a line item to trim.
INC-02 — Three weeks without backups, and the jobs were green
Symptom. No recent snapshot in the repository. The backup jobs kept reporting success.
Diagnosis. The 500 GiB S3 quota was full. The restic repository held 486 GiB across 349 snapshots that had never been pruned: the dump name embedded the date, so every backup created its own group and the retention policy applied to nothing. The cleanup had never had anything to clean.
Fix. Upgraded restic, then forget --group-by host
followed by prune: the repository went from 486 GiB to 48 GiB.
Backup script hardened with set -euo pipefail and a stable dump name.
Lesson. A green job proves nothing. What you monitor is the age of
the last restorable snapshot — not the cron's exit code. And a script without
set -e lies about its own status.
INC-03 — The Vault unseal chain eats its own tail
Symptom. Audit of Vault's startup chain, carried out after a platform restart.
Diagnosis. The primary Vault's transit auto-unseal depends on a secondary Vault, itself Shamir-sealed and deployed as a single replica. If the secondary goes down, nothing unseals — including the secrets needed to bring it back. A SPOF, with a circular startup dependency.
Fix. SPOF identified, documented and added to the backlog with its exit options. [TODO: remediation status]
Lesson. A circular startup dependency is invisible in steady state. You find it the day everything restarts at once — which is the worst possible day.
Resource pooling — a product decision
Each group had a fixed quota, sized tightly. The result was the worst of both worlds: reserved capacity sitting idle on one side, teams reduced to shipping mock-ups on the other.
I pushed the move to a shared pool. Capacity goes where the real demand is, instead of being frozen in a quota table. The trade-off is explicit: we accept some contention risk so that teams can run real projects rather than demos.
[TODO: metric] pool utilisation rate before and after the switch.
What I took from it
The technical part is learnable. What takes longer is deciding under constraint: saying no to a legitimate request because the capacity is not there, carrying a known SPOF while it gets fixed properly, and explaining both to a tenant without hiding behind jargon.