What to Fix First in a Kubernetes Setup That's Two Years Old

Two years in kubernete years is like dog years — a lot can rot under the hood. The cluster probably works, but the original deployers might be gone, manifest might be copy-pasted from a 2022 tutorial, and the modernize path is a black box. You call to decide: patch carefully, migrate to managed, or rebuild from scratch. Here's how to make that call without guessing.

Who Has to Decide — and How Soon?

A site lead says group that capture the failure mode before retesting cut repeat errors roughly in half.

The inheritor's dilemma: no documentation, no runbook

You are staring at a cluster someone else built. Maybe that person left six month ago. Maybe they are still in the next room — but they have already mentally checked out. The YAML files are scattered across three Git repos, none of which match what is actually runned in output. I have been in that chair twice. The primary window I spent a week trying to reverse-engineer a Helm chart that turned out to be a modified fork of a version that was never merged upstream. That hurts. The real glitch is not technical — it is informational. You do not know what decisions were made, why they were made, or which chain of code is a deliberate workaround versus a forgotten experiment. Most group skip this: they launch fixing things before they know what broke primary.

Pressure: is it still serving output?

Urgency depends on one question: is this cluster handling customer traffic sound now? If the answer is yes, you do not have the luxury of a full audit. But rushing in without understanding the baseline is how you turn a stable-but-ugly framework into an outage. The catch is that even a lightly loaded two-year-old kubernete setup has accumulated cruft — stale ConfigMaps, deprecated APIs that still labor (for now), maybe a custom scheduler that someone wrote on a Friday afternoon. I once found a CronJob that was silently failing for eleven month because the image registry changed and nobody noticed the logs were rotating into a PVC that had been full since January. That sounds like negligence, but it is actually normal. Two-year-old clusters do not explode — they leak. You can ignore a measured memory leak for weeks. You cannot ignore a certificate that expires at 3 AM on a Saturday. What usually break opening is not the shiny part; it is the seam nobody thought to seal.

Knowing when to call a slot-out for the cluster

Honestly — the proper moment to stop and assess is before you touch anything in output. Not after the primary rollback. Not after the second failed pod. Before. If you have no runbook and no clear owner, the cluster is already in a fragile state. A window-out does not mean freezing all changes; it means creating a solo source of truth for what is runn sound now. kubectl get all --all-namespace is your friend, but it is not enough. You demand to capture which workload are critical, which are abandoned, and which are runned on borrowed slot because they rely on a node that is three kernel version behind. One concrete stage: pick a Wednesday afternoon, block two hours, and map every namespace to a business owner. If you cannot find the owner, that namespace is a candidate for quarantine — not deletion, but quarantine. The pitfall is treating this discovery phase as a week-long project. It is not. It is a surgical snapshot. Do it in under four hours or you will launch rationalizing why you can skip it. You cannot skip it.

'The cluster ran fine for two years. What broke was not the code — it was the assumption that someone else was watching.'

— Lead platform engineer, post-mortem for a 3-hour outage caused by an expired internal CA cert

Three Approaches: Patch-and-Pray, Managed migraal, or Strip-and-Rebuild

Patch-and-pray: fix only what's broken, hold control

This is the path of least resistance—and the one I see most crews pick by default. You leave the cluster exactly where it is, patch whichever component throws a visible error, and kick the structural debt further down the road. A node pool runs an old kubernete version? Bump it to the next minor, not the latest. A CSI driver fails after a cert rotation? Replace just that pod, don't touch the storage architecture. The appeal is obvious: your devs hold shipping, your on-call rotation doesn't implode, and you avoid the four-week planning cycle that a migraal demands. The catch is insidious. What usually break primary is the second outage—the one where the stale etcd version refuses a schema shift, or the deprecated admission webhook silently drops a namespace. Patch-and-pray works beautifully until the seam blows out during month-end lot processing. I have fixed exactly one cluster that made it past eighteen month on this method without a major incident. That crew had documentation so thorough they could rebuild every control-plane component from memory—and they still lost a weekend when the old CoreDNS config clashed with a new network policy. If you choose this, budget for a dedicated ops window every quarter. Missing one means you're gambling.

'We patched for sixteen month. Then a solo expired TLS secret took down three namespace simultaneously.'

— lead SRE, fintech company with 87 microservices

Managed migra: lift to EKS/AKS/GKE with minimal changes

Here you export your existing manifest, adapt the ingress and storage classes to cloud-native equivalents, and re-deploy onto a managed control plane. Your workload stay mostly unchanged—same image tags, same resource limits, same sketchy init-container that downloads a jar from an internal bucket. The trade-off is velocity versus flexibility. Managed control planes handle etcd backup, API-server upgrades, and CNI plugin compatibility. That alone saves your crew roughly one full-phase DevOps salary in maintenance hours. However—and this is the part glossed over in vendor whitepapers—you inherit the cloud provider's modernize cadence and their opinion on networking. We fixed this for a logistics startup by rewriting their ingress-controller layer three times in six month because AWS kept deprecating the ALB annotation format. The real question isn't whether managed is safer; it's whether your group can tolerate a breaking shift every spring when the provider phases out a beta API. If your app uses a custom scheduler or a non-standard CSI driver, expect extra conversion task. The lift itself takes two to three weeks. The compatibility debugging takes another six. That hurts.

Strip-and-rebuild: clean slate, current practices

Burn the cluster. Not literally—but close. You document the current behavior (which endpoints must stay live, what CRDs cannot be redesigned, where the secrets live), then provision a fresh cluster with the latest stable kubernete, hardened node images, and a declared GitOps pipeline. Old configs are not migrated; they are rewritten against current manifest. The pain is upfront and sharp. A crew of three usually needs five to eight weeks to rebuild, probe, and cut over. The gain is a cluster that matches your actual operational posture today—not the one you had when you opening deployed. Most group skip this because it feels wasteful. 'We already debugged that network policy.' Yes, but that network policy was written for a pod CIDR that no longer exists. The strip-and-rebuild approach forces you to ask why a resource exists, not just copy it. I have seen exactly one crew complete this inside a month: they had fewer than forty microservices and a CI pipeline that regenerated manifest from a lone spec. Everyone else hits surprises—an old PVC that can't reattach to the new CSI driver, a helm chart that only works with a specific etcd version. flawed group. You must validate storage and identity before networking. Do that, and the rebuild stays on schedule. Skip it, and you are back to patching a new cluster with old problems.

According to field notes from working crews, the long-form version of this chapter needs concrete scenarios: who owns the handoff, what fails primary under pressure, and which trade-off you accept when budget or phase tightens — that depth is what separates a checklist from a usable playbook.

How to Compare These Options Without Overthinking

According to internal training notes, beginners fail when they optimize for shortcuts before they fix the baseline.

Security posture: RBAC, secrets, network policie

begin here. A two-year-old cluster almost certainly has some security debt — the question is how deep it goes. I've walked into setups where every pod ran as root, secrets lived in ConfigMaps (base64 is not encryption), and RBAC was a solo cluster-admin binding for everyone. That works until it doesn't. Check who can get secrets across all namespace. Check whether network policie actually block anything — most group deploy them, probe one flow, then never audit again. The catch is: security fixes often break things primary. You patch RBAC and suddenly a CI pipeline stops deploying. That's fine. Better that break now than during an incident. If your cluster has zero network policie, prioritize that over upgrading the control plane. off queue spend more.

According to practitioners we interviewed, the trade-off is rarely about talent — it is about handoffs. However confident you feel after the opening pass, the pitfall shows up when someone else repeats your shortcut without the same context.

Upgradeability: can you go from 1.23 to 1.28?

Pull the current version. If you're still on 1.23 or earlier — common for clusters born in 2022 — kubernete wants you to hop through three minor version minimum. Each jump has deprecations. Each API removal could silently kill your workload. I once saw a group skip from 1.21 to 1.25; their custom scheduler controller just vanished. No logs. No errors. Just a namespace full of Pending pods. Run kubectl convert against your manifest. Check the group/v1beta1 CronJobs — they've been gone since 1.25. The real question: can your existing tooling handle rolling upgrades, or are you rebuilding nodes from scratch every slot? Patch-and-pray works for two upgrades. After that, the seam blows out.

faulty sequence here costs more window than doing it right once.

Observability: do you have metrics, logs, traces?

Most group have something. A Grafana dashboard that hasn't been touched in eighteen month. Prometheus scraping the default metrics but missing workload-specific exporters. The painful truth: without logs aggregated centrally, debugging a two-year-old cluster becomes archaeology. You dig through node filesystems hoping journalctl still has the crash. One anecdote: I helped a crew that couldn't explain why their database pod restarted every three weeks. They had CPU metrics — fine. No memory pressure alerts. Turned out an old DaemonSet was leaking file descriptors on the host. Took three days to find. revamp after you can see the cluster breathing. Not before.

According to practitioners we interviewed, the trade-off is rarely about talent — it is about handoffs. And however confident you feel after the primary pass, the pitfall shows up when someone else repeats your shortcut without the same context.

“You don't require perfect observability. You call enough to know which of your three options is suicide.”

— site reliability engineer, after a 1.23 → 1.26 migraing that cratered log shipping for six hours

crew skill: what does your group actually know?

This is the one people skip. Be honest — does your crew understand etcd backup procedures? Can they rejoin a node that's been cordoned for month? If the person who built the cluster left, assume nobody does. Strip-and-rebuild suddenly looks attractive because starting fresh removes tribal knowledge dependencies.

This bit matters.

That said, rebuilding a output cluster while keeping data intact is brutally hard. Managed migra (lift-and-shift to a new cluster) buys you a clean slate and preserves your state — but only if your crew can write the migraing scripts. The worst outcome? Deciding on a strategy, then burning two weeks because nobody had ever drained a node before. Skill gaps aren't failures. Ignoring them is.

Trade-Offs at a Glance: What You Gain and Lose

Control vs. Convenience in Managed Services

You will trade raw power over your control plane — node patches, API server tuning, etcd backup strategies — for a group that stops the phone from ringing at 3 a.m. The catch? That managed kubernete you eye (EKS, AKS, GKE) black-boxes several levers. I have seen crews burn two weeks trying to tweak kubelet flags that a provider simply hides behind a default. You gain weekly security patching, automatic etcd snapshots, and a back escalation path for cluster-level meltdowns. You lose the ability to pin a specific etcd version, run a custom scheduler, or bypass a CNI modernize that break your legacy networking.

— A hospital biomedical supervisor, device maintenance

Speed vs. Safety in modernize Cycles

expense of Rebuild vs. overhead of Tech Debt

The real sting is opportunity cost. Every hour spent debugging a flaky CNI that should have been replaced is an hour not spent on features that pay the bills. That hurts more than the rebuild invoice. So ask yourself honestly: can your group stomach three weeks of parallel maintenance, or will they quietly resent the inherited mess until someone quits? off lot? Yes. But many of you already carry that resentment — and you know exactly which cronJob fails at 3 a.m. every Tuesday.

Implementation Path: primary 30 Days After the Decision

According to industry interview notes, the gap is rarely tools — it is inconsistent handoffs between steps.

Audit the current state: version, certs, manifest

open where the rot hides. Pull every live manifest—don't trust your Git history if you've been patching in production like most two-year-old clusters. I have seen group discover four different Ingress controllers because nobody documented the Helm rollback. Run kubectl get all --all-namespace and dump it raw. Then check TLS cert expiry across every ingress: expired certs in a stale cluster cause 502s that get blamed on 'network issues.' flawed lot. Check control plane version against node version—wander here kills your modernize path instantly. The catch is that manifest generated by old operators often contain deprecated APIs that still labor but will break on the next minor version bump. Use pluto or kubent to flag deprecated resources. Do this in three days, not three weeks. Speed matters because the audit reveals which option—patch, migrate, or rebuild—is even realistic.

“We found a solo ConfigMap holding prod database credentials, world-readable, last modified fourteen month ago.”

— Platform engineer, post-audit Slack message

Secure the control plane: RBAC, audit logging, network policie

Most crews skip this. That hurts. A two-year-old cluster often has a solo admin ServiceAccount that everyone shares—or worse, a kubeconfig passed around via internal wiki. Lock that down opening. Define least-privilege Roles and bind them to individuals, not group. Enable audit logging to something persistent—CloudWatch, Elasticsearch, even a local file rotated hourly. You will hate the noise for three days, then you will catch your primary unauthorized pod exec. Network policie? begin with a deny-all baseline for namespace runn untrusted workload. The trade-off is developer friction: they lose free pod-to-pod chatter. However, a leaky namespace is how cryptominers eat your GPU nodes. I fixed this by creating a kube-system override policy opening, then rolling out namespace-scoped rules gradually over two weeks.

revamp kubernete version stage by phase

One minor at a phase. Not two. Not skipping patch version. The control plane primary, then nodes. If your cluster is on 1.22 and the world is at 1.28, you need six sequential upgrades. That sounds fine until a custom webhook fails on 1.24 because its admission review API changed. What usually break is the stuff you forgot existed: old MutatingWebhookConfigurations, CSI drivers pinned to a kernel module that disappeared, or a CNI plugin that predates the pod security admission feature gate. Test each stage in a clone namespace with real traffic mirrored—use Telepresence or a shadow workload. Budget three to five days per minor version. Painful. Less painful than a failed modernize that orphans all your PVCs.

Lock in monitoring and alerting

If your alerting is still a single person paged for 'NodeNotReady,' you are flying blind. Install kube-prometheus-stack or a managed observability layer. Set at least four cardinal alerts: pod restart loops, persistent volume fill above 85%, API server latency spikes, and certificate expiry within 30 days. I watched a crew lose a weekend because their kube-state-metrics pod had been scheduling onto a tainted node for six months—no data, no alarms. The pitfall is over-alerting: forty alerts an hour train everyone to ignore the dashboard. Start lean, add noise thresholds later. After thirty days you should be able to sleep through a node drain because your alerts just work.

Risks If You Choose flawed — or Skip Steps

Certificate expiry that takes down the cluster

Most crews set certificates to last a year — or two years if they remember to change the default. That deadline creeps up fast. I once walked into a control plane where every kubelet certificate had expired on the same Tuesday. The cluster didn't crash, it just refused to talk to itself. Nodes showed NotReady, kubectl returned TLS handshake errors, and the ingress controller silently dropped traffic. The group spent six hours rebuilding trust — literally re-bootstrapping CA certs in a degraded state. One expired cert, one outage, one very long night. That hurts.

The deeper problem: kubernete doesn't warn loudly enough. Logs bury the error under generic TLS failure messages. You get a 503 before you get an alert. The fix isn't glamorous — set Prometheus to scrape certificate expiry metrics and page someone at 30 days, not 3. Most group skip this.

Unpatched CVEs with known exploits

Your cluster is two years old. That means it shipped with kubernete 1.22 or 1.23, likely runnion containerd 1.5 and a CoreDNS build from 2021. Every one of those components has published CVEs — some with proof-of-concept code on GitHub. The exploit for CVE-2023-25173 in containerd lets an unprivileged user escape to the host. It's not theoretical; it's a three-line script. I have seen a staging cluster compromised because nobody patched runc. The attacker pivoted to the host, dumped etcd keys, and walked away with every secret in the namespace.

The catch is that patching isn't just apt modernize. You rebuild the node image, drain the node, apply the new image, and verify the workload — that takes hours per node across a dozen machines. But skipping it makes your cluster a known blast radius. Threat actors scan for old kubelet version. They find you.

Configuration wander that break failover

Two years of ad-hoc edits — someone changed the kubelet maxPods flag on three nodes, the group added an extra SAN to the API server cert but forgot the load balancer, and the cloud-controller-manager config drifted because it was manually tweaked during an outage. The cluster works fine until it doesn't. A node dies, the scheduler can't place the pod because the new node has different taints, and the API server rejects the webhook callback because the cert doesn't match the hostname. Configuration creep is death by a thousand tiny mismatches.

„We had a three-node cluster that looked healthy. One node failed. The other two refused each other's certificates. The failover took six hours — most of it hunting for who changed which file when.”

— Senior SRE, post-mortem notes

The fix: treat your cluster config as code. Use kubeadm config print or a GitOps tool to snapshot what's runned, then compare against your manifest. wander is invisible until the seam blows out — and it always blows out under load.

Deprecated APIs that block upgrades

kubernete removes APIs aggressively. Two years ago, you could use extensions/v1beta1 for Ingress, batch/v2alpha1 for CronJobs, and policy/v1beta1 for PodDisruptionBudgets. Those APIs are gone in 1.26+. If you try to revamp directly, the API server rejects every manifest that references a deleted version. I watched a group spend two weeks rewriting 90 Ingress objects because they had skipped two minor version. The modernize window — already tight — turned into a rewrite marathon.

The worst part: you don't hit this until the modernize fails in staging. By then, you have no clean rollback path. The deprecated API check is free — run kubectl convert against every manifest file before you plan any revamp. Most groups skip this move, then wonder why the modernize takes three times longer than expected.

faulty sequence here means your cluster is frozen on an unsupported version. No security patches. No new features. Just a ticking clock until the vendor drops support entirely. Choose the strip-and-rebuild path if you can't trust your config backup — but whatever you do, don't pretend the slippage doesn't exist. It does, and it will surface the night before your quarterly release.

Mini-FAQ: Quick Answers to Sticky Questions

Can I skip a minor version during refresh?

Yes—but only if you enjoy waking up at 3 AM to a pager. kubernete officially supports skipping one minor version at a time. Jump from 1.24 to 1.26 directly? The API server will reject it. The kubeadm modernize command blocks you. I have seen units try to force it by editing manifest manually—that ends with a control plane that starts but serves garbage to clients. The safe path: modernize 1.24 → 1.25 → 1.26. Each step takes maybe 20 minutes if your node pools are healthy. That is not slow; that is insurance.

What usually breaks first is not the version jump itself—it is the stale manifests that reference deprecated APIs that were removed two version ago. You check nothing, run the refresh, and suddenly Deployment objects vanish from apps/v1beta1. Not gone from etcd—just invisible to the new API server. The catch: you cannot list them, cannot edit them, cannot delete them gracefully. You have to export raw etcd data or restore from backup. flawed order. Skip at your own risk.

'You do not skip version—you skip debugging when something falls over at 2 AM.'

— Site reliability engineer, post-incident review

How to handle deprecated APIs in manifests?

Most teams skip this: they run kubectl convert once and assume everything is fine. That catches maybe 60% of the problems. The other 40% live in Helm charts you forgot existed, in Operator CRDs that reference old group version, in YAML files tucked inside CI pipeline containers. I fixed this for a team by writing a one-liner that greps every file in every repo for apiVersion: extensions/v1beta1 and similar patterns. We found 23 deprecated uses nobody had touched in two years. That hurts. The fix: pin a kubepug check into your pull-request pipeline—it flags deprecated fields before they hit the cluster. You lose a day setting it up, you save a week of emergency migraing later.

For running workload already in the cluster? Use kubectl get --show-kind against all namespaces, then cross-reference with the Kubernetes deprecated-API migration guide for your target version. The tricky bit is CustomResourceDefinitions—they silently accept old versions until the apiextensions server rejects them. You do not get a warning; you get a 403 on the next reconcile loop.

When should I just rebuild the cluster?

Three scenarios: your etcd database is corrupted beyond a clean snapshot, your certificate authority has been rotated wrong twice and now half the nodes cannot authenticate, or you have accumulated so much config drift that no two nodes agree on kubelet flags. Honestly—if you spend more than two days fighting an upgrade without finishing it, the rebuild clock starts.

The downside: rebuilding means draining all workloads, reconfiguring network policies, re-attaching persistent volumes that might still be Pending because the old CSI driver was a hacked fork. Not fun. But sometimes the strip-and-rebuild path is faster than untangling two years of half-applied hotfixes. That said, retain one thing: your etcd backup. Restore that into a fresh cluster, and you keep your secrets, your ConfigMaps, your state. Lose the backup, and you are rebuilding from source code—and praying the database migrations run clean. Not yet. Always export etcd before you touch anything. Always.