Case Study: Building an Enterprise Kubernetes Platform That Cut Costs 40%

This is a detailed look at one of the more significant platforms I've built and operated, a production Kubernetes platform on AWS EKS serving 50+ microservices and over a million requests a day. Names and specifics are generalised for confidentiality, but the architecture, decisions, and outcomes are real.

The starting point

The team was running ~50 microservices on a sprawl of EC2 instances managed largely by hand. Symptoms:

• Deployments were weekly and scary. Releases were manual, inconsistent, and often rolled back.
• Costs were climbing with no clear owner: heavily over-provisioned instances running at 10-20% utilisation.
• Incidents took hours to resolve. Limited observability meant a lot of "SSH in and guess."
• No reproducibility. Infrastructure lived in people's heads and the AWS console.

The mandate: make delivery fast and safe, get costs under control, and make the system observable, without a big-bang rewrite.

The target architecture

                         ┌────────────────────────────┐
        Git (source of   │        AWS EKS cluster       │
        truth)           │                              │
   ┌──────────┐  sync    │  ┌────────┐   ┌────────────┐ │
   │  ArgoCD  │◀─────────┼─▶│ Istio  │──▶│ 50+ micro- │ │
   └──────────┘          │  │ mesh   │   │ services   │ │
        ▲                │  └────────┘   └────────────┘ │
        │                │       │                       │
   Terraform (VPC, EKS,  │  ┌────────────────────────┐  │
   IAM, node groups)     │  │ Prometheus + Grafana   │  │
                         │  │ Loki + Tempo (o11y)    │  │
                         │  └────────────────────────┘  │
                         │  Karpenter (autoscaling)     │
                         │  Velero (backup/DR)          │
                         └────────────────────────────┘

See the architecture diagram at the top of this post for the full view.

The building blocks:

• Terraform for everything below the cluster: VPC, EKS, IAM, and node groups: so the foundation is version-controlled and reproducible.
• GitOps with ArgoCD so the cluster state is whatever is committed to Git. Deploys become pull requests.
• Istio service mesh for mTLS between services, traffic shaping, and canary releases.
• Full observability: Prometheus + Grafana for metrics, Loki for logs, Tempo for traces.
• Karpenter for fast, bin-packed node autoscaling.
• Velero for backup and disaster recovery.

Key decisions and the reasoning

1. GitOps over push-based CI deploys

Rather than have CI run kubectl apply, I made Git the single source of truth with ArgoCD reconciling the cluster to match. Why:

• Auditability: every change is a reviewed commit.
• Self-healing: if something drifts (a manual change, a crashed controller), ArgoCD pulls it back to the declared state.
• Easy rollback: revert the commit, the cluster follows.

2. Migrate service-by-service, not big-bang

We moved services off EC2 onto EKS one at a time, behind the mesh, with the ability to shift traffic gradually. Zero-downtime was a hard requirement, and incremental migration meant each step was small and reversible.

3. Cost control built in from day one

The 40% cost reduction didn't come from one change, it was several:

• Right-sized requests/limits based on actual usage (most pods were massively over-requested).
• Karpenter consolidating workloads onto fewer, better-packed nodes and scaling to zero where possible.
• Spot instances for stateless and fault-tolerant workloads.
• Removing idle resources: orphaned load balancers, unattached volumes, oversized node groups.

4. Observability before scale

I deployed the metrics/logs/traces stack early, so that as we migrated we could see exactly how each service behaved. SLOs and burn-rate alerting replaced the old "alert on everything" noise.

The hard parts

• Multi-tenancy and isolation. 50+ services from different teams sharing a cluster meant strict namespace isolation, resource quotas, and NetworkPolicies so one noisy service couldn't starve others.
• Istio's learning curve. The mesh delivered, but it's heavy and the config is intricate. Worth it at this scale; I'd think twice for a smaller estate.
• Migration discipline. The temptation to "just move everything" was constant. Holding to service-by-service migration is what kept it zero-downtime.

The outcomes

Metric	Before	After
Deployment frequency	Weekly	Daily (on demand)
Mean time to recovery	~2 hours	~15 minutes
Infrastructure cost	Baseline	−40%
Uptime	Inconsistent	99.9% SLA met
Provisioning a new service	Days (tickets)	Minutes (GitOps)

What I'd tell another team doing this

1. Start with observability and IaC, not the mesh. You need to see the system and reproduce it before you add complexity.
2. GitOps is the highest-leverage change: it fixes delivery and drift and auditability at once.
3. Cost optimisation is continuous, not a project. Build the habits (right-sizing reviews, budgets, showback) into how the team works.
4. Migrate incrementally. Boring, reversible steps beat heroic big-bang rewrites every time.

This is the kind of platform work I do, the open-source versions of these building blocks live in my projects, including a GitOps Kubernetes platform, an observability stack, and a Terraform AWS reference architecture.

---

Need this kind of platform built or rescued? That's exactly what I do, see my services or get in touch.