Six things I’ve done to one system

Moved the services onto Kubernetes

The services ran on virtual machines under configuration management. The company was moving to Kubernetes; I owned that move for the messaging services — the infrastructure definitions, the deployment pipelines, and the verification built into them rather than bolted on afterwards.

Deploys went from hours to under three minutes. Rollbacks to under two. Resource utilisation improved substantially at the same time, because a migration is the moment you get to size things honestly instead of inheriting whatever was set years ago.

I also built production clusters from nothing on three different clouds, including one region taken end to end — infrastructure, five services, active-active enablement, edge migration. Earlier builds sat on segregated accounts to meet government data-residency requirements, with restricted access paths and pipelines of their own.

When you build a region from an empty account you find out exactly which of your assumptions were actually just defaults.

Cut a seven-hour release to fifty-two minutes

Verification used to be manual. An engineer triggered jobs in sequence and read each result by hand, and a full multi-cluster release took about seven hours. It now takes fifty-two minutes.

The number isn't the interesting part. The path there is.

It started as a checklist. Then shell and Ansible scripts. Then CI jobs. Then gates native to the deployment platform itself. That progression took years and every step was deliberate — you don't get to skip from a checklist to a gate, because until the check is right, automating it only produces a wrong answer faster.

The obstacle was that the internal workflow engine couldn't run shell-based checks against pods. The usual answer is to work around it: run the checks somewhere else, glue the results back in, accept the seam. I went into the pipeline runtime underneath and extended it until it could.

What the check reads: process health, logs, metrics, ports, and the release version actually running. Then it parses that output and decides — fail on what matters, flag what needs a look, pass when it's clean. That distinction matters more than it sounds. A check that tells you something is wrong sends someone off to find out what. A check that tells you what is wrong ends the question.

It runs against every pod on every release now, instead of one sample per cluster. That change on its own surfaced edge cases the sampling had been quietly missing.

The parallelism came next, built alongside a colleague. Both regions carry live traffic, so there was no reason to deploy them in sequence — and the useful consequence is that rollbacks run in parallel too. Recovery got faster as a side effect of making deploys faster.

Other teams picked it up, which was the intent rather than a surprise.

I'd rather build the thing the next service can pick up than the thing that only solves my problem.

Replaced the public edge while it was serving

The edge is where every request arrives. There is no maintenance window for it and no staging environment that resembles it, so the work has to be provably safe before it touches anything.

I led the replatform across the production fleet on two clouds. Phased traffic cutovers, with automated verification and failover validation at each stage. No customer impact. Edge infrastructure cost on the largest cluster came down by half.

Cutovers run through a Python CLI I wrote. The DNS zone holds more than a hundred thousand records, which is well past the point where anyone can hold the resolution chain in their head. The tool traces that chain with live health-check data overlaid, changes weights in bulk, and writes an audit line per change. It became the primary tooling for cutovers and was packaged and distributed to DevOps teams elsewhere in the company.

Every change goes through preview, then confirm, then apply. The preview step isn't there to catch typos — a validator catches typos. It is there so an engineer can look at what is about to happen and change their mind. Those are different problems, and only one of them is solved by better validation.

Alongside it, the cache layer moved from Redis to Valkey. I ran that as a documented, repeatable process rather than a one-off, and it became the team standard.

Then the old edge stack came down in stages, with DNS, certificates and monitoring cleaned up behind it.

Nobody notices decommissioning done properly, which is the point.

Made losing a region uneventful

Active-active was a fleet-wide programme. I ran the rollout for the chat backend and edge services, across fourteen of the fifteen production clusters.

Both regions live, traffic weighted between them by DNS, failover wired into the traffic-orchestration console. Recovery time is near zero because there is nothing to recover to — the other side is already carrying its share.

Very little of that was architecture. It was VPC peering, route tables, cluster DNS resolution conflicts, load balancer target groups and scheduler placement — the layer where nothing is conceptually hard and everything is specifically annoying. Those were my problems, and solving them is what made the rollout possible. Monitoring, certificate automation and secret handling all had to exist twice without doubling what it costs to operate.

Certificates on a global edge are unforgiving; one mistake takes chat down worldwide. So rotations are validated on a node running the full production configuration that takes no live traffic, verified through a direct path, and only then admitted.

Prove it somewhere nothing can be hurt, then let the traffic in.

Built the view of what's happening

The company moved off Grafana onto an internal metrics platform. I took my services across and built out what they needed — hundreds of dashboards covering infrastructure, application, API and load balancer metrics.

Alerting is sorted into severity tiers with defined escalation paths, and every alert links to a runbook. Service level indicators and objectives were defined with the development team rather than handed to them, and they get revisited after incidents.

Coverage grew as the environment did: load balancer traffic monitoring once workloads moved to Kubernetes, process-level health checks, log-based alerting. But the stated goal was always fewer alerts, not more, so I pruned continuously.

I am often the person who reads the logs on behalf of developers without direct access. That has made me unusually fluent in how these services actually fail, which is a different kind of knowledge from how they are supposed to work.

I have been on call since my third month.

An alert that fires often enough to ignore is worse than no alert.

Took 29 to 77 percent off what it costs to run

Instance right-sizing, away from oversized general-purpose families toward mixed, appropriately sized ones. Storage-mode and capacity-mode tuning on the document store. Cache sizing. Replacing managed load balancers with proxy-based routing where that made sense. Kubernetes requests and limits set against observed usage rather than defaults.

Per-cluster reductions between 29 and 77 percent. Six-figure annual savings. Edge cost on the largest cluster halved.

None of it was possible before the observability work. You cannot right-size anything until you can see what it actually uses — before that you are only guessing with more confidence.

The first wave of savings is easy: everything is oversized, you resize it, the number moves. The discipline afterwards is the hard part, so I wrote the roadmap that keeps it from drifting back.