← the light list
THE LIGHT LIST · NO. 002 Iso G 7s lit

The Great Un-monolithing

An event-driven microservices architecture built on Kubernetes. More buzzwords please.

ownership design to operations, solo
outcome 5 minute publishes to a few seconds
scope 29 microservices + infrastructure
established · 2024 tags · kubernetes · python · rabbitmq · varnish · docker
01 The starting point

The Post-Gazette's publishing stack had been accreting since roughly 2016. One enormous application that did everything and feared nothing, least of all my weekends. Deploys were [ how deploys worked, and how scary ], and every change carried the whole system's risk.

The constraints were plain. One engineer. Bare metal on premises, no cloud autoscaling to hide behind. And a paper that publishes every single day, so there was no maintenance window big enough to swap the stack in one move.

from the log Drew the boxes before the arrows. Always the boxes first.
02 The shape of it

Services stay small, own their data, and talk through RabbitMQ instead of calling each other directly. Varnish absorbs read traffic at the edge, so a couple million visits a month never becomes a couple million origin hits. MetalLB hands out real IPs, because bare metal does not come with a load balancer.

metallb

nginx ingress

varnish cache

rabbitmq · the event bus

python services · x29

mongodb

redis

The rule that made 29 services survivable for one keeper: every service looks the same. Same repo layout, same base image, same health checks, same way onto and off the bus. The novelty budget goes to the problem, not the plumbing.

03 Migrating while the paper ships

The monolith came apart one capability at a time. Peel a function off, stand it up as a service on the bus, route traffic to it, watch it for a while, then delete the old code path. Repeat 29 times. The first peel was [ the first service extracted, and why it went first ].

Order mattered more than speed. Low-risk, high-annoyance pieces went first, to prove the pattern. The scary ones went last, once the platform had earned trust.

timeline [ months to production, months refining ]
first peel [ service name ]
scariest peel [ service name, and why it waited ]
04 What went wrong
from the log The lamp swap happened at night. Obviously.

Events everywhere is worth it for the decoupling, and you pay for it in debugging: tracing a story across queues is harder than reading a stack trace. [ how you trace across the bus today ]

The mistake worth admitting: [ the thing that actually broke once, and what it taught you ]

05 What I'd change now
· Fewer, fatter services from day one. The honest number is probably [ your honest number ].
· [ the second thing you'd do differently ]
· [ the opinion only someone who ran this would have ]
06 Outcomes
29 services in production, uniform, one keeper
~2M visits/mo served through the platform
0 missed papers it published every day, the whole way through
old stack: dark decommissioned 2025 · see light no. 014