Side project
EoneShop: Microservices, Live
A mini shop built step by step from a single monolith into six services on Cloud Run: a public gateway, four business services that each own a Neon Postgres database and talk to each other only through Google Pub/Sub, and a live feed for the demo. I learned it by building it, and wrote up every step in public as I went.
How to use the demo
No sign-up, and nothing you do there is real: no money is charged and no email is sent.
- Open micro.eonelabs.my (on a laptop the form and the live view sit side by side; on a phone the page scrolls down to the live view for you).
- Pick a product and a quantity.
- Choose what should happen: a normal order, or one of the three ways to break it.
- Click Place order.
- Watch the diagram: every dot is a real message sent by one service, through Pub/Sub, to the services that listen for it. A service box lights up when it receives one, and turns red if it keeps failing.
- Read the timeline underneath: each line is one step, with the time it happened (measured on the server, counted from when the order was saved).
- Keep watching for a few seconds after the order finishes. Clean-up steps, like giving stock back, can still arrive after the order says CANCELLED. The button comes back when it’s quiet.
Normal order
Four messages, one after another, ending in COMPLETED.
Card declined
CANCELLED, then the cancellation fans out to three services and the stock is released.
Payment too slow
Nothing seems to happen, then the order times out. The late payment is never charged.
Catalog is down
Catalog turns red after five failed tries, then the order times out.
The first order after a quiet spell can take a few extra seconds: the services and their databases sleep when nobody is using them, and have to wake up first. Your scenario only ever affects your own order, never anyone else’s.
Break it on purpose
On the live page you place an order and watch each event travel through Pub/Sub in real time: order placed, stock reserved, payment charged, order confirmed. Four scenarios are on offer, and each one only ever affects your own order:
- Normal: the order goes through the full flow end to end.
- Card declined: the saga unwinds and gives the reserved stock back.
- Payment too slow: the order times out first, so the late charge is blocked when it finally arrives.
- Catalog is down: Pub/Sub retries the delivery, dead-letters it after 5 attempts, and the order times out.
What is a “saga”? (in plain words)
I’m still new to microservices, and “saga” was one of the words that confused me most at the start, so here is how I understand it today.
Think of booking a holiday: a flight, a hotel and a rental car, each from a different company. If the car booking fails at the end, nobody magically cancels everything for you. You have to cancel the hotel and the flight yourself, one by one. A saga is exactly that, written down as a rule: every step has a matching “undo” step, and when something fails, the steps that already happened are undone.
Why is it needed here? When a shop is one app with one database, a failed order can be rolled back in a single go. EoneShop is split into services that each keep their own data, so there is no single “undo everything” button. Each service has to clean up its own part.
Catalog reserves the stock
The item is set aside for you, so nobody else can buy the last one.
Catalog gives the stock back
If the order is cancelled later, the item goes back on the shelf.
Payment charges the card
The money is taken, but only after the stock is safely reserved.
Payment refunds the card
If the order is cancelled after paying (for example it was too slow), the money goes back.
So when a card is declined, the order is cancelled and one message goes out: “this order is cancelled”. Catalog hears it and gives the stock back. Payment hears it too, and would refund if it had already charged. On the live page you can watch this happen, step by step.
There are two ways to run a saga, and I picked the simpler one for a shop this size:
- Choreography (what EoneShop uses): like a relay race. Nobody is in charge; each service hears what just happened and knows its own next move.
- Orchestration: like a conductor. One service tells every other service what to do, and when to undo it. Easier to follow when there are many steps.
For the technical readers: a choreography-based saga with compensating transactions, a
PENDING status acting as a semantic lock, and tombstones for events that arrive
out of order. The details are in
part 4 of the series.
How it's built
- Outbox pattern plus idempotent consumers, so a retried event never double-applies.
- A saga with one cancel event that every service undoes its own step for, and tombstones so an event that arrives too late can't reserve stock or charge a cancelled order.
- One trace per order across both HTTP and events, via OpenTelemetry into Cloud Trace.
- Private services are only reachable with ID tokens, across six least-privilege service accounts.
- Cloud Build only redeploys the services whose code actually changed.
The six services
Each runs as its own Cloud Run service with its own service account. The four business services never call each other: they only publish and react to events through Pub/Sub, so any one of them can be slow or down without the others waiting on it.
gateway
The only public service. Serves the live page, forwards /api calls to the private services with an ID token, and rate-limits each visitor. Owns no data.
order
Takes orders (with an idempotency key), announces order.created, and alone decides how each order ends: completed, or cancelled after a failure or a timeout.
catalog
Products and stock. Reserves stock when an order arrives and gives it back when the order is cancelled. A tombstone stops a late event from reserving stock for a dead order.
payment
A fake card provider. Charges once the stock is reserved, never charges the same order twice, and refunds if the order is cancelled after it paid.
notification
“Emails” the customer when an order completes or is cancelled. Exactly one email per order, even when Pub/Sub delivers the same event twice.
feed
Listens to every topic and the dead-letter topic, and keeps recent events in memory so the live page can animate them. Nothing depends on it.
Around them: 10 Pub/Sub topics (one per event type, plus a dead-letter topic), four Neon databases, and Cloud Trace. Two more services exist only for running it on a laptop: a small Pub/Sub stand-in and a trace collector.
Seven parts, written up
Bugs it caught
- Windows resolved "localhost" over IPv6 first, adding about 2 seconds per hop, so orders looked stuck for no reason.
- Creating a fresh HTTP client per call cost about 130 ms each; reusing clients dropped a 2.6 second order down to 0.6 seconds.
- A duplicate event could trigger two emails; fixed by claiming the event before doing any work on it.
- Cloud Trace stayed empty because Cloud Run pauses the CPU right after a response is sent; fixed by flushing spans before the last byte goes out.
- Two overlapping timeout sweeps could cancel the same order twice; fixed with a compare-and-set instead of a plain read-then-write.