How We Load-Tested the Matching Engine to 110,000 TPS

The first problem I found in that load test wasn't in the system under test. It was in my own tooling.

This story starts with what looked like a "system bug."

10 Users, 110,000 TPS

The business requirement: 10 users placing orders at the same time, with the order placement endpoint sustaining 110,000 TPS.

Ten users doesn't sound like much. But each user is placing orders at high frequency and canceling them constantly, which means the system has to process more than 100,000 orders every second.

Our approach was staged load testing: the same 10 concurrent users, running for 1 hour, then 2, then 4, then 8, then 24 hours — gradually extending the duration.

Because what we had to verify wasn't an instantaneous peak. It was whether the system could run stably over the long term. An exchange's matching runs 7×24 without ever stopping. Surviving one hour doesn't count; surviving a full day without a single error counts.

Why We Dropped JMeter: Manual Triggering Is Fake

The hardest part of load testing isn't "sending requests." It's "simulating real users."

For ordinary scenarios — placing orders, canceling orders — JMeter is good enough. But one scenario is special: forced liquidation.

Liquidation isn't something you can trigger on demand — a user only gets liquidated when the price drops to the liquidation price line. In the real world, market prices move in real time, and suddenly a large batch of users can be triggered at once. To test this scenario properly, the load test script has to "follow the market": the moment the price hits the threshold, it fires off liquidation requests at high concurrency.

JMeter can't do this. Its parameterization mechanism struggles to express logic like "dynamically trigger based on real-time market data." Forcing it means writing very complex preprocessors — and the result is still fake.

Manual triggering is even worse: it can't reproduce a real high-concurrency impact, and it can't cover the burst traffic that market fluctuations bring.

Rewriting the Load Test Script in Go

Eventually we found Locust — a Python-based load testing tool where the script is just code. Every virtual user can independently listen to the market and independently trigger liquidations, which comes close to real user behavior.

But it has a fatal flaw: Python's GIL. Single-machine concurrency hits a ceiling, and you can't push the load high enough.

Digging deeper, we found that Locust has a solution: rewrite the Worker in Go (boomer). You keep the flexibility of "script is code" while breaking through the GIL concurrency bottleneck.

In my spare time, I wrote a Go version of a "market-data-driven liquidation load test client" as a technical proof of concept: each virtual user independently listens to the simulated market feed, automatically triggers liquidation when the price hits the threshold, and I implemented the market data communication and local caching myself to keep it stable under high concurrency. Once it ran end to end, single-machine TPS improved more than 10x over what we expected.

For the first delivery round, we shipped the JMeter version on time (a team decision, to protect the schedule). We documented the research findings and the technical validation and shared them with the team; later rounds switched to the Go approach.

The "Bug" the Load Test Found Was in the Tooling

With the new tooling in place, we started the official run. Midway through, an anomaly showed up: the fill rate for conditional orders was noticeably low.

My first reaction was that the system had a bug. Conditional orders are a core feature — an abnormal fill rate means users' money could be mishandled. That's a big deal.

We traced the whole chain for a long time and finally pinned it down: it wasn't the system under test. It was the load test script itself.

Conditional orders are triggered by market prices, and the market data source in the script was distorted — the price the script thought it saw didn't match the actual market price, so the conditional orders naturally never triggered.

That's when it hit me: if your load testing tool itself can't be trusted, every number it produces is fake. You think you're testing the system; in reality, you're testing the tool's delusions.

The fix: we built our own market data cache service. The load test scripts subscribe to the real market data stream, and if the feed drops, they fall back to a default price — the scripts neither crash nor hardcode anything.

After the fix, the fill rate returned to normal.

The Real Bottleneck: A Single Cluster Can't Carry It

Only once the tooling was trustworthy did the system's real problem surface: the maximum concurrency of a single service cluster was barely above 10,000 — a long way from 110,000.

This wasn't a script problem. It was an architecture problem: a single cluster simply can't carry this scale.

I compiled the data into a report and sent it up. The tech lead made the call to scale out: the order, settlement, and equity services were expanded to dozens of instances, the matching engine and risk control were doubled, and the entry gateway got traffic distribution.

After scaling out, we ran the test again: average response time of 1–2 seconds, 24 hours of stable running, and measured TPS above 110,000. Target met.

Final Thoughts

This load test taught me two things:

First, simulating reality matters more than producing numbers. Manual triggering and hardcoded prices look like "load testing," but they're really self-deception.

Second, before you stress a real system, first stress your own tooling until it's trustworthy. If the tool lies to you, the numbers are all wrong, and every conclusion you build on them is sand.

Those two lessons have run through my entire testing career ever since.

Next time, I'll write about how I wrote a script myself and uncovered a system-level bug where longs and shorts were liquidated at the same time. After that: a bug that almost made two developers start fighting, and what the test team at a 200–300 person exchange actually looks like.