From 20 messages a second to nearly 1,000: the prototype fleet that proved ModDog
TL;DR: Before ModDog was a product, it was a question: can a tiny self-funded team serve AI moderation cheaply enough to sell it? This is the story of the prototype fleet that answered it, three mismatched boxes, a used $500 NVIDIA card, two AMD workstation cards, an eGPU dock, and one banished Intel card, going from 20 messages a second to 983. The throughput didn't come from hardware. It came from learning, repeatedly and sometimes embarrassingly, that the fastest inference is the inference you don't run.
ModDog is an AI Discord moderation bot: it reads a server's rules channel and enforces it, flagging what matters to human mods. The economics only work if judging a message costs close to nothing, because Discord messages are endless and moderation revenue is not. Which means the whole product lives or dies on one number: messages per second per dollar of hardware.
When I started, that number was 20 to 50 on a single card, and I did not know what a KV cache was. Today the fleet benches 983.8 messages per second, sustained, measured end to end through the real pipeline. This is the story of the 20x, told honestly, including the parts where the answer was luck, tenacity, and more vibe coding than I'd ever want to admit.
One thing to hold onto while you read: this is the prototype, the rig I cobbled together to learn the domain and find out whether ModDog was viable before asking anyone for money. It did its job. What the paid service runs on is a different, harder question that this fleet earned the right to answer, and I cover it at the end.
The prototype fleet, in its full glory
- A used RTX 3080 Ti I paid about $500 for. It runs the main 3B safety judge and does 612 of the 983 all by itself. The best hardware money I've spent, by an order of magnitude. It also hosts the bot itself, because every box works two jobs here.
- Two AMD Radeon AI PRO R9700s. One serves a second copy of the safety judge plus a 12B specialist model. The other one is my personal card that the fleet is contractually forbidden from touching.
- An RTX 4080 hanging off an OcuLink eGPU dock, running the topic model. Yes, an eGPU in production. It once died with a cryptic Xid error that I "fixed" by power-cycling the dock, which is about as scrappy as it sounds.
- An Intel Arc Pro B70, currently sitting on a shelf. The tragicomedy of this card is that it was destined to be the fastest in the fleet: 633 messages per second solo, which beats every other card I own. Then Intel rolled a nat 1, tripped on their own shoelace so hard they snapped their neck, and shipped firmware that crashes the card under exactly the sustained load it was born to serve. That story has its own post. RIP.
Three boxes, mixed vendors, mixed drivers, one LAN. Nothing about this was planned. Every piece was bought used or on sale, and the architecture grew around whatever the hardware turned out to be good at.
Where the 20x actually came from
If you're expecting a story about CUDA kernels, I have to disappoint you. The gains, in rough order of size:
Don't run the model. The single biggest multiplier. Most Discord messages are "lol" and "gm". They don't need a neural network; they need a regex. The pipeline is tiered: a deterministic rules layer handles the obvious stuff for free, a 3B safety judge reads what survives, a small topic model runs only where server owners turned topics on, and a 12B specialist wakes up only for rare jobs like appeals. The expensive models see a tiny fraction of traffic. Every tier I added moved more work to something cheaper.
A 3B model, aggressively trained, beats a 7B model politely prompted. The safety judge retrains in a nightly loop on a synthetic, Discord-shaped corpus, steered by error analysis from runs on my own test servers. (Deliberately not on user messages: ModDog discards message content immediately by design, and Discord's developer policy bars training on it anyway.) A small model that has seen your exact problem shape outperforms a bigger generic one, and serves at three times the speed. When I A/B'd a 3B distill against the 7B it replaced, the gap was data, not parameters.
vLLM everywhere, then fight for the config. Moving off my first llama.cpp setup onto vLLM was a step change on its own. But the bigger lesson was that serving config is worth real benchmark time: continuous batching depth, concurrency caps, prefix caching (92 percent hit rate on moderation prompts, since every message shares the same system prompt), and n-gram speculative decoding, which bought about 20 percent on the AMD cards for free.
Quantization is per-vendor, and nobody tells you this. int8 made the NVIDIA card 33 percent faster. The same quantization made the AMD card 41 percent slower. FP8 straight up doesn't work on my AMD generation. AWQ was a dead end on both. The fleet ended up deliberately heterogeneous: int8 on NVIDIA, fp16 with speculative decoding on AMD, one approved 4-bit format for the big models. If you run mixed vendors, bench every format on every card and believe nothing.
Pull, don't push. The dispatcher started life pushing work at cards and drowning the slow ones. Rewriting it so idle cards pull work when they have a free slot fixed tail latency across the whole fleet and made mismatched cards composable. This one change is why a 612 msg/s card and a 371 msg/s card can share a queue without either babysitting the other.
Falsify before you ship. The culture that saved me the most money: every "this should be faster" idea gets a scripted bench before it touches production, and the ledger only records whole-system numbers, never a single card's heroic solo run. A Mixture-of-Experts model that looked great on paper got falsified in an afternoon. A throttle that was supposed to save the Intel card got falsified in a week. The bench scripts are the real senior engineer here.
The vibe coding confession
Now the part I'd rather skip. I built almost all of this with an AI pair programmer. I came in not knowing inference serving, Discord's gateway internals, GPU driver debugging, or half the Linux I now use daily. The honest description of my process is: describe the problem, let the AI draft, argue with it, bench the result, keep what survives.
Some context on who "I" is, because it changes how you should weigh that confession. I'm not a software developer by trade. I'm an ops engineer who has spent close to a decade in security: building and deploying OS images, patching fleets, writing the reports nobody reads, and above all pulling data out of disparate systems and smashing it together well enough that my bosses could make decisions on data instead of vibes. That career left me with solid scripting, light-duty coding, and genuine depth in two things: infrastructure and data systems.
If you're thinking "so this guy isn't a developer at all, what the hell," yeah, me too, some mornings. But look at what this product actually is: a hardened deployment, an inference fleet, an encrypted data pipeline, and a security posture. That's my home turf, and it's the part AI can't vibe for you. And I'm not doing this alone: my business partner brings more than 20 years of IT experience, so between us most of this project's unknowns had at least one person in the room who'd seen the failure mode before. The application code is where we lean on the pair programmer, and never on trust: nothing ships without surviving the test suite, the benches, and the gauntlet. We do the parts we've spent our careers getting good at, and we put a harness around the parts we haven't.
Here's what I learned about making that actually work, because "vibe coding" fails exactly where you'd expect: vibes don't know your hardware. The system that kept it honest is boring and non-negotiable. One canonical capacity benchmark that everything gets measured by. A test suite (3,500 tests now) that runs before anything merges. A dual-judge gauntlet that replays real moderation scenarios through the actual pipeline, no mocks, before any model swap goes live. The AI writes a lot of my code; the benches decide what stays. Every number in this post comes from a script, not from either of us feeling good about a change.
Tenacity did the rest. The B70 postmortem alone represents two weeks of falsification ladders. The nightly retrain loop exists because week one of live testing flagged half my test server for saying "kys" in a gaming context. None of this was talent. It was refusing to stop, with a very patient robot.
Where the prototype landed
983.8 messages per second sustained, which rates the fleet at roughly 17,000 average Discord servers smoothed, or half that with peak headroom. Total GPU spend: less than a single MI300X module. The question the prototype existed to answer, "can a tiny self-funded team serve AI moderation cheaply enough to sell it," came back a clear yes, with a 20x margin of proof. The binding constraint stopped being GPUs a long time ago; it's now the only constraint that matters for any product: finding customers.
What subscribers actually run on
Here's the part I don't want anyone to misread. If you're a server owner considering ModDog, the takeaway should not be "my moderation depends on an eGPU dock and vibes." The scrappy part of this story is the learning. What came out the other side is a service built, from the start, to assume hardware fails:
- No single model instance is load-bearing. Every judging tier runs on multiple cards across multiple machines, behind a dispatcher where healthy cards pull work the moment a sick one stops. This isn't theoretical: during the Intel investigation the fleet absorbed dozens of real GPU crashes, live, and moderation kept flowing while the failed card recovered. The B70 saga was, accidentally, months of chaos testing.
- Messages survive failures. A durable queue means a message being judged when something dies gets judged anyway, by whatever's healthy, not dropped.
- Boxes are disposable. Watchdogs restart wedged services, whole machines survive reboots unattended, and leader election with hard fencing means the Discord connection itself can move to a standby without a human awake.
- The data outlives the hardware. Encrypted daily offsite backups with a tested one-command restore, keys included. A separate-site disaster recovery expansion is designed and on the roadmap for launch scale.
The prototype taught me where everything breaks. The product is what I built with that knowledge.
If you're a Discord server owner and want the thing all this serves, that's ModDog. If you're someone with a pile of mismatched GPUs and a product idea: the water's fine, bring benchmarks.