July 29, 2026 · Nate, building ModDog

Good models come in small packages

TL;DR: We train our own moderation model for ModDog. When it kept missing things, we assumed it was too small and moved up to a bigger one. The bigger model failed in exactly the same ways. The real problem was our training data, and once we fixed that, the big model's only lasting job was teaching everything it knew back to a small one. The small model now runs in production, at more than twice the speed.

The upgrade that didn't work

ModDog is an AI moderation bot for Discord. At its core is a judge model: a small language model that reads a message and decides whether it breaks a server's rules. Ours kept missing things it should have caught, especially in a couple of categories, while at the same time it was well behaved about not flagging innocent chat. Our fix list said what everyone's fix list says: the model is too small. Get a bigger one.

So we took our exact training data and recipe and trained a model twice the size.

It failed the same way, in the same categories, with the same kinds of misses. A model with twice the capacity, handed the same data, had simply learned the same blind spots. That result stung, but it was the most useful negative result of the whole project, because it settled the question: the bottleneck was our data, not our model.

Why we train our own model at all

Worth pausing on why we were training anything in the first place, because the answer shapes everything after it.

Off-the-shelf models moderate like a helicopter parent. Real Discord chat is sarcasm, gamer trash talk, dark jokes between friends, and in-jokes that look alarming out of context. A general-purpose model flags all of it, and a bot that punishes regulars for banter gets kicked out of the server before the trial week is over. We needed a model with judgment tuned to how people actually talk, and the only way to get that was to train it ourselves.

There was a second reason: privacy. ModDog reads messages, makes a call, and discards them immediately. We don't keep chat logs, so we could never train on them even if we wanted to. And Discord's developer policy bars training on user messages anyway. Whatever the model was going to learn, it had to learn without ever seeing a real user's message.

The data story

This constraint sounds crippling, but it turned out to be the door to the highest value educational point of the entire project.

Since we couldn't use real chat, we built our training corpus from scratch: openly licensed public datasets, plus a lot of generated conversation written in the register of Discord, the short messages, the slang, the affectionate insults between friends. Teaching the model what not to flag took as much corpus as teaching it what to catch.

The lesson that kept repeating: where your data comes from matters more than how much of it you have. When the model was weak in a category, adding more data from the wrong source barely moved it. Adding data from a source that matched the deployment domain fixed the category, sometimes completely. We watched one stubborn category go from broken, to partially fixed with off-domain data, to perfect with matched data, and each attempt used the same model, the same recipe, and roughly the same amount of data. The only thing we changed was where it came from.

Once the corpus had proper coverage, the bigger model finally earned its keep. With the same improved data, it beat our small model across the board: caught more, wrongly flagged less. For a while, it ran production.

Coming back down

Here's the part that made the whole detour worth it. A bigger model costs you serving speed forever. On our self-hosted hardware, that's the difference between comfortable capacity and buying more GPUs.

So we used the big model as a teacher. It labeled a large synthetic corpus, tens of thousands of messages shaped like the traffic we see, and we trained the small model on its answers. Where we knew the teacher had blind spots, we corrected the labels by hand so the student wouldn't inherit them.

The student closed most of the gap. On some categories it beats its teacher. And it runs more than twice as fast, which on fixed hardware means more than twice the capacity, for free.

One thing kept this honest the whole way: a regression gate. Every candidate model has to match or beat the current one on a fixed benchmark before it ships, including all the old wins. Without that, every improvement quietly breaks something you fixed three rounds ago, and you find out from an angry server owner instead of a failed eval.

What training our own model bought us

The honest answer is not a better score on a leaderboard, it's a model that fits the job from every side.

A model tuned to how our users actually talk, instead of a general model's nervous idea of politeness. A model that runs well on hardware we own. A model whose mistakes we can study and fix on our schedule, with a retrain that finishes overnight. And a model that got good without ever training on a single user's message, so the privacy promise isn't a policy we have to maintain, it's just how the model was built.

If you're thinking about doing this