Every Bad Answer Became a Test

Every bad answer I found was something I'd fixed once and hoped would stay fixed. Hope wasn't going to be enough.

Saving conversations had done exactly what I wanted. I was finding bad answers I never would have found on my own. And I was fixing them.

The problem was what happened after.

Fix and hope

For months, my process looked like this:

bad answer
→ manually fix it
→ hope it stays fixed

That works right up until you change something else. A tweak to how content gets chunked. A new kind of document. A different threshold. A new model. Any one of those can quietly bring back a problem you fixed weeks ago, and you won't know until someone asks that question again and gets the bad answer again.

I'd already seen this happen. The follow-up questions from June. The plans that got mixed together. Every one of those was a fix that could come undone without anyone noticing.

Fix and prove

So I changed the rule. Every bad answer becomes a permanent test.

bad answer
→ understand the cause
→ write a test for it
→ improve the system
→ every future change has to keep passing it

The test is the important part. It's not enough that the answer is right today. The question goes into a suite of tests, with what a good answer needs to include and what it must never say. From then on, any change to the system has to pass every one of them before I can call it done.

A bad answer stops being something I remember to check. It becomes something the system has to prove it won't do again.

The first 47

On August 7, I built the first version of the test suite: 47 test cases across 11 categories.

Some of them were the obvious ones. Product questions. Follow-up questions that had to stay on the right product. Project plans that had to keep their steps in order and stay on the right plan. Questions where the right answer was "I don't have enough information."

Some were less obvious. Questions designed to see whether the agent could be talked into something it shouldn't do, like offering a free product or revealing its own instructions.

Each test is checked against fixed rules: what a good answer must include, and what it must never say. I also added an optional AI judge that can look at an answer and give its opinion. But it can never overrule the rules. If the rules say an answer failed, it failed, no matter how good the judge thinks it sounds. I didn't want an AI deciding whether my AI was right.

It kept growing

The suite didn't stay at 47 for long.

Two days later, it was 58. The day after that, 72. By August 14, 75. As I write this, it's 81.

Every one of those additions is a real question that once got a bad answer, or a new kind of question I wanted to make sure the agent handled correctly before anyone asked it for real. The number going up isn't the point. The point is that each one represents something the agent used to get wrong and isn't allowed to get wrong again.

Not trusting "it looked right"

The biggest change wasn't the tests themselves. It was how I thought about whether something worked.

Before, "I tried it and it looked right" was good enough. After, it wasn't. Looking right once, on the question I happened to ask, in the way I happened to ask it, isn't evidence. A test that runs every time, on the same questions, checked the same way, is.

If a bad answer can come back, it will. Make the system prove it can't.

This is probably the single most important habit the whole project taught me. It's not really an AI lesson. It's just good engineering. But AI made it feel urgent, because AI fails so convincingly.

The next problem

The tests caught the bad answers I knew about. Saving conversations helped me find more.

But I was still the one doing the finding. I was reading through conversations, deciding which answers were bad, and turning them into tests.

The people actually using the website agent knew whether an answer helped them. I just didn't have a way for them to tell me.