Building a System That Can Learn From Its Mistakes

Back in May, I noticed that every "I don't know" was a real question someone wanted answered. In July, I finally did something about it.

When I taught the website agent to say "I don't have enough information to answer that confidently," it felt like a win, and it was. An honest "I don't know" is a lot better than a confident wrong answer.

But I'd also written something down at the time and then mostly left it alone. Every one of those answers was a question someone actually asked. Someone wanted to know something, and the agent couldn't help them.

Those moments were just disappearing. The agent said "I don't know," the conversation moved on, and nobody ever looked at the question again.

Knowledge Gaps

So I started capturing them.

Whenever the agent couldn't answer a question well, that moment got saved as a Knowledge Gap. The question that was asked. What the agent found when it went looking. Why it decided the evidence wasn't good enough.

And because the admin existed now, those gaps had somewhere to show up. Instead of finding problems by accident, I could open the admin and see a list of real questions the agent had struggled with.

That list changed how I thought about the whole project.

The loop

Once gaps were being captured, something clicked. It wasn't a list of failures. It was the first half of a loop:

Customer asks something
        ↓
Agent can't answer well
        ↓
Knowledge Gap
        ↓
A person reviews it
        ↓
Knowledge improves
        ↓
Next time, the agent can answer

The agent didn't have to know everything. It just had to be honest about what it didn't know, and leave a trail I could follow.

Not every gap is the same

Looking through the gaps, they weren't all the same kind of problem.

Some were missing content. The answer was on the website, but the agent didn't have it. A page that was never crawled. Something buried in a manual that didn't come through cleanly. Those meant fixing what the agent was collecting.

Some were answers that were never on the website at all. Those were exactly what Human Intelligence was for. A person writes down the answer, saves it, and the next customer who asks gets it.

And some were questions the agent probably shouldn't answer in the first place. In those cases, "I don't know" was the right answer, and the gap was really just confirming it.

Each one needed a person to look at it and decide. That's the "human reviews it" step, and it's the part I don't think you can skip.

Don't let it break again

There was one more piece that made the loop feel real.

Fixing a gap once is good. But I'd already learned, back in June, that a problem you fixed can quietly come back later when you change something that seems unrelated. So once a gap was fixed, the question that caused it could become a test. If that question ever stopped getting a good answer, I'd know.

That idea would grow into something much bigger later on. But this is where it started: a real question from a real conversation, turned into something the system would be checked against from then on.

A system that admits what it doesn't know can be taught. A system that bluffs can't.

From a chat window to a learning system

Looking back, July is where the project changed character.

In March, it was a crawler. In April, it was a pile of data turned into something searchable. In May, it started answering. In June, it learned to follow a conversation.

In July, it started learning from its mistakes. Not on its own, and not magically. It needed people to review and teach it. But for the first time, every bad moment had somewhere to go and a way to make the next answer better.

The project stopped being only a chat interface and started becoming a learning system.

The next problem

Knowledge Gaps only caught one kind of failure, though. They caught the moments the agent knew it didn't know.

What about the answers it gave confidently that were still wrong? Or the conversations that went fine, but could have gone better? I wasn't seeing any of those, because I wasn't keeping them.

I needed to start saving the conversations.