Human Intelligence: Sometimes the Website Doesn't Contain the Answer

Some of the most useful answers a company has were never written on its website.

Now that the admin let me see what the website agent knew, I kept noticing the questions it was answering the hard way, and the ones it couldn't answer at all.

"I'd like to speak with someone"

The first one that got me thinking was simple:

I'd like to speak with someone.

It's one of the most common things anyone types into a chat window on a website. And the way the website agent handled it was exactly the same as every other question. Turn it into an embedding. Search everything. Score the results. Hand the best pieces to the model. Let the model write an answer.

All of that, for a question with one right answer that never changes.

It worked, more or less. But it felt wrong. I was sending a question that a person could answer in one sentence through the most complicated part of the whole system. That's slower than it needs to be. It costs more than it needs to. And every time it goes through search and a model, there's a chance it comes back a little different, or a little wrong, on a question where it should never be either.

Why go to the model for an easy question like that?

The answer isn't always on the website

The second thing I kept noticing was the opposite problem.

Some questions the agent couldn't answer well because the answer just wasn't written down anywhere it could find. Not on a product page. Not in a manual. Not on a policy page. The answer lived in someone's head, or in how the company actually does things day to day.

No amount of better crawling fixes that. You can't crawl something that was never published.

Letting a person teach it

Both of those problems pointed in the same direction.

What if a person could just tell the website agent the answer?

Not by editing a web page and waiting for the next crawl. Not by digging into the database. By writing it down, in the admin, as knowledge the agent can use. I started calling it Human Intelligence: knowledge that comes from a person instead of from the website.

That idea had been rattling around in my head for a while before I built anything. Once the admin existed, it finally had a place to live.

Live as soon as it's saved

Then I had to decide how it would work, and one decision mattered more than the rest.

When someone saves a piece of Human Intelligence, it's live. There's no waiting for a crawl. There's no review queue. The very next question can use it.

That's exactly what you want when something needs fixing. If the agent is giving a bad answer, you want to fix it now, not tomorrow.

It also means there's no safety net. Whatever gets saved is what the agent will say. If it's wrong, the agent is wrong, and it'll be wrong confidently, because now it has evidence.

That made me think much harder about something I'd been taking for granted. With the website, I could always point to where an answer came from. With Human Intelligence, the source is a person. So how do you keep that knowledge controlled, trustworthy and easy to check later?

It's still evidence

The answer I landed on was to treat Human Intelligence like everything else the agent knows.

It's evidence. It goes through the same rules. The agent answers from it the same way it answers from a product page or a manual: grounded in what it was given, and nothing more. And it stays visible in the admin, so anyone can see what was taught, and change it or turn it off.

The difference is who's responsible for it. When the website is wrong, that's a website problem. When Human Intelligence is wrong, it's because someone saved something wrong. Whoever saves it owns it.

Teaching an AI something is easy. Making sure it's still true next month is the real work.

The next problem

Human Intelligence gave me a way to fill gaps. But I was still finding those gaps by accident, whenever I happened to ask the right question and notice the answer was bad.

Then I remembered something from back in May. Every time the agent said "I don't have enough information," it was pointing at a real question someone wanted answered.

It had been telling me where the gaps were all along.