At Some Point, You Need More Than a Chat Window

For four months, the website agent's whole control panel was a chat window, a terminal and a lot of SQL.

That's how it works when you're building something new. You type a question into the chat. You read the answer. If something looks off, you go into the terminal, check the logs, open up the database and start writing queries until you figure out what happened.

It worked. For a while, it worked really well. I knew every piece of the system because I'd built every piece of it, and I could find almost anything if I dug long enough.

The pile kept growing

The problem was how much there was to dig through.

Back in March, it was products. Then support and policy pages. Then manuals, with all their messy text. Then project plans, broken down step by step. Each one was its own kind of content, with its own quirks, its own way of being chunked and its own ways of going wrong.

And the questions I was asking about the system kept getting harder:

• What does the agent actually know right now?
• Where did this piece of information come from?
• Is this manual still being used, or is it an old version?
• What chunks came out of this page?
• Why did it give that answer?

Every one of those questions had an answer. But getting to it meant writing another query, or scrolling back through terminal output, or opening three tables and lining them up in my head.

It wasn't one specific problem that pushed me over the edge. It was the amount of time I was spending just trying to figure out what was going on. Every time I wanted to check something, I was back in the database or digging through logs. I was spending more time looking for information than actually working on the agent. I needed a way to open something up, take a quick look, and see what was happening without turning it into another investigation.

I was the only way in

That's when it really sank in.

The only way to understand what the website agent knew was to ask me. And the only way I could answer was by digging. If I wasn't sitting at the keyboard with a terminal open, the system was basically a black box.

That's fine for an experiment. It's not fine for something a company is going to depend on. Someone should be able to look at the agent and see what it knows, where that knowledge came from, and why it answered the way it did, without needing to know how to write SQL.

Honestly, I wanted that too. I was tired of writing the same queries over and over.

When the only way to understand your system is to dig through it, you've outgrown your tools.

A system you can query vs. a system you can see

I'd spent my career on both sides of this. I've built plenty of systems that were perfectly fine as long as a developer was around to look things up. And I've seen what happens when the people who depend on those systems have no way to see inside them. Every question becomes a ticket. Every ticket goes to the same person.

I didn't want to build that again.

Being able to query a system and being able to see it are two very different things. The website agent was very queryable. It wasn't visible at all.

The next problem

So I started building an admin.

Not a dashboard to make the project look impressive. A place to answer the questions I kept asking, without having to open a terminal to do it.

It turned out to be one of the most important things I built all year.