The Idea Started Much Smaller Than This

If you'd asked me in March what I was building, I could have drawn it on a sticky note.

Four boxes and three arrows:

crawl website
      ↓
give content to AI
      ↓
customer asks a question
      ↓
AI answers

That was the whole plan. And honestly, it didn't feel naive at the time. It felt like a pretty reasonable way to break the problem down.

Why it seemed doable

Every box looked like something I either already knew how to do or understood well enough to figure out.

Crawl the website. I work on this site. I know how the pages are built and where the product data lives. Pulling content off of it felt like familiar territory.

Give the content to the AI. This was the part RAG was supposed to handle. I understood the idea: store your own content somewhere the AI can search, pull back the pieces that matter for a question, and hand those to the model so it answers from real information instead of from memory. I'd read about it. I'd seen it explained. It made sense.

Customer asks a question. That's just a chat window. Remember the chat icon on our site that wasn't really a chat? This was the box that would replace it with something that actually talked back.

AI answers. The model is good at writing. Give it the right information and it would turn that into a clear, friendly answer.

Four steps. Each one made sense on its own. Put them together, and you'd have a website that could answer questions about its own products.

If anything, I figured giving the content to the AI would be the hardest part. Getting it to understand what someone was asking and find the right information to answer seemed like the biggest challenge.

What I thought a website agent was

I want to be clear about what I thought I was building, because it matters for where this series goes.

In March, my idea of a website agent was pretty simple: a chat bubble on the site that could answer product questions better than a contact form. That was the goal, and it felt like a good one.

I didn't think about conversations that go back and forth. I didn't think about what should happen when the answer wasn't on the website at all. I didn't think about how I'd know whether an answer was right, or what to do when it wasn't. None of that was on the sticky note.

What the sticky note didn't show

Here's the thing about a diagram like that. Every box looks like one step, but every box is hiding its own project inside it.

"Crawl website" sounds like one task until you look at how many different kinds of pages a real website has.

"Give content to AI" sounds like one task until you find out how many decisions sit between "here's some content" and "here's the right piece of content for this exact question."

"AI answers" sounds like the easy part, right up until it answers confidently and wrong.

I didn't know any of that yet. I'd find it out one box at a time, and almost every time, solving one problem uncovered another one underneath it. That pattern turned out to be the real story of this project:

Almost every simple AI idea has another problem hiding underneath it.

If you'd shown me in March what this would eventually turn into, with PDF manuals, project plans, human-curated knowledge, testing, an admin to manage it all, I probably would have laughed. Or I might have reconsidered. I'm glad nobody showed me.

Starting with the first box

Even so, the sticky note did its job. It gave me a place to start.

And the place to start was obvious: box one. Before the AI could answer anything, it needed something to answer from. That meant getting the content off the website and into a form I could actually work with.

It was also the box I felt most comfortable with. I knew how websites worked. I knew this website better than most.

How hard could it be?