My First Crawl: 526 URLs and a Very Small Victory

The sitemap said 526.

That was the first real number in this whole project, and I remember liking it. It felt manageable. A website is a big, messy thing when you're looking at it page by page, but a sitemap turns it into a list. A list felt like something I could work through.

Then I started looking at what those 526 URLs actually were, and I decided the smart move was to crawl ten.

Starting with the map

If the first box on the sticky note was "crawl website," the sitemap was the obvious place to start. It's the website telling you what it thinks it contains. No guessing, no clicking around, just a list of addresses.

Not all of those addresses were the same kind of thing, though. About 249 of them looked like product pages. The rest were something else.

There were category pages, content pages, support and policy pages, and even pages tucked away in parts of the site you wouldn't normally stumble across. A lot of different kinds of information, all sitting there in the sitemap.

At the time, I didn't think much about the rest. I was building something that understood products, so I went after the products. That decision comes back in the next post.

Why only ten

I could have pointed the crawler at all 249 product pages and let it run. I'm glad I didn't.

The goal of the first crawl wasn't to collect everything. It was to find out what "collecting" even meant on this website. With ten products, I could open every result and compare it against the real page by eye. I could see exactly what came back, what was missing, and what looked wrong. With 249, I'd have been counting results instead of reading them.

So I picked ten:

PRD-101  PRD-102  PRD-103  PRD-104  PRD-105
PRD-106  PRD-107  PRD-108  PRD-109  PRD-110

(Those are placeholders standing in for the real product codes. The specific products don't matter much here.)

I picked a mix of best sellers, a few newer products, and some completely random ones. I'm a developer, after all. There's always room for a little randomness.

There was one product I kept checking first. If the crawler couldn't get that one right, it wasn't going to get anything right.

What the crawler pulled back

The crawler was written in Python, and its job was simple to describe: visit each product page and pull out the parts that matter.

For each product, that meant:

• the product description
• product details
• specifications
• key features
• manuals

That list is the difference between a web page and something a website agent can actually use. A web page is built for a person to look at. It has navigation, images, buttons, reviews, related products, and a lot of layout. What I wanted was just the knowledge, pulled out and labeled.

Something closer to this:

{
  "product": "PRD-101",
  "name": "Example Product",
  "description": "...",
  "details": ["..."],
  "specifications": { "...": "..." },
  "key_features": ["..."],
  "manuals": ["..."]
}

Some of it worked beautifully

When the crawl finished, I started opening results.

Some of them were exactly what I'd hoped for. The description was there, the details were there, the specs were clean, the features were listed, and the manual was linked. I could read the record and recognize the product page it came from.

Others were only partly there.

One of the newer products was giving me trouble. Something about the way its page was built was just different enough that the crawler wasn't picking up all the information. It looked perfectly normal on the website, but the data coming back told a different story.

That surprised me more than it should have. These were ten products from the same website, built by the same team, and I'd assumed their pages would look roughly the same underneath. Mostly they did. But "mostly" turned out to be the important word. A field in a slightly different place, a section that only some products had, a manual that existed for one product and not another. Fetching a page was easy. Pulling the same clean pieces out of every page was the real work.

I didn't solve that in March. I noticed it, and I let it bother me a little.

A very small victory

Still, I want to be honest about how this felt, because it was a good day.

For the first time, real product information wasn't just sitting on a web page. It was structured. Labeled. Sitting in a file in a shape I could work with. It wasn't an answer yet, and there wasn't any AI involved yet, but it was the raw material a website agent would eventually answer from.

Ten products out of 249. Some of them only partly captured. On paper, that's not much.

It felt like a lot.

Start small enough that you can actually look at every result. Ten products you can read will teach you more than 249 you can only count.

Eventually I'd crawl the whole catalog, and that went less smoothly. But that was later.

The next question

With ten products in a file, I started doing the natural thing: imagining the questions people would ask a website agent that knew about them.

What does this product do? Which one should I buy? Does it come with a manual?

Then I started thinking about the questions people actually ask when they come to a website like ours.

That's when I noticed most of them weren't about products at all.