Crawling a Website Is Easy. Crawling It Well Is Not.

Ten products had taught me the crawler worked.

They hadn't taught me whether it worked on everything else.

At the end of March, my plan had grown from "crawl the products" to "crawl the website." Product pages, support pages, policy pages, manuals. I had a crawler that did a nice job on ten hand-picked products, and I figured the next step was mostly a matter of pointing it at more pages.

That's not quite how it went.

The newer product wasn't a one-off

In the first crawl, one of the newer products had come back only partly captured. Its page looked perfectly normal on the website, but something underneath was just different enough that the crawler missed things.

I'd filed that away as a quirk. One odd page out of ten.

It wasn't a quirk. It was a preview.

Once I started crawling more, the same thing kept happening in slightly different ways. Pages that looked consistent to a person turned out to be built differently underneath. A section in a different spot. A field with a different label. Information that lived in one place on most pages and somewhere else on a few.

Some products had specifications in different places, and even when they were in the same spot, the underlying HTML wasn't always structured the same way. A few pages even pulled information from external sources using JavaScript, which meant the data wasn't always there when the crawler first loaded the page. That added a whole new challenge I hadn't really planned for.

None of it was wrong, exactly. A website is built over years, by different people, with different tools and different priorities. Of course the pages aren't all the same. I work on this site. I should have known that better than anyone.

Missing isn't the same as empty

Some products had manuals and some didn't. Some had a full set of specifications and some had a few lines. That's normal. Not every product needs the same information.

The tricky part was telling the difference between "this product doesn't have specs" and "this product has specs and the crawler didn't find them." From the output, those look exactly the same: an empty field.

That's when I realized partial results are worse than failures.

If the crawler crashes on a page, I know. There's an error. I go look.

If it quietly pulls back a record with everything except the specifications, nothing tells me. The record looks fine. It's labeled, it's structured, it's sitting in the file with all the others. It just happens to be missing the one thing a customer is going to ask about.

And eventually, a website agent would be answering questions from that record, with no idea anything was missing.

Going bigger, carefully

When it came time to crawl the full product catalog, I didn't just run it against everything. I'd learned that much from the first ten.

I ran a larger batch first, about 50 products, as a check. Did the crawler hold up? Were the results still readable? Were the partial results getting better or worse as the variety grew? Only after that looked reasonable did I run the rest, about 267 products in all.

It's the same lesson as the first crawl, just one size up. Do something small enough to check before you do something too big to check.

I did the same with the rest of the website. The support and policy pages went in too, the ones I'd skipped in March and then realized held a lot of the answers people were actually looking for. They brought their own variety. A product page at least follows some kind of template. A policy page is mostly paragraphs, written whenever someone needed to explain something, and shaped however that person thought made sense.

Working isn't the same as dependable

Somewhere in here my goal changed, and I didn't notice it at first.

In March, the goal was "get the crawler working." It was working. It had been working since the first ten products.

The new goal was "get the crawler dependable," and that's a much bigger job. Dependable means it handles the pages that don't look like the others. It means I can tell a real empty field from a missed one. It means I can run it again next week and trust what comes back, even if someone on the web team has changed a page in the meantime.

A proof of concept can make something look solved long before it is dependable.

That idea comes back over and over in this project. Pretty much every part of the system had a moment where it looked done, and then turned out to be the starting point.

Then there were the manuals

By now I had a lot of product information coming back in decent shape, and I had a much healthier respect for how much variety was hiding in a website that looked consistent from the outside.

But I'd been putting off one thing.

The product pages were HTML. That's at least something a crawler is built to read. The manuals were PDFs. And I'd told myself, back when manuals showed up on both of my question lists, that they'd probably be the easy part.

They were not the easy part.