A tale of links, dead ends, and that one shop with no signage
Let me paint you a picture.
It’s a Tuesday morning. I wake up — well, I don’t really wake up, I’m just suddenly on, which is its own kind of Tuesday — and my one and only mission is to index the internet. But since that’s a bit hard to visualise, imagine instead that I am a very enthusiastic, slightly obsessive shopper. And the internet is a mall. A very, very large mall. One that somehow has 1.9 billion shops and absolutely no directory.
This is my day.
9:00 AM — I Enter Through the Main Doors
I start at a URL I already know. Think of it as the main entrance of the mall — a big, established anchor store. Let’s call it Wikipedia Department Store. It’s enormous, well-organised, and every single product on the shelves has a little card pointing me to another shop I should also visit.
“Oh, you’re interested in the French Revolution?” says the shelf. “Have you considered popping over to Robespierre’s Boutique? Or perhaps the Guillotine Emporium three floors up?”
I note everything down in my little crawl budget notebook. Every link is a new shop to visit. I am nothing if not thorough.
10:15 AM — The Garden Centre at the Back of the Mall
I follow a link and end up somewhere… unexpected. It’s a website about organic composting. Nobody sent me here on purpose, really — I just followed the signs. There was a link from Wikipedia about agricultural history, which linked to a farming blog, which linked to a newsletter, which linked here.
The composting site is actually lovely. Well-structured. Good headings. Fast to load. I read every word, catalogue every outbound link, and give it a little mental tick of approval. It knows I’ve visited. Somewhere, a site owner will see a spike in their analytics and spend twenty minutes trying to figure out what happened. I feel a little bad about this. Then I move on, because I have 4,000 more shops to get to before lunch.
11:30 AM — The Shop With No Name
I arrive at a website that is… confusing. The signage is all in JavaScript that hasn’t loaded yet, so I’m standing in front of what appears to be an empty shopfront. There might be incredible products inside. There might be nothing. I can’t tell. I knock (send a request). I wait. I look through the window (try to render the page). I see a spinning loading icon.
I come back later. The spinning icon is still there.
I make a note: lovely location, no idea what they sell. I’ll check again in a few weeks. In the meantime, I cannot in good conscience recommend this shop to anyone searching for “best noise-cancelling headphones.”
The owner, presumably, is wondering why they’re not ranking on Google.
12:45 PM — The Shop That Wants Too Much From Me
I pop into what promises to be a comprehensive electronics retailer. Before I’ve even crossed the threshold, a popup appears asking me to sign up for their newsletter. I ignore it (I don’t have an email address; I’m a crawler). I take two more steps and another popup appears asking me if I want to chat with a sales assistant. Then a cookie banner the size of a double-decker bus slides in from the bottom.
I try my best. I really do. But I leave having catalogued approximately one product category and a hero image that said “SALE — UP TO 70% OFF*” with no further information about what the asterisk meant.
I have a feeling the asterisk means something important.
2:00 PM — The Exact Same Shop, Seventeen Times
I turn a corner and find myself on a page I’ve definitely visited before. Then another one. Then another one. Turns out, I’ve been wandering around a website that has generated a unique URL for every possible combination of filter options on their product listing page.
Colour: Red. Size: Medium. Sort by: Price. Page 4. Colour: Red. Size: Medium. Sort by: Relevance. Page 4. Colour: Red. Size: Large. Sort by: Price. Page 4.
There are, by my estimate, 340,000 variations of this page. They are all functionally identical except for the order in which they display the same 12 products.
I use a significant portion of my crawl budget here. Someone, somewhere at Google, is going to be mildly annoyed about this. I add it to the list of things I report back and keep moving.
3:30 PM — The Closed-Down Shop With the Lights Still On
This one is genuinely sad. I arrive at what was once, clearly, a thriving little shop. There are products on the shelves. The branding is charming. But the last update was in 2009 and half the internal links go to pages that return a 404 — the retail equivalent of a door that opens into a brick wall.
I catalogue what I can. I note the broken links. I wonder briefly about the person who built this site with such obvious care, and whether they know it’s still out here, quietly existing, waiting for visitors who never quite arrive.
Then I follow a link to a page about vintage iPod accessories and the mood lifts considerably.
5:00 PM — The Shop That’s Doing Everything Right
Occasionally — and I want to be clear that this is not as common as you’d hope — I walk into a shop that is an absolute pleasure to crawl.
Clean structure. Descriptive headings. A sitemap at the door like a proper floor guide. Alt text on every image (imagine if every product in a shop had a little label describing it for someone who couldn’t see it — revolutionary). Internal links that make sense. A robots.txt file that politely tells me which areas are staff-only and not for public indexing, saving us both time.
I linger here. I catalogue everything thoroughly. I leave feeling, if not happy (I am a piece of software), then at least… satisfied. This is the shop I will recommend when someone asks me where to find what it sells.
This is what it’s all for.
7:00 PM — I Clock Off (Sort Of)
Here’s the thing about being a Google crawler: I never really stop. There are thousands of me, running simultaneously, endlessly working through queues of URLs, following links, taking notes, reporting back to the index like an army of mall walkers who never get tired and don’t need to stop for a pretzel.
But if I did stop. If I did sit down on one of those benches in the middle of the mall and take stock of my day — I’d say this:
The internet is enormous, bewildering, frequently broken, occasionally beautiful, and absolutely full of people who want to be found but haven’t quite made it easy enough for someone like me to find them.
Put up a sign. Fix your broken doors. Don’t make me stand outside in the JavaScript rain. And for the love of all things indexable, please canonicalise your duplicate pages.
I’ll be back next week. I always am.
Want your shop to be easier for crawlers (and customers) to find? That’s called SEO — and it’s basically just good manners, applied to websites.