Meet the Search Engine Bots

Have you ever wondered what a search engine bot looks like? Well, I've met them first hand at Search Engine Strategies, and they're a scary beast. Yahoo's Slurp, Google's Googlebot and others are strong bots, running around and grabbing pages from all over the web. Once trapped, you'll have a hard time getting them to let it go. They'll hold on to it for a very long time, until at some point it either gets banned or it vanishes into the supplemental index. They're truly mean bots.

On the other hand, MSNbot (some may say it's a-LiveBot) has the same core concept, but it seems to have missed one important piece. While it still runs around and finds pages, it's got a much tougher time holding on to them. This has led to numerous indexing problems. Hopefully they can get their bot's issues repaired soon, but I'm not holding my breath on that one. They're just too corporate for pincers (defined as "Noun : roach clip, roach holder - metal tweezers used by marijuana smokers to hold a roach").

If you ever find yourself face to face with a bot, remember to let it take whatever content it likes. You can always sue it later to force your intellectual property from its greedy indexers.