There are spiders on your internet.

On its surface, the internet typically seems like a friendly place - or at least generally neutral. You'd hardly expect to find an infestation of spiders online (although other sorts of infestations would probably be unsurprising).

Unbeknownst to most casual browsers, though, the World Wide Web is (fittingly, I suppose) teeming with them. Also known as web crawlers (or bots, web scutters, or automatic indexers), these spiders are computer programs that trawl the internet for content and report their findings back to search engines and other similar services. Without web spiders, search engines like Google would be forced to assign humans to browse through and catalog each and every existing web page by hand - an impossible task, given that billions of different web pages exist.

Instead, though, search engines can deploy these automatons to do their dirty work for them (and depending on what parts of the internet you frequent, I'm sure it can be very dirty work). Most webmasters are familiar with the scurryings of internet spiders; they typically come during the night, and poke and prod and crawl over and under each part of your website. Adverbly.net has been visited by at least nine different "robots/spiders," including Googlebot, Yahoo Slurp, Ask, Relevant Noise, and several somewhat-creepy "unknown robots" identified only by short strings like "robot," "spider," or "bot." I can't help but wonder who directed these unknown bots to crawl this website; one visited nearly 200 times.

If you're interested in learning more, you can visit the quite-extensive Wikipedia page about web crawlers, or Google's helpful-but-brief Googlebot FAQ.

Some say that a spider in your house isn't all bad, because spiders hunt and kill house centipedes. I'll take house spiders over house centipedes any day. And I definitely don't ever want to meet an internet centipede.