Search Engine Spiders – Do You Know What They Are?
So, I’m working on a top secret project that I’m very excited about. As part of it, I took on the onus of defining what a search engine spider does.
People who aren’t into SEO probably don’t really know. They probably have a general idea, but it may not be what you think.
Spiders don’t decide where your page will rank in Google, the algorithm does that.
Spiders don’t need to be alerted, they’ll just show up.
Spiders aren’t always good spiders.
Spiders do crawl the web. They go into every nook and cranny where they’re allowed and they gather pages. They take the data on those pages and condense it, then send it back to the search engine so that the algorithm can be applied. That’s what determines where your page will show up in the SERPs (search engine results pages).
You don’t need to do anything special for them to show up. Submitting your website to all the search engines is a waste of time, and paying a service to do that for you is a waste of money. Spiders will come.
How well you prepare your pages by having the right META tags and keywords on your page, and perhaps a robots.txt, to tell them which pages to notice and which to ignore, will help you when it comes time for the algorithm to be applied.
But that’s if the spider is sending your page to a search engine. Some spiders are black widows. Some of them go around gathering pages that have email addresses so that creepy spammers can harvest those addresses and spam you with abandon.
Yet, most of the spiders that frequent your page are the good kind, and without them, there would be no search engines.
The nicest thing you can do for a spider, kind of like leaving cookies out for Santa, is to give them the information they need to send to the search engine. It’s the nicest thing for you, too, because unlike Santa, they are real, and they mean the difference between you showing up in the search indexes or being an orphan.
If you do nothing else, learn what META tags are and include a title and description for every page. Remember, too, that each page is different and requires a different title and description. And don’t be lazy!
Remember, search engine traffic is free. The better you prepare your pages for the spiders, the better your website will do.
