One of my associates recently sent me an email requesting:
One of the questions during the [recent sales presentation] was: “How often do search bots visit a site?”
I know this is a “it depends” kind of answer, but I’d love to get a paragraph from you on that so I can send it to them.
A paragraph or two? Nonsense. A half dozen? That I can do 🙂 My response:
—
It depends.
Some sites get “crawled” by search engine spiders every day or even several times per day. Typically these sites are either very highly authoritative (i.e. very popular websites) or they have frequently updated content. Ideally both.
If a website has new content on a regular basis a search engine should visit it frequently to see what has changed. If a website is only updated or has new content only once or twice per year, a search engine won’t be encouraged to visit it very frequently b/c chances are nothing has changed since the last visit.
Further, all “crawls” are not the same. Most frequently a spider will visit the homepage or a handful of pages just to see if there are any changes since last visit, and if so get an idea for the magnitude of them. If major changes have been made (such as a redesign), new webpages added, etc. than a “deep crawl” will come later on. As such, the homepage could get crawled 4 times a week but the first level interior pages only 1 time per week, and the second or third level pages only 1 or 2 times per month.
However, don’t mistake frequency of being crawled with search engine “mojo” or high rankings. There are many sites that haven’t been update in years, might not get crawled frequently but can still rank well. The inverse of that is true as well.
Really there are two steps here. 1) get crawled, 2) improve rankings. Step one merely gets you “in the game”, meaning you are a result in the database, though it may not be in the position you’d like. Step 2 goes from merely being one of 10 million results to actually be a good one, in the top 10 in most instances.
Getting crawled is generally not the challenge, its almost a given provided a few basic things are done correctly. Getting top rankings is where the challenge is and that’s where an expert provides their value. The Gravograph website should not have any issues being crawled properly and frequently enough.
I am also looking for the answer for the same question… Is it weekly, monthly, bi-monthly etc? My PR dropped from 4 to 2 and I don’t know when I can have it at 4 again..:(
Interesting explanation. I hope to get a site crawled frequently, and hopefully often as well. I can see how frequently crawled sites can gain higher ranking especially if the new content is valuable. Would you rather have one very deep crawl, or many “shallow” ones? Thank you for the post, it answered some questions I had.