Comments on: How often do search engines crawl the web?

By: MarkRogers

MarkRogers — Sat, 15 Mar 2008 06:35:44 +0000

I think that having good inbound links definitely helps being crawled. Having a flattish site helps with being crawled too. And publishing regularly … hence the continuing effectiveness of blogging (despite the sharp decline in hype!). Corporate sites, with their deep structure and their Stalinist control over the message, fare badly. University websites, with their devolved structure, multiple home pages and well-maintained individual pages, do rather better. One could imagine a corporate site of the future having more of a wiki structure where any department or individual could post (within their guidelines) and everything useful to a company’s multiple stakeholders (clients, partners, employees, investors) got published.

By: Adept Infotech

Adept Infotech — Mon, 04 Feb 2008 12:24:20 +0000

I think it depends on the inbound links from some sites having good PR.

By: Individual Post Pages

Individual Post Pages — Wed, 12 Sep 2007 07:45:00 +0000

[...] Over the past few weeks, I have been paying close attention to how search engines crawl my blog. I have been paying attention to how often my blog is crawled, the number of pages that get crawled per visit, the length of time between visits as well the depth of each crawl (both internal and external depth). The new design for my individual post pages is a direct reflection on what I have learned over the last few weeks of studying the crawling behaviors of my blog. All in all, and to keep the explanation simple, references and links within my articles typically weren’t getting crawled and if they did get crawled it didn’t seem to hold much weight towards honoring any kind of credit to the source page I was linking to. An even easier way to explain it is that by the time my articles pages were crawled, the crawler typically ran out of gas before it finished following all the links in my article. The crawler would continue where it left off the next time it came back, which typically took about a week. [...]