Web admins edging out Google's competitors at system level
In what sounds like a self-fulfilling prophecy, researchers at Penn State University found that files used by Web administrators to regulate search engine agents (called crawlers or bots) significantly favored Google over it’s close rivals, Yahoo and MSN.
Systems administrators use text files called (robots.txt) to regulate web crawlers, and to keep their servers from getting overloaded. In the robots.txt file, the Web administrator has to explicitly specify which search engines are “favored”, and researchers found a strong correlation between a search engine’s market share and the access given to it’s crawlers.
Google “bots” were favored more than twice as often as those of Yahoo and MSN.
This raises a few interesting points, among them being that Google’s dominance is being further reinforced, at another level — the level of Systems administrators and policy decision makers, who are the de-facto gatekeepers of the web — with an almost-*power law* kind of effect. On the other hand, this could arguably be called a human bias in endorsing what one feels is a superior product.
Scott Cleland, who generally seems to have it in for Google, makes some interesting points in his analysis of the study results, among them being :
“…. what these findings may mean is that consumers are not in fact choosing a search engine based on its algorithm, but choosing implicitly through learned behavior that Google has better access to the content the user wants than other search engines do. “
Is Google slowly heading in the direction of Microsoft, where the force of it’s market share and name brand gradually overshadow the quality and innovativeness of the product?