The Web Czars

A few people are deciding what you can see on the web.


Search Engine Honesty (


Ted Goldsmith

Azinet LLC


The Internet has brought enormous change in our ability to obtain information.  For the first time in history, anyone with an Internet account can publish information to be read by anyone else with Internet access, worldwide.  This was a fascinating development with the promise of changing the world forever. 


However, access to all this information has become for all practical purposes dependent on search engines, which are now the main method used by people to find Internet data.  Search is even increasingly used to connect to web sites the user already knows about.  It is easier to do a search for “Ebay” than to remember and enter the actual Internet address or deal with "favorite places."  Internet users perform more than 170 million searches per day. 


The situation with search engines is the opposite of the situation with web publishers.  Barriers to entry of major new players are extremely high and there are huge economies of scale and other advantages that accrue with increased market share.  Google is thought to own 150,000 computers.  The software development and maintenance effort is gigantic.  The bandwidth and other resources required to frequently acquire, process, and store data from billions of pages on millions of sites boggle the mind.  As a result, there are currently only three major search engines (Google, Yahoo Search, and MSN Search) processing more than 80 percent of searches worldwide and further consolidation is likely.  Google and its partners handle more than half of all searches.


Most people don’t think of editorial policies in connection with search “engines”, which are seen as essentially mechanical devices and therefore incapable of bias.  Search engines do everything they can to reinforce this impression.  A search for a pornographic word or phrase turns up millions of hits so obviously they are not censoring anything, right? 


Wrong.  In fact, the editorial policy of a search engine operates in a manner not all that different from that of a newspaper.  The policy determines what articles (web sites and pages) are accepted for dissemination through the engine and the prominence given to each accepted article (page) in response to a search.  Relevance to the submitted keywords is only one of the factors determining placement.  Censored sites cannot be found regardless of their relevance.


Search technology is ideally suited to the execution of any desired editorial policy.  For example, it would be technically trivial for an engine to emphasize sites or pages containing the words “republican candidate” relative to those containing “democratic candidate”.  Much more complex and subtle filtering is possible.  In addition to influencing world politics, the ability to bias “organic” (non-sponsored) search engine results has many other obvious applications in areas such as suppression of competition.   Keyword targeted ads handled by major search engines are now the gold standard of web advertising.  These “sponsored link” ads are displayed on thousands of web sites.  Biasing organic results to throw traffic toward sites running a particular engine’s ads would have an instant positive effect on the engine’s bottom line and an instant negative effect on competitors.


Do search engines use their editorial policies for these purposes?  Perhaps.  Search engines have weaknesses that can be exploited by web sites.  The major engines all admit to censoring sites that they consider to be using “deceptive practices”.  Yahoo Search also censors sites that it considers to be of low “quality”.  Engines providing data to citizens of China censor access as specified by the Chinese government.  As history has repeatedly shown, once you start censoring it is hard to stop.  As with newspapers, engines are under no obligation to disclose their editorial policies. 


By all accounts, major decisions at Google are made by its two co-founders.  The other major players are also not noted for wide distribution of management control so it is probable that less than a dozen “Web Czars” are setting the policies that control most of what people see on the Internet.


If there were only three major newspapers in the world, many people would have concerns about their ability to access unbiased information.  These people should be equally concerned about their free access to Internet data.  Search engine censoring is not just a problem for the Chinese.  Bias and censoring on the web are actually more insidious than bias in a newspaper.  It is hard to miss what you don’t know is there.  People who can’t find a site that they have enjoyed in the past tend to assume that the site has gone out of business.  It doesn’t occur to them that their access may have been blocked by the search engine.


When I began working in web site development in the early days of the web, our goal was to build sites that made customers and site visitors happy, and such an approach worked fairly well and was also creative and satisfying.  But the world’s best web site is of little use if few can find it and now a developer’s main task is to satisfy the whims of the Web Czars, a much less satisfying and often very frustrating occupation. 


Copyright © 2006 - 2009 Azinet LLC