Search Engine Guidelines for Webmasters
Search Engine Honesty
Webmaster Guidelines - Search Engine Censoring Policies
All the major search engines provide official guidelines for webmasters that speak to their censorship (banning) policies.
Google’s guidelines strongly suggest that Google does not censor English language (as opposed to Chinese) web sites except for “deceptive practices.” Google, possibly because of their position as industry leader, takes a somewhat higher road than their competition. Google’s unofficial motto is “Do No Evil”.
However, there is much evidence that Google censors small-business sites for inconvenient practices and probably for competitive reasons. Some webmasters think that a more appropriate and honest motto would be “Our Tentacles are Everywhere.” Google censors many more sites than the other majors. See Case Studies on Search Engine Banning and Bias.
While Yahoo cites deceptive practices, its guidelines allow censoring for overtly editorial reasons. Yahoo is more concerned with site “quality” and indicates that they censor sites “that provide a poor user experience.” “YST's Content Quality Guidelines are designed to ensure that poor-quality pages do not degrade the user experience in any way. As with Yahoo!'s other guidelines, Yahoo! reserves the right, at its sole discretion, to take any and all action it deems appropriate to insure the quality of its index.”.
“Quality” is of course a totally subjective concept that could be used as an excuse to censor any site. It is hard to imagine a design so bad or quality so low as to justify banning an entire site from Yahoo’s extremely un-exclusive index, which is said to contain 19.2 billion web pages.
MSN Search: http://search.msn.com/docs/siteowner.aspx?t=SEARCH_WEBMASTER_REF_GuidelinesforOptimizingSite.htm
MSN Search guidelines for webmasters are very brief. Here is what they say on the subject of site censoring:
"Items and techniques discouraged by MSN Search
The following items and techniques are not appropriate uses of the MSN Search index. Use of these items and techniques may affect how your site is ranked within MSN Search and may result in the removal of your site from the index.
§ Loading pages with irrelevant words in an attempt to increase a page's keyword density. This includes stuffing ALT tags that users are unlikely to view.
§ Using hidden text or links. You should use only text and links that are visible to users.
§ Using techniques to artificially increase the number of links to your page, such as link farms."
However, our data indicate that MSN Search also censors sites for inconvenient practices not stated in their guidelines and may also censor to suppress competitive small businesses. See Case Studies.
Is Your Site Banned?
You can find out if a search engine censoring department has singled out your site for exclusion by doing a search for “site:www.domainname.com”. The engine will report the number of your pages it has indexed. If it says zero or one, your site has been censored. If your site has been de-listed, “link:www.domainname.com” may also report zero even though other search engines report that you have incoming links.
Yahoo Search is using a second link-only banning method (See SeekOn Case.) in which the site's pages are not indexed by the search engine but links to the site that have been found on other sites are still included and can theoretically be found in a search. The pages have not been indexed so text on the pages cannot be found in a search. The pages are not cached. No descriptive text is provided in the unlikely event that such a listing is included in a search result, so a user is unlikely to be able to determine what the site is about. The title text provided in the search results is the "link text" provided by the site linking to you. These "supplemental results" are essentially useless for search users and are apparently included so Yahoo Search can claim to have a larger index than its competitors. If the site: search reports many pages, check to see if the listings consist entirely of "link-only" entries. Your site is banned if the search engine is not indexing your text. The link-only entries result from indexing other people's sites. Link-only banning adds insult to injury. Yahoo Search is claiming your pages as included in their index while actually refusing to index them!
Another test method is to search for some unique phrase that appears only on your pages. Some webmasters include a phrase such as “Vmxzwh S34pp” on their pages for this purpose. This search will avoid listing “link-only” entries and will also disclose some of the other sites (not banned by the search engine) that have stolen your content, if any. (See The Redundancy Explosion.) The “ultimate misfortune” is to find that your site has been banned but that many sites that have stolen your content have not been banned by the same search engine!
We would expect that spiders would stop visiting a censored site. They do visit much less frequently. However, they usually continue to access some pages and to check /robots.txt apparently to monitor that the site still exists. Yahoo’s robot “Slurp” checks /robots.txt up to several hundred times per day. One reason for doing this might be to determine if a domain name is sold to another owner by detecting a substantial "down" period.
Search Engine Site Reinclusion Procedures
All the major search engines have procedures whereby a webmaster who has determined that his site has been censored, has somehow determined which was the offending feature, removed the offending feature, and is suitably repentant, can apply for reinstatement:
Google: You can go to: https://www.google.com/webmasters/tools/reinclusion?hl=en
You will have to sign up for Google's Webmaster's Tools Service first.
Google only solicits reinclusion requests from sites that admit and stipulate that they violated Google's guidelines or inherited a domain name that they suspect violated Google's guidelines.
"By submitting this form, I acknowledge that: I believe this site has violated Google's quality guidelines in the past. This site no longer violates Google's quality guidelines. I have read and agree to abide by Google's quality guidelines."
You can include a note pleading your case, explaining all the corrective actions you have taken, and promising on your genuine Batman ring never to do it again. Google might eventually reinstate your site. There are persistent reports that Google has “punishment periods” that vary depending on the “offense”. If your site has been banned for editorial or competitive reasons, was banned by mistake, or if you don't know why it was banned, you are out of luck. There are some indications that Google generally bans larger small-business sites regardless of merit.
Google people, looking down on lowly webmasters from their lofty positions as industry leaders and employees of a $100 billion company, tend to be unusually condescending. Google treats webmasters like children with “punishment” and “penalization”:
“Tommy, you’ve been bad. Go to your room.”
“Why Mommy? Why?”
“I’m not going to tell you why. I’m the parent and you’re the child. I don’t have to tell you why.”
“How long do I have to stay in my room Mommy?”
“I’m not going to tell you that either. I’ll let you out when I feel like letting you out.”
Yahoo: You can request a “re-review” at: http://add.yahoo.com/fast/help/us/ysearch/cgi_rereview
In Yahoo parlance, the “original” review was the one they did when they decided to ban your site. Your message should include all the changes you made in order to be in compliance with Yahoo guidelines.
MSN Search: You can go to: http://support.msn.com/eform.aspx?productKey=search&page=support_home_options_form_byemail&ct=eformts and fill out an MSN Search problem report form “Search, Problem, Finding a Web Site” stating that your site has been deleted.
You will get an immediate web reply with a ticket number promising a support person will be in touch. Soon, you will get an email from a support person directing you to help pages, guidelines, etc. You reply to this email with your plea, including that you have indeed confirmed that your site has been intentionally de-listed by MSN Search. You will then eventually get a reply that the support person has forwarded your plea to the MSN Search Product Specialist Team. Eventually, your site might be reinstated.
A very curious note: The reply email from Microsoft Support contains a number of features usually associated with email spam and therefore was tagged as spam by our Spamassassin anti-spam software. The items flagged by the filter included: "HTML obfuscation", "Envelope sender in postmaster.rfc-ignorant.org ", "Message-Id has pattern used in spam", and "Reply-To is empty". Be careful or you might miss the reply message.
Alternately, you can try to follow this note on MSN's site: "We may remove a website from the index if the website was reported as spam. If you suspect that your website was incorrectly identified as spam, please send an e-mail message to firstname.lastname@example.org."
Microsoft is a huge company and MSN is a large division but MSN Search is a relatively small group. Although MSN has an extensive, responsive, and well managed support department, it is our impression that the “team” responsible for MSN Search is still very new, relatively small, and feeling its way. Despite all the hype by Microsoft that they are going to “bury” Google, Microsoft does not appear to be expending all that much effort on search. Just about everything about MSN Search is pretty primitive when compared to Google and Yahoo Search. Once your request has been sent to the “MSN Search Product Specialist Team” you may be in for a very long wait before you get any response. The “reinclusion” department at MSN Search may well be essentially non-existent.
Handling reinclusion requests at search engines is not a priority and reinstatement can reportedly take months or even years. Search engines often do not advise sites if they are going to be reinstated so site owners spend a lot of time waiting and wondering.
Search Engine Honesty
Copyright © 2006 Azinet LLC