Google Apps for Your Domain and ROBOTS.TXT

I’m a technical sort of guy, so when it comes to setting up a website the natural thing for me would be to build it using raw HTML/Javascript or maybe use a web framework such as Zimki, Ruby On Rails or MODx. But when it came to setting up a web presence for our new IT coaching business (helping individuals and VSBs – very small businesses – make use of IT ), I decided to use the type of service we would recommend to others. I picked Google Apps for Your Domain.

While the email and calendar facilities offered by Google Apps are superb and the web page building tools are easy to use, there is (or was see UPDATE below) one major flaw. Google Apps applies a robots.txt file that prohibits all search engines (including Google’s) from indexing the site. This wasn’t always the case, it appears to have first raised its ugly head sometime in November.

Where’s the logic in this ? As it turns out none, it’s a bug, well it may be a bug as a comment I left on Peter Coopers blog prompted him to email a contact he had within Google, who contacted somebody else within Google, who confirmed it was a bug. Makes you worry about handing over too much of your infrastructure to “faceless companies”. Who do you call, or email? Where are the bug databases? Questions on the support forum, remained unanswered. I guess we can always ask guys like Peter Cooper, thanks again Peter.


The bug may already be fixed, at least for my domain. Still seeing the offending robots.txt appearing occasionally on this side of the Atlantic, but accessing the apps sub-domain via an Amazon EC2 image (hosted in Washington State, USA) the robot is now consistently blank.

Now I have to make a decision, will I move the site back onto Google Apps (had moved the “www” domain to Zimki )? Better do something fast before the duplicate content on apps / ends up being penalised!


2 responses to “Google Apps for Your Domain and ROBOTS.TXT

  1. Pingback: Zimki - trust and hope. « Gobán Saor

  2. Pingback: Google forgets to renew JotSpot domain! « Gobán Saor