Recently a single text file was inappropriately moved from a test environment to our live servers and it killed almost 40% of our search-driven traffic for 3 weeks. Trust me, it sucked.

Now that it’s over and my heart rate is back to normal and search traffic is back better than ever, I thought it would be a good item to share.

Any techie will quickly guess that the text file that farked our search traffic is the robots.txt file. Robots.txt is a web-root-level file which tells respectful search engine spiders which files and folders they should not index. It’s a great way to ensure that cache files, admin areas, archived content, test files, etc. do not get indexed. It’s also very handy to place on beta testing servers so no beta URLs will ever appear in search results. A while back we found that a better way to block outside access was to password protect all non-production servers. However, we didn’t remove the robots.txt file form those servers and they sat there like a ticking time bomb.

So this lessons isn’t about being vigilant about your robots.txt file, it’s about being vigilant in keeping a holistic view of your whole web app and keeping it as small and efficient as possible even as your project gets more and more complex. Inevitably, a series of events unfolds where the unthinkable manages to occur due to unrelated decisions happening over a period of time.

Our search Traffic from 5/26 - 6/15

We allowed the bomb to detonate during a recent upgrade to our code deployment system which inadvertently moved the useless-but-still-there robots.txt file from a staging server to our production web servers which immediately began rejecting all requests by search engine spiders to index our site. At first it just blocked new pages from being indexed, but after about a couple weeks, it started down ranking and removing all our pages from their search results. This was when we first realized something was up.

I’ll be honest and admit even when I isolated our reduced traffic to reduced search traffic, I didn’t prioritize researching the problem. If I hadn’t let myself believe a national heatwave might be causing a reduction in search engine queries, I might have taken the time to look at the robots.txt file. If I had immediately gone to Google Webmaster Tools, I would have seen a warning that our SiteMap.xml file was inaccessible. But we ended up letting a couple too many days go by before putting out the red alert and when we found the robots.txt file had been overwritten we fixed it in about 8 seconds. Yet just as it took a while for search engines to start down ranking and removing our pages from their indexes, it took a couple weeks for them to revert all pages to being indexed and highly ranked.

The result is that our numbers for May were down and June is a complete anomaly. Fortunately, this mistake was not too costly from a long term revenue or usage perspective. The affect was limited to reduced unique visitors and to a lesser degree overall page serves. We lost some momentum with new user growth and had to sustain a heart attack or two until we knew the traffic was on it’s way back. In fact, this past week marks our highest level of search traffic ever =)

For laughs, here’s a chart of our weekly search traffic that shows the aftermath. The affect is also clearly visible in our Compete’s numbers which I won’t look at again until July’s number shows us entirely back on track. (which our daily stats indicate we are ;)

* Starting around May 28th search traffic starts dropping off
* For three weeks we averaged about 60% of our normal search traffic
* Traffic didn’t fully resume until early July
* Ouch