A couple weeks ago, Dogster Inc. had the pleasure of an evening IT session with LAMP-master Tony Guntharp. Tony co-founded the uber-geek Sourceforge and currently is coding up a storm at Feedster.com.

In the fall and winter of 2005 Dogster had completely upgraded our serving architecture to both web and database clusters, along with add email, squid cache we had become a 10 server operation, all being built, and run by two accidental system administrators. We knew we had increased capacity significantly but were not seeing our serving latency improve, in fact it had gotten worse. We combed thru every log file, tested our network, hardware, DNS, you name it without any success. It was a very frustrating period.
For months we had been seeking a LAMP consultant and found no one. But then Scott Rafer introduced us to Tony and it was the beginning of the end of our problems. Tony spent an evening with us at the control seat methodically going thru our entire server architecture. Debian, Apache, MySQL, PHP, bind9 were all found to be well configured, though Tony realized that though I had tried to initiate MySQL cache querying there were about 4 required ancilary config settings I had omitted, but it hardly solved our problem. For a while we thought the problem may be the very slow buffered disks reads Tony noticed using HDparm, but an eventual call to a friend who also used ServerBeach confirmed that was normal.
It started getting late, but Tony kept right in there, corssing off apps one by one. Turns out the silver bullet was a very inefficient phpAdsNew configuration caused by server hopscothc we played while going from 3 to 10 servers. Within 15 minutes we had the problem resolved and cut 50% off our page loading times.

So thanks Tony! You must be The Lone Ranger because you found the silver bullet no one else could see. Within a day we were seeing 20% increases in page serves simply due to the increased speed. We’re on target to server 12M pages this month, well above our 10M peak. It’s a shame we couldn’t meet our peak demand earlier, but bootstrappers gotta do it as best they can.