.

Sending Email is a Big Headache for Small Companies with Popular Websites

While many web technologies are getting much cheaper (if not completely free) some services are becoming much more expensive and time consuming. Sending emails is a perfect example.

About 10 months into existence we experienced our first misclassification as a spammer, eaming all our sent mail goes straight to bulk or trash folders. Since then it’s been a day-by-day battle to remain white-listed by all large ISPs (e.g. Hotmail, Gmail, Bellsouth, Comcast, AOL, MSN, Yahoo, etc.) We understand their services have been crushed by spam for years and they’ve have to over-aggressively filter, bounce, or silently delete incoming mail, making big headaches for innocent small and medium businesses

When we first had problems in late 2004 we were sending out 50,000 member-requested emails a week. Since mail functions are so deeply wired into core server technologies the average web engineer has no ideas how many emails are going out. I sure didn’t. But now I do. We’re sending about 400,000 member-requested emails a week. We’ve had to dedicate significant amounts of time (and thus money) to become sendmail geeks, mastering the myriad error codes, loopback systems and contact policies of the large mail providers and scripting many back-end and website functions to stay in compliance as best we can. We’ve even had to retain a mail delivery analysis partner, and still debate if we’ll have to pay a third party to send all out emails. If one of us was already a master of sendmail, and had connections at big mail providers this would be easy, but how many people fit that descirption in the average 10 person start-up?

Here are all the steps we’ve had to take to be able to send our own emails - some are much harder and more expensive than others.

Perfect and frequently review our outgoing mail headers so they always conform to the ISPs preferred settings.

Collect, filter and database every single bounce-back email. Depending on the response error we than must wait 30 days or never send an email to that addresses again. (ISPs keep track of each bounceback they send out. If they send a bounceback twice they consider you at-risk even though the member has asked us to send them an email at that address.)

We have a double opt-in system for all members that want to get emails form us. Since day one there has been an option to ‘never get email’

We have a two-click unsubscribe feature at the footer of each newsletter. Member password is not required.

All member-request emails, such as friend-invites and virtual gift notifications offer direct links to modify their account to not get such email notifications.

We have set-up loopback emails accounts for all the major ISPs. Emails are sent to these accounts each time a person marks mail from us as spam. Marking email as spam is a huge problem because the member innocently believes they are harmlessly keeping their inbox free of email they are no longer interested in, but they are also telling their mail provider we are a probably a spammer. The ISP tehn forwards a copy of the email the person marked as spam and for now we manually open each one of those figure out who it was sent to and manually unset that user’s preference to stop receiving such emails. (Yes, we’d love to script this but their are so many variants we still do by hand.)

We have added code to our website that alerts members if an email we sent them was bounced backed and how they can fix it. Once we get a bounce-back we can’t email them for at least 30 days even if it was a 5 minute problem because someone sent them a 50Mb attachment. (Asking members for an alternate email from the get-go now sounds much more appealing).

We have retained a professional mail delivery monitoring service (ReturnPath) which has hundreds of test accounts over all the big ISPs and we can determine which ISPs are routing our mail to bulk folders or worse. It’s important to realize that of the many spam checks an ISP deploys is messages-sent-per-hour. Thus, we can be a member in good standing, then a spammer, then a member in good standing all in the course of a three hour period.

We have notification code in a couple places on the site that alert members who are using an ISP that misdeliverying our mail to bulk or spam folders that they should either whitelist us by adding our email address to their mail client’s address book, or switch to using a email address with a more trusting mail server.

While I used to be disgusted at AOL’s and Yahoo’s plan to start charging email senders to deliver their messages, considering the money we have to spend to stay in their good graces, we would now prefer to simply pay in advance for our emails to be delivered and know that they will.

Also want to say we’re not proud or bragging about any of our solutions. If anyone knows how to do these things better, more effieciently or has better connections in the mail departments of any major ISPs we’d really appreciate sharing any ideas in the comments.

UPDATE 12/19/06: AOL just whitelisted our entire domains. Praise Dog! This has been a very long time coming. It’s hardly as simple as asking. ALL email-sending IPs have to be considered spam-free for decent length of time. To stay whitelisted we’ll have to keep their complaint rates under 0.1%, bounce rates under 10% and bounce acceptance rates above 90%. Fortunately we can now dip under these rates and get a postmaster email instead of an immediate spammer classification. As long as the problem is rectified quickly we should remain whitelisted.

Digg It or Del.icio.us It

Share it! These icons link to social bookmarking sites where readers can share and discover new web pages.

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

Barked by Ted Rheingold

11 Woofs

1. 12/2/06 at 12:20 am Doug Karr
   

   Investing in an Email Service Provider is a good investment. I would highly recommend ExactTarget (http://www.exacttarget.com), one of the fastest growing technology companies in the country.

   I’d also add that the rumors of ‘charging email senders’ is just that… misinformation. Paid delivery is no different from comparing FedEx to USPS. If you want to get your package there, you pay a little more. I believe it’s a solid plan.

   ExactTarget has a robust API and integration capabilities. It’s a SaaS (Software as a Service) so all you need is a browser to get started.

   Warmest Regards,
   Doug

2. 12/15/06 at 5:21 pm Jay
   

   You briefly mentioned that you are still debating outsourcing the email. How did you decide to keep it in-house given that it is such a hassle? I’ve been working for a micro-business, and we have the same problem, but never really solved it.

   Thanks for going into such detail on this post and the other one on SEF urls. It’s very enlightening to see how this works behind the scenes.

3. 12/17/06 at 12:54 am Ted R.
   

   Jay,

   Spending money we didn’t have was a big reason we kept it internal. Another was that we’ve yet to encounter a technical problem we couldn’t resolve ourselves.

   Not sure how this one will play out yet.

4. 12/31/06 at 12:25 am paul king
   

   Talk about an eye-opening blog post. Holy Dog! is right.

   The reason I’m reading your blog is I found you as a result of reading Jeff Clavier’s blog looking for more information in starting my own SN (nothing like yours or anyone else’s - for once someone can say with certainty, it doesn’t exist). Anyhow, being as I like to say I am “brilliantly creative, technologically inept” my only hope was to provide the vision, look, feel, biz dev, etc - and after reading this, thank the lord! - as I now know for certain that even if I *could* write a smidgen of code (and I can’t) it would not be where my passion truly lies.

   I understood your problem, and I felt your pain through that post…. just pray it’s not something I’ll ever have to personally deal with… and should some miricle happen and my SN actually get off the ground, and we were to run across a similar problem, I will certainly have those in charge come over and read this.

5. 1/21/07 at 8:52 pm Crooked Number » Blog Archive » Thinking about Sending youe Own Mass Emails?
   

   [...] Sending Email is a Big Headache for Small Companies with Popular Websites              [...]

6. 1/26/07 at 10:52 am innonate » links for 2007-01-26
   

   [...] Dogster Inc. Company Blog » Archive » Sending Email is a Big Headache for Small Companies with Popular Websites With the fast approach of VentBox, I’m just starting to think about the email issue. This is a good overview of what I should have been learning about these past months. (tags: tips whitelist spam email business) [...]

7. 12/25/07 at 4:29 pm links, ideas and geek stuff » Blog Archive » links for 2007-12-26
   

   [...] Dogster Inc. Company Blog » Archive » Sending Email is a Big Headache for Small Companies with Pop… [...]

8. 1/2/08 at 12:31 am links for 2008-01-02
   

   [...] Dogster Inc. Company Blog » Archive » Sending Email is a Big Headache for Small Companies with Pop… While many web technologies are getting much cheaper (if not completely free) some services are becoming much more expensive and time consuming. Sending emails is a perfect example. (tags: business email entrepreneurship howto mail marketing network social startup sysadmin tips toread whitelist spam) [...]

9. 1/2/08 at 3:46 pm derek
   

   Is it possible for smaller sites to set up “loopback email accounts” with the major ISPs?

10. 1/2/08 at 5:11 pm Ted Rheingold
    

    Hi Derek,

    You gotta set them up one by one. Not sure if they deny anyone based upon volume. The good news is that as a smaller service, the total number of flagged emails a day is unlikely to cause problems, but better to get your account in shape early vs. later.

    To get ya going here are some postmaster pages

    Yahoo - http://help.yahoo.com/l/us/yahoo/mail/postmaster/index.html

    AOL - http://postmaster.aol.com/

    Hotmail/MSN - http://postmaster.msn.com/

11. 4/24/08 at 1:10 pm Nabeel Hyatt
    

    Triggermail is trying to do just this, btw. Although I haven’t used their service so no idea if they are successful at it.

Leave a Reply