Web and email hosting providers without automated outbound spam control are at risk of incurring excessive costs. Spammers compromise provider services, leading to reduced IP reputation and problems delivering email. In this whitepaper, we analyze and explain these hidden costs.
Spammers constantly seek new ways to deliver to their mailing lists. Recently, spammers have increased their use of compromised accounts, servers, and scripts to send spam. In 2012, this type of compromise increased dramatically, as spammers switched from bot-net spamming to the abuse of more reliable and scalable hosting resources (see Figure 1).
IP reputation services like Spamhaus keep track of abusive IPs. When spam is sent from a web or email hosting machine, that machine attains a negative IP reputation and can no longer reliably deliver email to the Internet. In especially severe cases, a provider’s entire network is blocked.
What providers do today
Step 1: Discovering the problem
When an IP address is blocked or rate limited because of a reputation issue, end-users complain of their email being rejected. Complaints typically reach the provider within several minutes to a few hours. Depending on the number of endusers whose email is affected, complaints may swell in volume, clogging support queues.
More sophisticated providers track IP reputation using automated services like MXToolbox, or ReturnPath. These providers may be able to respond to reputation issues before customer complaints get out of control.
Step 2: Who’s blocking us?
Once the provider is aware that email delivery is impaired, attempts are made to discover which IP reputation provider is responsible for the problem. If a blocklist is responsible, identification is usually quick and easy. Reputation issues generated by anti-spam products and services are more difficult to uncover, because email logs will show wide spread connection failures without concise or consistent error messages.
Step 3: Please unblock our IP
The next step is typically to request de-listing of the IP address from the reputation provider. In some cases, de-listing takes 30 seconds via a web-based form. In more severe cases, a dialog must take place with the reputation provider, in which the provider requests proof that some action has been taken to deal with the spam problem. In the most serious cases, the provider is unwilling to de-list the IP address, and may even list the entire provider network
Step 4: Who sent the spam?
Attempts are made to discover which server, script, or user account has been sending spam. Once identified, the offending resource is deactivated – and cleaned up to prevent reoccurrence. Note that this step is often conducted in parallel with the de-listing step, on request of the IP reputation provider.
What it costs
Having a bad IP reputation costs money:
While tracking down compromised accounts and servers may make for an exciting Hollywood film plot, providers wishing to remain competitive need to be strategic about how they invest scarce operational dollars.
Analysis: Calculating the cost of a Spamhaus SBL blocklisting
To illustrate the high cost of IP reputation issues, let’s analyze a hypothetical shared hosting provider with 20 employees and an average per-head fully loaded cost of $100,000 per employee per year ($50/hr), whose shared hosting server is blocklisted on the Spamhaus SBL blocking list. For sake of analysis, we assume that each shared hosting server hosts 500 domains.
Cost 1: Handling tickets related to email rejections: $416
Since we’re analyzing a Spamhaus SBL block, we can safely assume that email from the hosting server will be blocked by nearly every significant email receiver on the Internet (Spamhaus SBL is used by Yahoo!, AOL, Google, and many other receivers as a reason for blocking IPs). Once the block happens, of the 500 domains in service on the shared hosting machine, let’s assume that 10% of the customers lodge a support ticket. Each support ticket requires a total of 10 minutes to resolve, implying 500 x 10% x 10 = 500 minutes of support time in total. The total “cost” of this first round of support tickets is going to be $50/hr x 500 minutes / 60 minutes/hr = $416.
Cost 2: Determining which blocklist was responsible for the listing: $37.50
Spamhaus SBL is a widely used blocklist. The first symptom of a listing such as this is a dramatic increase in rejected SMTP connections to servers all over the Internet. In some cases, connections are silently dropped (i.e. with no error message provided); in others, the server will reject the connection with an error message; and in still others, a rejection message describing the reason for rejection will be provided. The first step of the support team upon receiving a substantial increase in email delivery related support tickets is therefore to analyze the email delivery logs. Let us assume that the hosting provider has access to a log analytics package such as Splunk, which greatly speeds the discovery of this type of problem. A senior support engineer (fully loaded cost: $75/hr) investigates the mail delivery logs using Splunk and lodges an internal high priority ticket to resolve the blocklist event. Total time to investigate, determine the email problem is related to Spamhaus, and lodge the high priority ticket: 30 minutes. Cost: $37.50.
Cost 3: De-listing the IP address: $300
De-listing an IP that has become listed on Spamhaus SBL requires contacting Spamhaus via a support email address. A Spamhaus agent usually responds within one hour to the request, and a dialog then ensues during which Spamhaus requests more information from the provider to determine whether the provider has actually solved the ongoing spam issue that caused the listing. The dialog between the provider and Spamhaus requires a senior technical support resource ($75/hr) and typically takes four hours from start to finish – this is assuming that Spamhaus accepts the provider’s word that the problem has actually been cleaned up. Cost: $75/hr x 4 hrs = $300.
Cost 4: Identifying who sent the spam: $750
The most costly part of the process is the final step: determining who sent the spam, and ensuring that party cannot do so again. There is a high degree of variability in this step of the process. Sometimes the spammer is identified quickly by analyzing email delivery logs. In other circumstances, identification takes longer, particularly when multiple user accounts or application scripts have been compromised. For this analysis, let us assume that the spammer has compromised a PHP script by exploiting a relatively newly discovered vulnerability. Because the issue requires an understanding of PHP, as well as a deep knowledge of the hosting system, two senior support engineers are required ($150/hr). The spent a total of three hours determining
After determining these things, the support engineers manually remove the affected script and lodge a support ticket to notify the customer whose script had been compromised. An internal knowledge base article is written describing the attack and notifying others in the organization to be on the lookout for this vulnerability. Vulnerability information is further disclosed to CERT as part of the hosting provider’s involvement with the security community. Total time required by the two engineers to clean up the script and notify the relevant parties: two hours.
Total cost of this fourth step: 5hrs x $150/hr = $750.
Total cost of a single Spamhaus SBL listing: $1503.50
In our analysis, we have only considered the direct support-related costs of the blocklisting event. We have not analyzed customer churn nor the impact on the hosting brand; however, one can assume that if incidents like this happen regularly, the hosting provider will begin to lose customers, and the cost of such customer churn could easily dwarf the support costs. Additional costs include:
Incorporating these additional costs can drive the real cost of IP reputation issues to many tens of thousands of dollars per month.
MailChannels builds email security products for hosting providers to give users a more secure, more reliable email experience. We help businesses grow by providing tools that protect their servers from sending spam, identify bad actors, and deliver a superior email service. MailChannels is an active member of the Messaging, Malware, and Mobile Antiabuse Working Group (M3AAWG), and the Anti-Phishing Working Group (APWG).