1. Fastmail
  2. Mail
  3. Technical

SMTP stage spam checks

One of the most powerful ways of stopping spam from entering your mailbox is to stop it from being received in the first place!

The way email is transmitted on the internet is via the SMTP protocol. We apply a number of checks at the SMTP stage to stop spammers.

RBL/RHSBL blocking

RBLs (real-time blackhole lists) and RHSBLs (right hand side blackhole lists) are a very common way to detect and stop known spam sending machines. The main problem with RBLs is that they often have "false positives" and block legitimate systems, incorrectly causing email to be bounced back to senders. To stop that happening, Fastmail minimises the use of RBLs and RHSBLs for outright blocking.

We use only one RBL for blocking, the SpamHaus XBL. The XBL is a highly accurate block list that lists the IPs of machines with known trojans and proxies. Independent testing shows that the ZEN RBL (of which XBL is a part) has a high block rate, with basically no false positives.

We use a few RHSBLs for blocking the hostname or HELO string of connecting servers. RHSBLs usually have lower false positives because the domains tend to have to be explicitly added or require multiple collected reports, unlike RBLs which can easily automatically collect IPs based even on single user reports in some cases.

A number of other RBLs and RHSBLs/URIBLs are used during the spam checking phase that occurs after email is received, but these RBLs never outright block email; they're used as part of an overall scoring analysis along with many other factors to filter spam into your Spam folder.

Address enumeration detection

Address enumeration detection is designed to stop other people trying to find what email addresses are valid at Fastmail. It does this by detecting attempts to send to many different, but similar, email addresses in a short period of time from a non-email server host.

Address enumeration detection provides a defense against spammers trying to discover valid email addresses simply by trying many different addresses over and over.

RFC violations

When a server is transferring email from one system to another, there's a series of conventions they have to follow defined in the SMTP RFC 2821 specification. Spam bots are often poorly implemented, or try and use tricks to work around these conventions. Many spam bots can be found and blocked by detecting some of the common violations.

Custom spambot detection

Through our extensive, long-term experience running an email service, we've been able to closely analyse how spam bots try to deliver email to our servers. By using this data, we're able to pre-emptively detect spam bots soon after they connect, and block them from delivering their spam email at all.

Sending host rate limiting

Using a number of methods of detection, we limit the rate that external servers can send email to our servers. Again through our extensive, long-term experience running an email service, we've been able to tune these patterns to minimize any effect on legitimate sending services, while significantly limiting the ability of spammers to deliver spam emails.

Greylisting

Greylisting is a process designed to detect and stop spam being accepted from poorly written sending email servers, which almost all spam bots are. One of the main concerns users have with greylisting is that poor implementations will often delay all email. The Fastmail implementation uses a number of adjustments to ensure that legitimate email is delivered with no delay, while spam email is delayed or stopped from delivery entirely.

Greylisting works by returning a temporary failure response (4xx code) to the first attempt to deliver an email, but accepting it on the second attempt. Every proper email server will attempt to redeliver a message after a temporary failure response, while currently almost all spam software does not, thus effectively blocking the emails from the spam software.

To avoid delaying proper email servers, our greylisting implementation performs a number of checks on a connection before applying the greylisting policy:

  • We only greylist hosts that appear to be dialup/DSL hosts of some sort, or hosts that don't have any valid reverse DNS. This ensures that the vast majority of email servers are immediately not subject to greylisting, and their email is not delayed.
  • If a host has been greylisted, and it successfully passes greylisting twice in a 24 hour period (e.g. it correctly attempts to re-deliver a piece of email twice in 24 hours), then that host is whitelisted and should not be greylisted again for the next 24 hours. If it continues to deliver emails, each new delivery will extend the whitelist period. This means that any real email servers (a real email server will always retry) connected via dialup or DSL will quickly be whitelisted and not subject to email delays.

  • If a host opens an SMTP session with a HELO that is not an IP address and is not the same reverse DNS as its connecting IP, but the forward DNS of the name does resolve to the connecting IP, then that host is not subject to greylisting.

    For example: The machine at IP 206.223.169.73 connects to us. The reverse DNS for 206.223.169.73 is 206-223-169-73.beanfield.net, which looks like a common dialup/DSL IP name, and would be a candidate for greylisting. However, the machine advertises itself to us with a HELO mx3.hub.org line. Doing a forward lookup of mx3.hub.org gives the IP 206.223.169.73, which is the same as the connecting IP, so we exclude it from greylisting.

  • If the SMTP MAIL FROM address passed in the SMTP session is in the user's contact list, then the email is not greylisted.

When combined, these features provide an excellent balance of greylisting hosts which should not be sending emails, and allowing those hosts which should be sending emails to get their message straight through.

Additionally, to help with the tracking of any problems, once a message passes greylisting and is accepted, a new header is added called X-Spam-greylist. This header tells you how many seconds the email was delayed and whether that host has been whitelisted for 24 hours. (Technically, the delay figure actually indicates the length of the last delay for the IP/sender/recipient combination, so in the case of multiple emails from the same person, to the same person, from the same machine in a short time period, the figure will be unclear and hard to calculate.)

If you feel that email is being delayed by greylisting, please check the headers of your email for the X-Spam-greylist header first. If it's not present, then the email was not delayed by greylisting, and it was probably just held up on the sender's side, something beyond our control.

Forwarding and custom domains

The result of all these tests is that most spam is blocked before it even enters our systems. However, we can only run these tests when the email is delivered directly to us, not forwarded on from another mail server. If you use your own domain, we strongly encourage you to host your email directly with us rather than having it forwarded on from your old host, as this will enable us to be much more effective at filtering out spam.

Was this article helpful?
28 out of 44 found this helpful