How spam filtering works at Fermilab

As with your postal mail, it’s challenging to eliminate all the junk in your e-mail. The people behind FermiMail have been modernizing the laboratory’s e-mail spam detection system to keep junk mail to a minimum.

Did you know that by some estimates, as many as nine out of 10 e-mail messages are spam? As part of the FermiMail implementation project, the e-mail team modernized the e-mail system’s approach to detecting spam. But, as with your postal junk mail, achieving an error-free filter to catch all spam is not possible, partly because one person’s junk mail is another’s valuable correspondence.

The current Fermilab spam filtering process relies on spam signatures – a collection of known attributes of previous spam e-mail messages – that are provided with near real-time signature updates. The process uses a weighing system that evaluates several characteristics of each e-mail to rank the likelihood that it is spam. Messages of a high enough weight are considered to almost certainly be spam and are discarded by the e-mail system. Messages of a sufficiently low weight are considered to almost certainly be legitimate and are delivered to your inbox. Messages in the middle range are considered suspect and are tagged according to their weights, but are still delivered to your inbox. All messages in your inbox have a weight assigned known as a spam confidence level, or SCL, which allows your email client (or mailbox filter rules) to take certain actions, such as filing the message in the junk mail folder, based on this weight.

This is how the e-mail filtering process is done on the e-mail servers. Each e-mail client, however, may have its own evaluation method for spam. For instance, Outlook clients have a built-in spam filter that looks not only at each individual message, but also at patterns, frequency of and similarities in groups of messages. This is why you may see changes in the way your client handles legitimate messages over time, causing messages not previously classified as spam to start being marked as such.

E-mail clients often have options for you to mark messages with certain characteristics as legitimate, a process known as “whitelisting.” For example, you might choose to indicate that all messages from a certain sender should be considered legitimate. These features help you control what is classified as spam in your mailbox with your specific e-mail client.

No spam filtering system is perfect – some spam will always get through. But the tools on the e-mail servers and in your e-mail client should prevent the vast majority from flooding your inbox.

Learn more

—Tom Ackenhausen