Last Updated: Wed, 20 Nov 2013 21:45:18 +0000 GMT
SourceForge Logo
URL: http://qmail-scanner.sourceforge.net/

Qmail-Scanner: Content Scanner for Qmail

Copyright 2011 Jason Haar. This software is distributed under the terms of the GNU General Public License. See COPYING for additional information.

Description

Qmail-Scanner is an add-on that enables a Qmail email server to scan gatewayed email for certain characteristics (i.e. a content scanner). It is typically used for its anti-virus and anti-spam protection functions, in which case it is used in conjunction with external scanners. It also enables a site (at a server/site level) to create "Policy blocks": i.e. react to email that contains specific strings in particular headers, or particular attachment filenames or types (e.g. *.VBS attachments).

Qmail-Scanner is integrated into the mail server at a lower level than some other Unix-based virus scanners, resulting in more thorough coverage. It is capable of scanning not only locally sent/received email, but also email that crosses the server in a relay capacity. Qmail-Scanner also leverages the wealth of meta-information provided by Qmail (such as client IP address, and whether or not the client is allowed to relay).

Features

Download

The latest release is 2.11 (via http), and is kindly housed by SourceForge. GnuPG signature of qmail-scanner-2.11.tgz.asc is also available. Of course, you'll be needing my GPG Public Key to verify that.

Requirements

Patches

Qmail-Scanner relies on Bruce Guenter's QMAILQUEUE patch to enable qmail-1.03 to call a different qmail-queue program than the one compiled in by default. If you are using netqmail-1.05, then you already have the patch.

Qmail-scanner's qmail-scanner-queue.pl perl script is used instead of Qmail's qmail-queue binary. After qmail-scanner-queue.pl has run, it calls the original qmail-queue binary to resubmit the message back into the system.

Supported Virus Scanners

The following virus scanners are known to work with qmail-scanner. Remember that only the current releases of scanners are supported. There is little point in running an old scanner - you miss too many viruses.

Other Unix-based scanners should be simple to add support for.

CHANGES

There is a separate page listing changes that have been made between releases

TODO

There is a separate TODO page.

FAQ

There is a separate FAQ page.

Performance/Resource Usage

Adding content/virus scanning to an email server will considerably add to the resource usage of that server. As this "wrapper" is written in perl instead of low-level C, quite a lot of memory and file opens/stats occurs just to get it going. Adding to this the actual scanners (i.e. SpamAssassin and AV) memory and CPU usage and it becomes quite complicated (certainly the debugging info shows that the scanner harness spends more time running the external scanners than it does doing things itself [that is to be expected as they do quite a lot of thinking...]).

As a "rule of thumb" I'd suggest you look at how many simultaneous SMTP sessions you are willing your box to have going at any one point in time. Each SMTP session can invoke up to 'n' different virus scanners (although they run one after the other - not simultaneously) and I'd estimate that leads to around 10-20Mb of memory usage per SMTP session. Thus if your dedicated SMTP host has 1024Mb RAM + 2048Mb swap - that should mean you can handle - well heaps ;-) The scanners cause the CPU to be thrashed while they're running, so I'm making sure for our site that our Qmail servers will only accept up to 40 incoming SMTP sessions at any one time - that way I know the box will handle it. As this leads to an increased memory usage, don't forget Qmail's memory limits will need to be increased to deal with it (set via ulimit or softlimit calls with Qmail system startup scripts).

One thing you should test for is what happens if connectivity between this server and another local SMTP server is down for any length of time (due to failure/power outage). When the link is restored, can your server handle the other trying to dump 1,000's of email msgs onto it at once? You need to use softlimit and tcpserver's limit options to ensure your box doesn't get killed. Note that this resource issue isn't "a bug" in Qmail-Scanner. The same thing will happen with a pure, untouched Qmail (or any other) system - it will just happen sooner...

After that scare-mongering I should say that I have tested Qmail-Scanner under ridiculously low resource conditions - and it reacts as it should - so at worst your system should start deferring email. Thankfully DJB's layering of programs is such that this is easy to accomplish :-)

Installation

At this stage qmail-smtpd will need to be "told" that Qmail knows to use qmail-scanner-queue.pl instead of qmail-queue. This is done via the tcpserver control files for SMTP. Look to see where tcpserver for qmail-smtpd gets its rules from - it's the file after the "-x" option (well, that's the CDB version actually - find the text file yourself! ;-). Edit that file and tell qmail-smtpd which IP address ranges (corresponds to SMTP client IP addresses) you want Qmail-Scanner to be invoked on - typically all of them.

#/etc/tcpserver/smtp.rules
#
# No Qmail-Scanner at all for mail from 127.0.0.1
127.:allow,RELAYCLIENT="",RBLSMTPD="",QMAILQUEUE="/var/qmail/bin/qmail-queue"
# Use Qmail-Scanner without SpamAssassin on any mail from the local network
# [it triggers SpamAssassin via the presence of the RELAYCLIENT var]
10.:allow,RELAYCLIENT="",RBLSMTPD="",QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl"
#
# Use Qmail-Scanner with SpamAssassin on any mail from the rest of the world
:allow,QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl"

The above example means from now on all SMTP mail will be scanned, but with different characteristics. Mail from the LAN (10. network) will be scanned by the supported virus scanners, whereas mail from the Internet will be scanned for virii AND tagged by SpamAssassin. This finer control allows you a lot of versatility, e.g. virus scanning only performed on mail coming from your Exchange server, and not from your Unix mailing-list servers.

You must increase the amount of memory your system allows qmail-smtpd to run with, as it it now running the entire perl interpreter PLUS virus scanners. Typical installs of Qmail have system rc/startup scripts (e.g. /etc/rc.d/init.d/qmail or /service/smtp/run) that limit the amount of RAM qmail-smtpd can use via ulimit or softlimit. You must increase that to around 20Mb (totally dependent on your OS and choice of anti-virus scanner). If you don't qmail-smtpd will crash with a "qq" error on the receipt of the very first message... The actual amount is dependent on the OS in question as well as the virus scanners being used, so be prepared to experiment a little. Whatever you do, don't just set it to something stupid like 100M "just to be sure". The whole point about limiting RAM usage is so that "unusual" mail messages (e.g. from spammers or hackers) can't cause your system to become unusable by making it run out of RAM. THIS IS A FEATURE OF QMAIL - NOT A BUG. Also note that as the pattern files for any given non-daemonized AV are upgraded, they catch more and more viruses - and as such need more RAM to load into. So your memory sizes will increase with time. This is a good reason to try to stick to daemonized AV products if you can - as the daemon's memory constraints don't impact Qmail-Scanner (i.e. you can reduce the amount of RAM Qmail-Scanner needs by exclusively using daemonized AV)

To scan all mail sent by local shell users, the QMAILQUEUE will also need to be defined within /etc/profile or the like so that when they send mail, it will be affected as well. Similarly, if you are running Webmail apps, that environment variable will need to be available from within the Web server for Qmail-Scanner to scan any emails sent.

Also, think twice before running Qmail-Scanner in front of any mailing-list servers. Do you really think it's a good idea to have 10,000 messages banging away at your anti-virus system at the same time? Either put your mailing-list servers beyond the reach of your Qmail-Scanner servers, or put the mailing-list on the Qmail-Scanner servers themselves - that way each message is only scanned once and the load issues disappear.

If "$DEBUG=1" (the default) is set within qmail-scanner-queue.pl, then every transaction will be logged to /var/spool/qscan/qmail-queue.log - so you'll see how it goes. Regardless of debugging, errors (and attachment info if enabled) should also be recorded in the qmail logs (probably via syslog) - just look for entries containing the string "X-Qmail-Scanner".

Any SMTP sessions that are dropped (due to network outages/etc) may lead to files lying around in /var/spool/qscan . Running /var/qmail/bin/qmail-scanner-queue.pl -z at least once daily will ensure such files are deleted when they're over 30 hours old - make a cronjob to do that (see contrib/ for a logrotate script). Also realize that /var/spool/qscan/qmail-queue.log will grow without bounds. At some stage turn debugging off ($DEBUG=0) and delete the logfile. Personally, I like the logfile, so I run a cronjob that just does "mv -f qmail-queue.log qmail-queue.log.1" at 3am every morning. That way logs don't grow without bound, but you still end up with the logs from the past two days. The file can be safely deleted at any time if it becomes a disk-hog, but unless "$DEBUG=0" is set, it'll just get re-created the next time a message comes through (again, the contrib/ logrotate script can be used to take care of this).

Qmail-Scanner contains an internal scanner which allows you to quarantine email based on attachment filenames and/or email headers. Read the minimal document on it for details.

Philosophy behind Quarantining...

When Qmail-Scanner decided to quarantine a message, it moves it into mail folders (maildir format) under /var/spool/qscan/quarantine/. They are split into three different maildir folders based on whether Qmail-Scanner thinks they are a virus, a policy block or high-scoring spam (if you have enabled that option). This means the message can be read in its pure "adulterated" state (e.g. still containing virii/etc) by maildir clients like mutt - or via IMAP (if maildir format supported - you'll have to work that out for yourself). At worse you can just read it with an editor - it's just a MIME text file...

If you want a good IMAP server that supports maildir natively - try Courier-IMAP.

I made the decision to write it into maildir format for performance and reliability reasons - and it expressly makes it difficult for any Windows admin to click on it with their vulnerable Windows mailer and read it :-) Qmail actually comes with a program called /var/qmail/bin/maildir2mbox which can do just that... (you could run it from cron to automatically suck all the new mail messages from /var/spool/qscan/quarantine/*/new/ into a mbox.)

Also note that Qmail-Scanner only quarantines. It doesn't "clean" messages.

Also this event is logged in /var/spool/qscan/quarantine.log in a tab-delimited format (for post-processing). See QSS for an example of one way of generating stats.

If Qmail-Scanner was configured with the "--log-details" option, then a one-line summary of every message processed is recorded either in mailstats.csv or via syslog. e.g:

Aug 14 16:22:41 srvname qmail-scanner[30802]: Clear:RC:1(1.2.3.4): 0.030769 11569 root@x.y jdoe@y.z More_Power! <20020814042234.27902.qmail@x.y> 1029298961.30804-0.srvname:10649 
Aug 14 16:23:17 srvname qmail-scanner[30820]: Clear:RC:0(1.2.3.4): 0.033618 2021  root@x.y jdoe@y.z Cron__run-parts_/etc/cron.daily <20020814042243.28092.qmail@x.y> 1029298997.30822-0.srvname:895 
Aug 14 16:23:17 srvname qmail-scanner[15885]: Clear:RC:0(1.5.4.3):SA:0(3.0/5.0):CR:PGP(old-signed): 4.66578 5549 fedora-devel-list-bounces@redhat.com Jason.Haar@trimble.co.nz Re:_RFC:_Soname_in_rpm_name <20050128003606.GD16634@neu.nirvana> 1106872589.15888-0.mailsrv2.trimble.co.nz:1046 1106872589.15888-1.mailsrv2.trimble.co.nz:189 1106872589.15888-2.mailsrv2.trimble.co.nz:120 orig-mailsrv2.trimble.co.nz110687258948815885:5549


The format is as follows:

Note: fields are normalized (i.e. converted from base64/Q-P back to glorious 8bit), space-delimited and limited to 1024 chars when syslog is used (with spaces within fields replaced by underscores), and tab-delimited in mailstats.cvs format

Support

This software is released under the GPL as found in the COPYING file enclosed.

This package is housed on SourceForge.

Any questions, suggestions, etc must be sent to the mailing-list set up to discuss this, subscribe via http://lists.sourceforge.net/mailman/listinfo/qmail-scanner-general , or subscribe to the announcements-only list via http://lists.sourceforge.net/mailman/listinfo/qmail-scanner-announce.

Last Updated: Wed, 20 Nov 2013 21:45:18 +0000 GMT