Getting the feet wet with DKIM

DKIM stands for DomainKeys Identified Mail, it’s an anti-phishing / anti-spoofing system for email that relies on private/public key system to authenticate the origin of the email. Kind of SPF records but on steroids. You can find a pretty concise explanation of the DKIM system here.

I needed to figure this out, as my SPAM (sorry let me just correct this), my outgoing SOLICITED bulk email sending box was getting lower deliverability rates. So it seems having this system helps, because as everybody knows all the lower grade spam senders can’t figure this out.

First thing is to create a private/public key pair. You can go all out with OPENSSL, but as I feel a bit lazy, and not in the mood for man pages and the macho man CLI black magic, just used the online tool that the good people of Worx (the PHPMailer people) provide here.

You must provide the filenames for the pair, just click next, and in the next step fill the domain (pretty self explanatory), the identity (just left blank), the selector, and a passphrase (I left that blank also). The selector is mandatory, as the DKIM system allows to have several keys per domain, it uses the selector to identify which key the email was signed with. Even if you are using just one key per domain you need to assign a value here (I used “default”, but it can be anything like “potato” or “balls”.

You will get a zip file with everything you need, and even some instructions to protect the key files with .htaccess directives, because as the good people of Worx know, PHPMailer is PHP, and all PHP programmers are dummies, so they will happily put the key files under the domain website root and probably with a couple of links from the website homepage.

I bet that somewhere in the support forums somebody asked: why I can’t access my key files, always getting those 403 Forbidden errors.

First thing is to publish your public key on DNS. I use DJB tinydns DNS server with VegaDNS management, the old PHP version when the author placed key files in the public website root. Now he is smart, the 2.0 version is in Python and he doesn’t do this anymore… anyway add a new TXT record for the domain, in the hostname put “selector._domainkey” (change the selector part to the string you defined before), and in the address field put in one line only:

v=DKIM1; k=rsa; p=publickey

(replace publickey with the public key that you get previously). If you are using other DNS server check out the specific documentation on creating TXT records.

Now, to sign the email messages you send. Obviously the natural choice is to have your MTA auto magically sign the outgoing emails. To me that means to mess with my power beast Qmail production installation, this means messing with the macho man stuff, patches, libraries and compilers. Not so good to make a quick fix for sending SPAM (sorry, SOLICITED bulk email). So I went the easy route and use the PHPMailer to sign the outgoing emails.

Just have to add a couple of lines:

$email = new PHPMailer();
$email->DKIM_domain = 'domain.com'; // the domain
$email->DKIM_private = '/path/to/private/key'; // please even using PHP keep the file above the webroot
$email->DKIM_selector = 'default'; // the selector you published in DNS

and proceed to testing. There are plenty of online services that provide us this service. I liked this one. But after doing some testing I quickly realized that something was not well in the DKIM land…

All the outgoing emails were sent with some bad header char. Even sending to my own system, the Amavisd was quarantining the emails with virus suspicions due to a bad header. Further looking into this, just realized that between the last header line and the DKIM signature there was a dreadful ^M (a carriage return, the DOS \n\r not compliant with the email spec). This is in a plain vanilla Qmail running on Debian, and with latest stable PHPMailer.

There was just one thing to do, bring out the old quick and fix hammer and change the class.phpmailer.php file. If you have the same problem in line 1305 change the CRLF to just LF:

/*
$this->MIMEHeader = rtrim($this->MIMEHeader, "\r\n ") . self::CRLF .
    str_replace("\r\n", "\n", $header_dkim) . self::CRLF;
*/

$this->MIMEHeader = rtrim($this->MIMEHeader, "\r\n ") . "\n" .
    str_replace("\r\n", "\n", $header_dkim) . self::CRLF;

And everything is running nice and well.

Procmail with Qmail + Vpopmail

Following the qmail threads in this blog, and after a successful experience filtering emails in the server with a php script! time was upon to thinker a bit with the elder of all email server filters, Procmail – the mail processing utility for Unix.

Whe are talking really old stuff, as Wikipedia states the initial release in 1990, so 26 years from this writing, and about a zillion years in computer time (a date so old that it’s closer to the Unix Epoch than it is of today).

As usual, the easy part in FreeBSD is the installation:

cd /usr/ports/mail/procmail
make install

there. Now the tricky part, that is to make it play nicely with Qmail+Vpopmail setup. For the first experiences you probably should setup a couple of test accounts.

The concept is pretty simple, for an account that you want to filter email with Procmail we are going to add/or change the .qmail file that controls the email delivery to a filter script that invokes procmail and throw back a proper qmail exit code according to Procmail result.

It is very important to take into account that in this setup Procmail DOES NOT deliver the email directly, it filters the email, and according with the recipes rules, it can stop the delivery chain, forward the email, invoke an external command, etc.

Without further delays. The customized .qmail that calls the filter script:

| /home/vpopmail/domains/mydomain.com/teste/procmail_filter
/usr/home/vpopmail/domains/mydomain.com/teste/Maildir/

this is pretty simple and standard stuff. The qmail-local parses the .qmail file. The line with a pipe means to feed the message to the specified program. The command is invoked by qmail-command that runs sh -c command in the home directory, makes the email messsage available on standard input and setups some environment variables.

For this thread the most important stuff are the exit codes:

Command’s exit codes are interpreted as follows:
0 means that the delivery was successful;
99 means that the delivery was successful, but that qmail-local should ignore all further delivery instructions;
100 means that the delivery failed permanently (hard error);
111 means that the delivery failed but should be tried again in a little while (soft error).
Currently 64, 65, 70, 76, 77, 78, and 112 are considered hard errors, and all other codes are considered soft errors, but command should avoid relying on this.

With this info it’s pretty straight forward to devise the procmail_filter script.

#!/bin/sh

/var/qmail/bin/preline /usr/local/bin/procmail ./.procmailrc

status=$?

if [ $status -eq 99 ]
then
  status=99
elif [ $status -eq 0 ]
then
  status=0
elif [ $status -le 77 ]
then
  status=111
else
  status=100
fi

exit $status

and mark it executable.

The first line (after the shebang) the script calls qmail preline program, it simply inserts at the top of each message a UUCP-style From_ line, a Return-Path line, and a Delivered-To line because Procmail doesn’t understand the qmail environment variables. Calls procmail and sets Procmail configuration file the .procmailrc in the same directory.

Now for the .procmailrc stripped down to a very simple example:

SHELL = /bin/sh
LOGFILE=./pm.log
LOG="
"
VERBOSE=yes

# Exitcodes
# 0 normal delivery
# 99 silent discard a message
# 100 bounce a message

# Recipes

:0
* ^From: email@domain.com
{
  EXITCODE=99

  :0
  | /usr/local/libexec/dovecot/deliver -d delivertothis@email.com

}

# Avoid duplicates
:0
/dev/null

The top lines are the configuration variables, I would strongly suggest to use a log file for testing, in this example called pm.log that lives in the same directory. The strange LOG directive simple adds a new line to each log entry.

Then the recipes, in this example we match all emails from email@domain.com and deliver it to a local email delivertothis@email.com using Dovecot deliver command (so it takes care of Maildir quota), and set exitcode 99 to discard the message. Exit code 99 means that the delivery was successful, so all .qmail further delivery instructions will be ignored.

We could simply put an email address for a external email address, or even a local address but with less efficiency as it will trigger a new delivery process. This is auto-magical thanks to FreeBSD mailer.conf wrapper.

:0
* ^From: email@domain.com
{
  EXITCODE=99

  :0
  delivertothis@email.com

}

The last lines avoid duplicates, Procmail /dev/nulls the message and gives back the delivery control to the .qmail flow.

That’s it, everything playing nice with each other in old good UNIX tradition.

My Qmail installation guide reloaded

Qmail ReloadedA couple of years ago I posted my Qmail installation guide, and has expected it served me good when was time to reform to the old mail server. But, i made some changes on this iteration and i think is more polished and shiny than ever.

Again, this is to my own reference, but i will be very glad if it also can help someone. On the other hand, if you follow it, and nukes your system or kills every life form on Earth please don’t blame me. You are warned.

The old picture:

mail-system


2 Qmail instances, 1 published MX record that accepts emails from other MTAs, does the RBL checks and forwards the passed emails to the main Qmail instance via artificial smtproutes. The forwarded emails are then checked against virus (by Clamav) and spam (by SpamAssassin) trough qmail-scanner qmail-queue drop in replacement.

Users receive and send email trough the non published MX Qmail instance. They need to smtp-auth to relay email (send email to remote domains). Delivery to local domains doesn’t require smtp-auth.

Identified problems:

1 – One problem is that the main Qmail instance (that has no published MX records), that works with Vpopmail and holds all the accounts information, maildirs and email is somehow vulnerable:

The main weakness of this installation, is that if a clever spammer discovers that mail.domain.com accepts incoming emails for local domains, he can spam down your users bypassing the rbl tests.

also, one has to rememeber that has SPF and A records published, and it’s IP is printed on all outgoing email headers, so it’s not anonymous.

2 – The user debug is somewhat tricky, if there is a smtp-auth client configuration problem. The problem is that the user will be able to send emails to local domains, but will get the dreadful 553 sorry that domain isn’t in my list of allowed rcpthosts (#5.7.1) error.

3 – Qmail-scanner, is a very neat piece of software, but it is fundamentally flawed performance wise because for each and every email it must load the PERL interpreter.

4 – Restarting Qmail every 15m to recognize new or deleted domains is plain dumb.

The new picture:

mail-system-v2


All of the previous mentioned issues have been addressed and polished. The main Qmail instance (mail) will only accept outside authenticated connections for both local and remote deliveries. The external email comes trough the published mx record Qmail instance only, filtered by rbl, then routed to Amavis for virus and spam scans, and finally routed to the main Qmail instance (if virus and spam free). In this scenario you must trust your customers, because as they authenticate and send emails, these will bypass all the virus and spam checks.

Let’s put our hands to work, the first slice is on point 15 of the original guide “Clam Anti Virus, Spam Assassin and Qmail-scanner”, this version will move the virus and spam filter to the other Qmail instance. So follow the original guide until point 15, and then:

1 – Install qfilter

cd /usr/ports/mail/qmail-qfilter/
make install clean

2 – Make a shell script wrapper that will invoke the filters used by qfilter

mkdir -p /var/qmail/qfilter
edit /var/qmail/qfilter/qfilter-wrapper

and put these contents on the file

#!/bin/sh
exec /usr/local/bin/qmail-qfilter /var/qmail/qfilter/smtp-auth-only

save and mark it executable

chmod +x /var/qmail/qfilter/smtp-auth-only

Note:
actually there is only one filter being invoked (smtp-auth-only), but qfilter supports several filters (exec /usr/local/bin/qmail-qfilter /path/to/filter-one –/path/to/filter-two –/path/to/filter-three)

3 – Install the smtp-auth-only filter

This is just a very simple perl script that will test the presence of the environment variable TCPREMOTEINFO, as this variable is only set upon successful smtp-auth. If the mail comes from an authenticated user the script returns 0, else if it’s from a non-authenticated user the script returns 31 signaling a permanent error.

edit /var/qmail/qfilter/smtp-auth-only

the script is very simple

#!/usr/local/bin/perl

if (defined $ENV{'TCPREMOTEINFO'} == false) {
        use Sys::Syslog qw(:DEFAULT :standard);
        openlog("qfilter", 'ndelay,pid', 'mail');
        syslog('info', "No SMTP-Auth - Rejecting Email");
        exit 31;
}

exit 0;

save it and mark it executable

chmod +x /var/qmail/qfilter/smtp-auth-only

4 – Adjust /etc/tcp.smtp to use qfilter

this is my last line now of /etc/tcp.smtp

:allow,MAXLOAD="2000",SPFBEHAVIOR="0",RBLSMTPD="",QMAILQUEUE="/var/qmail/qfilter/qfilter-wrapper"

it accepts connections from everywhere (if cpu load > 20 rejects connections) it bypasses SPF and RBL checks, and it uses qfilter-wrapper as qmailqueue. After

qmailctl cdb

to build the new smtp tcp rules cdb file and reload qmail, the main Qmail instance will only accept authenticated user email. Email routed from mx should match a previous /etc/tcp.smtp rule.

5 – Install Clam Anti Virus, Spam Assassin and Amavis

This step kind of mimics the step 15 on the original guide, with two main differences. We are installing all the filtering software on the MX instance of Qmail. And qmail-scanner as been replaced by Amavis.

So, log in to the MX console and install the software.

cd /usr/ports/security/clamav
make install clean

options selected: ARC, ARJ, DMG_XAR, DOCS, ICONV, LHA, LLVM, TESTS, UNRAR, UNZOO

cd /usr/ports/mail/spamassassin
make install clean

options selected: AS_ROOT, GNUPG, UPDATE_AND_COMPILE, DCC, DKIM, PYZOR, RAZOR, RELAY_COUNTRY

cd /usr/ports/security/amavisd-new
make install clean

options selected: ALTERMIME, ARC, ARJ, BDB, CABS, DOCS, FILE, FREEZE, LHA, LZOP, MSWORD, MYSQL, P7ZIP, RAR, RPM, SPAMASSASSIN, TNEF, UNARJ, ZOO

as always on FreeBSD the installation is easy and a breeze. Now the fun part, configuring and make all this work together…

6 – Configure Clam Anti Virus

First ClamAV and FreshClam (the anti-virus updater daemon). Here’s a comment striped out of /usr/local/etc/clamd.conf

LogSyslog yes
LogFacility LOG_MAIL
LogVerbose yes
ExtendedDetectionInfo yes
PidFile /var/run/clamav/clamd.pid
DatabaseDirectory /var/db/clamav
LocalSocket /var/run/clamav/clamd.sock
FixStaleSocket yes
ReadTimeout 300
CommandReadTimeout 5
User vscan
AllowSupplementaryGroups yes
ScanMail yes

and the comment stripped version of /usr/local/etc/freshclam.conf

DatabaseDirectory /var/db/clamav
LogVerbose yes
LogSyslog yes
LogFacility LOG_MAIL
PidFile /var/run/clamav/freshclam.pid
DatabaseOwner vscan
AllowSupplementaryGroups yes
DatabaseMirror database.clamav.net
NotifyClamd /usr/local/etc/clamd.conf

There are few modifications to the distribution configuration files, mainly 2 things, to run clamd/freshclam daemons as the user ‘vscan’, the same user that will run amavis, and to log via syslog mail facility.

It makes perfect sense to take advantage of syslog and newsyslog automatic maintenance and log rotation. Also, having most of stuff logging to /var/log/mail makes it easy to spot any error message outputted by any of the several components. The downsize, is that in a busy server the log can become a bit messy.

Adjust the ownership on ClamAV directories:

chown -R vscan:vscan /var/db/clamav
chown -R vscan:vscan /var/run/clamav

add the rcvars to /etc/rc.conf
clamav_clamd_enable=”YES”
clamav_freshclam_enable=”YES”

and start both of the daemons

/usr/local/etc/rc.d/clamav-clamd start
/usr/local/etc/rc.d/clamav-freshclam start

7 – Configure Spamassassin

First, as Spamassassin uses the GeoIP database, you should have an updated database on /usr/local/share/GeoIP/GeoIP.dat, to do so automaticaly write this file on /usr/local/etc/periodic/daily/updategeoip

#!/bin/sh

cd /usr/local/share/GeoIP
/usr/local/bin/wget -q http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
gzip -d -f GeoIP.dat.gz

exit 0

and mark it executable

chow +x /usr/local/etc/periodic/daily/updategeoip

and run it manually (you should have installed on your system /usr/ports/ftp/wget)

/usr/local/etc/periodic/daily/updategeoip

update and compile Spamassassin rules

sa-update
sa-compile

and make this process automatic, edit /usr/local/etc/periodic/weekly/spamassassin

#! /bin/sh

/usr/local/bin/sa-update && /usr/local/bin/sa-compile

exit 0

mark it executable, and run by hand the first time

chmod +x /usr/local/etc/periodic/weekly/spamassassin

Spamassassin doesn’t need so much configuration, and it pretty much works out of the box, but i made some fine tuning to everything play happy, so there it is the commented striped version of /usr/local/etc/mail/spamassassin/local.cf

use_dcc 1
dcc_home /var/dcc
dcc_path /usr/local/bin/dccproc
dcc_timeout     10
add_header all  DCC _DCCB_: _DCCR_
use_pyzor 1
pyzor_path /usr/local/bin/pyzor
use_razor2 1
razor_config /var/amavis/.razor/razor-agent.conf
score RAZOR2_CHECK 2.500
score PYZOR_CHECK 2.500
score DCC_CHECK 4.000

create the /var/amavis/.razor directory, and set up razor

mkdir /var/amavis/.razor
razor-admin -home=/var/amavis/.razor -create
razor-admin -home=/var/amavis/.razor -discover

and change ownership to the vscan user

chown -R vscan:vscan /var/amavis/.razor

time to set up the rc vars at /etc/rc.conf and start Spamassassin (replace aaa.bbb.ccc.ddd for the allowed IP address to connect)

spamd_enable="YES"
spamd_flags="-A 127.0.0.1,aaa.bbb.ccc.ddd"

and start it

/usr/local/etc/rc.d/sa-spamd start

8 – Configure Amavis

Amavis will be the glue between Qmail and ClamAV and Spamassassin in a dual MTA setup. It will accept routed emails from Qmail (mx instance) on port 10024, fiter, and re-route to the main Qmail for local delivery (email instance).

Here it is the /usr/local/etc/amavisd.conf configuration file. Now some customizations required to amavis work properly:

  • set $mydomain and $myhostname to your host fqdn
  • configure $forward_method = ‘smtp:[aaa.bbb.ccc.ddd]:25’; Set aaa.bbb.ccc.ddd to the IP address of the main Qmail instance (email host) to where the filtered emails are forward. Remember that in the Qmail instance you need a corresponding entry in /etc/tcp.smtp that accepts the forward emails and skips SPF and RBL checks (replace aaa.bbb.ccc.ddd for the incoming IP of Amavis):
    aaa.bbb.ccc.ddd:allow,MAXLOAD=”2000″,SPFBEHAVIOR=”0″,RBLSMTPD=””,QMAILQUEUE=”/var/qmail/bin/qmail-queue”
  • the $max_servers should, as commented, match the width of your MTA pipe /var/qmail/control/concurrencylocal
  • @local_domains_maps = [‘.’]; we accept every incoming email as a local domain email, because by configuration the mx Qmail instance will only accept and forward to Amavis emails to local domains
  • customize $inet_socket_bind, generally the loopback address IP should be fine, but if your are running inside a jail (and if you are following this guide you are) replace the loopback IP for the main jail IP
  • setup @inet_acl list (this is space delimited list of IPs that Amavis will accept email from). If you are running everything in the same jail (Qmail mx instance and Amavis) this is the main jail IP, if Qmail mx is running in other jail or host add the mx Qmail outgoing IP

These are some of the most important things that you should consider to setup Amavis to your own taste, and as pretty neat software everything (or just about everything) is customizable. The configuration file has extensive comments so it’s easy to understand each and every option:

  • In this setup Amavis is logging to syslog mail facility, $DO_SYSLOG = 1; and $SYSLOG_LEVEL = ‘mail.info’; scroll up to find out why. You can change this to a $LOGFILE. Also setup the $log_level
  • Virus, banned and spam (after $sa_kill_level_deflt threshold) emails are plain discarded, bounce only in case of bad headers.
    $final_virus_destiny = D_DISCARD; # (defaults to D_DISCARD)
    $final_banned_destiny = D_DISCARD; # (defaults to D_BOUNCE)
    $final_spam_destiny = D_DISCARD; # (defaults to D_BOUNCE)
    $final_bad_header_destiny = D_BOUNCE; # (defaults to D_PASS), D_BOUNCE suggested
  • you can customize $virus_admin and $spam_admin with a email address to receive reports when virus/spam email is detected, in this case you should also configure the from addresses in $mailfrom_notify_admin, $mailfrom_notify_recip, $mailfrom_notify_spamadmin
  • this configuration example does not notify me of positives, but i keep them in a quarantine dir, so i can do postmortem analysis and recovery,
    $QUARANTINEDIR = ‘/var/virusmails’;
    # Separate quarantine subdirectories virus, spam, banned and badh within
    # the directory $QUARANTINEDIR may be specified by the following settings
    # (the subdirectories need to exist – must be created manually):
    $virus_quarantine_method = ‘local:virus/virus-%i-%n’;
    $spam_quarantine_method = ‘local:spam/spam-%b-%i-%n’;
    $banned_files_quarantine_method = ‘local:banned/banned-%i-%n’;
    $bad_header_quarantine_method = ‘local:badh/badh-%i-%n’;
  • you can also customize the spam score required to each action
    $sa_tag_level_deflt = undef; # always add spam info headers
    $sa_tag2_level_deflt = 5.0; # subject will be re-written with $sa_spam_subject_tag value
    $sa_kill_level_deflt = 10; # email will not be delivered, and we keep a copy in quarantine
    $sa_dsn_cutoff_level = 15; # Since we are using D_DISCARD, this setting will serve no purpose, but if you were using D_BOUNCE, you can use this to set a level at which the sender will no longer be notified

and many more options that you can/should look into. If you are going to quarantine emails you should create the quarantine directories:

mkdir -p /var/virusmails/badh/
mkdir -p /var/virusmails/banned/
mkdir -p /var/virusmails/spam/
mkdir -p /var/virusmails/virus/

chown -R vscan:vscan /var/virusmails

also, it’s not a bad idea to put a line in root cron to delete older (30 days older) quarantined emails:

crontab -e

05 05 * * * /usr/bin/find /var/virusmails/* -type f -mtime +30 -exec /bin/rm -f {} \;

Finally! add the rc var at /etc/rc.conf

amavisd_enable="YES"

and start it

/usr/local/etc/rc.d/amavisd start

9 – Configure Qmail to use Amavis

Just a simple php script run every 10 minutes by cron will take care of this. As a bonus when you add, rename or delete a domain the Qmail mx instance will pick up the changes.

Edit /var/qmail/control/make_smtp_routes and adjust aaa.bbb.ccc.dd with the Amavis listening IP:port ($inet_socket_bind in amavisd.conf):

#! /usr/local/bin/php
<?php

$smtp_route = 'aaa.bbb.ccc.ddd:10024';

$rcpthosts     = file('/var/qmail/control/rcpthosts');
$morercpthosts = file('/var/qmail/control/morercpthosts');

$hosts = array_merge($rcpthosts, $morercpthosts);
$hosts = array_filter($hosts);

$fp = fopen("/var/qmail/control/smtproutes.tmp", "w");
foreach ($hosts as $host)
    fwrite($fp, trim($host).":".$smtp_route."\n");
fclose($fp);

if (md5_file('/var/qmail/control/smtproutes.tmp') == md5_file('/var/qmail/control/smtproutes')) {
    unlink('/var/qmail/control/smtproutes.tmp');
    exit(0);
}

openlog('PHP', LOG_ODELAY|LOG_PID, LOG_MAIL);
syslog(LOG_INFO, "New /var/qmail/control/smtproutes");

rename("/var/qmail/control/smtproutes.tmp", "/var/qmail/control/smtproutes");

syslog(LOG_INFO, "Restarting Qmail");
exec('/root/bin/qmailctl restart');

exit(0);

?>

mark it executable

chown +x /var/qmail/control/make_smtp_routes

and add it to cron

cron -e
*/10 * * * * /var/qmail/control/make_smtp_routes > /dev/null 2>&1

That’s it, this is the end. Now go grab a well deserved beer and behold your brand new system.

Final toughts

The system is cool, addressed the issues of the old system and is maintenance free. But, there is some space to improvements:
– develop an API (work in progress) that allows for administration, domain management and email management of the system. With this piece in place is then easy to integrate and develop admin and control panels that replace the outdated qmailadmin panel and administrative tasks on the command line.
– related with the API, to give domain managers the possibility to fine tune per domain anti-virus, spam, quarantine and notification settings. This also implies a deeper knowledge of Amavis configuration.
– to compile a complete and comprehensive guide that incorporates the original guide and the stuff on this one.

FIN and CLOSED 🙂

Qmail/Vpopmail renaming a domain

qmailIn a Qmail/Vpopmail email server system, there is no direct way to rename a domain. Let’s say, you have a costumer that is using domain.net and then registers domain.com to replace domain.net. The obvious solution is to make domain.com an alias domain of domain.net, it’s just one simple command:

vaddaliasdomain domain.net domain.com

But if the old domain is to be dropped completely there is no logical reason to go for the domain alias route, except that is the quick and dirty solution. Also, it’s a bit stupid and confusing to be consuming all the email services (POP, IMAP, SMTP, Webmail, Control Panel) with credentials of the old domain when the domain in use is the new domain. All this hassle just because Vpopmail doesn’t provide a command do rename a domain.

Even with no direct command, it’s possible to rename a domain, keeping all the mailboxes (user@old-domain.com changes to user@new-domain.com), passwords, forwards, email alias, webmail preferences and contacts etc. It’s a bit painful, but doable:

  • move vpomail/domains/old_domain to vpomail/domains/new_domain
  • adjust if needed vpomail/domains/new_domain/.qmail-default
  • adjust if needed each mailbox .qmail file in vpomail/domains/new_domain/users(1,2,3)/.qmail
  • rename domain in rcpthosts or morercpthosts (if in morercpthosts run qmail-newmrh)
  • rename domain in qmail/users/assign and run qmail-newu
  • log in to mysql and in the vpopmail database rename the old domain table to new domain table (domain dots get converted to underscores)
  • in the new renamed table update the pw_dir field to match the new domain
  • update dir_control table to the new domain
  • update limits table
  • update valias (domain and valias_line field)
  • in the roundcube database update the users and identities tables to match the new domain in the respective old domain entries
  • restart qmail and test

If you have read all the way trough here, and are thinking in doing this procedure, WAIT i have a special treat, a script that takes care of all. It’s a PHP script, that should be executed in command line by root and accepts 2 arguments, the old domain and new domain. Just adjust the PHP path, the configurations (paths to stuff in the system), mark as executable and you are ready to go.

Beware, you will be running this as root, and i assure that it works fine on my server but i can’t assure that it won’t destroy yours. If you accept this terms please proceed and grab the script.

Continue reading “Qmail/Vpopmail renaming a domain”

.qmail delivery filtering by adress and subject

My goal was simple, to filter incoming emails by subject and to address (on a catch-all address…) in the server and move the matches to a specific folder instead of the normal delivery. I’m running a Qmail+Vpopmail system, and these directions should be valid for similar setups.

Probably i could do this with Procmail or Maildrop, but it seemed all so complicated to do just a simple one time task that i opted in for the fun route, doing it myself… for a recurrent email filtering task, multiple accounts, customization, any kind of heavy email filtering i strongly suggest to stop here and go read about Procmail or Maildrop.

Still here? Good. First, add a .qmail (see dot.qmail man) in the user Maildir folder that you want to setup a filter:

| /home/vpopmail/domains/domain.com/user/filter
/usr/home/vpopmail/domains/domain/user/Maildir/

Save it and take care with the permissions, vpopmail user should be able to read it. What we are doing here is really simple, in the first line we pipe incoming emails trough filter, and according to filter exit code qmail execute (or not) the second line and proceed with the normal delivery.

Now setup the filter itself, it’s written in dirty and messy PHP but it gets the job done. Also it depends on PEAR Mail_mimeDecode, so go ahed and install it:

pear install Mail_mimeDecode

Now the filter script itself, it must be customized to your needs:

#! /usr/local/bin/php
<?php

/*
 * QMAIL FILTER EMAILS
 *
 * invoked by .qmail files
 * | /path/to/this/script
 *
 * parses email to address and subject against target strings
 * if BOTH are matched email is saved in $save_matched_dir
 * and qmail is instructed to ignore further .qmail lines
 *
 * CONFIG:
 */

$max_bytes        = 262144;            // mail size > 256Kb is not scanned
$to_address       = 'to@domain.com';   // to address filter
$subjects         = array('subject 1',
                          'match this subject',
                          'other subject');
$save_matched_dir = '/home/vpopmail/domains/domain.com/save-matched-emails/';

/*
 * QMAIL EXIT CODES
 *
 * 0 - Success (go to next .qmail line)
 * 99 - Success and abort (do not execute next lines)
 * 100 - permanent error (bounce)
 * 111 - soft error (retry later)
 */

try {
  function decodeHeader($input) {
    // Remove white space between encoded-words
    $input = preg_replace('/(=\?[^?]+\?(q|b)\?[^?]*\?=)(\s)+=\?/i', '\1=?',
                         $input);

    // For each encoded-word...
    while (preg_match('/(=\?([^?]+)\?(q|b)\?([^?]*)\?=)/i', $input, $matches)) {

      $encoded  = $matches[1];
      $charset  = $matches[2];
      $encoding = $matches[3];
      $text     = $matches[4];
      
      switch (strtolower($encoding)) {
        case 'b':
          $text = base64_decode($text);
          break;

        case 'q':
          $text = str_replace('_', ' ', $text);
          preg_match_all('/=([a-f0-9]{2})/i', $text, $matches);
          foreach($matches[1] as $value)
            $text = str_replace('='.$value, chr(hexdec($value)), $text);
          break;
      }

      $input = str_replace($encoded, $text, $input);
    }

    if (! isset($charset))
      $charset = 'ASCII';

    $input = strtolower(
               preg_replace('/[^a-z ]/i', 
                            '', 
                 iconv($charset, 
                       'ASCII//TRANSLIT//IGNORE', 
                       $input)
               )
             );
    return $input;
  }

  $mail  = '';
  $bytes = 0;

  $fr = fopen("php://stdin", "r");
  while (!feof($fr)) {
    $mail .= fread($fr, 1024);
    $bytes += 1024;
    if ($bytes > $max_bytes) {
      fclose($fr);
      exit(0);
    }
  }
  fclose($fr);

  require_once 'Mail/mimeDecode.php';
  $decoder   = new Mail_mimeDecode($mail);
  $structure = $decoder->decode(array('decode_headers' => true));

  // check from address
  $patt = '/[a-z0-9]+([_\\.-][a-z0-9]+)*@([a-z0-9]+([\.-][a-z0-9]+)*)+\\.[a-z]{2,}/i';
  preg_match($patt, 
             $structure->headers['to'], $matches);
  if (isset($matches[0]) && $matches[0] == $to_address) {
    // check subject
    $structure->headers['subject'] = decodeHeader($structure->headers['subject']);

    foreach ($subjects as $subject) {
       if (strpos($structure->headers['subject'], $subject) !== false) {
         $fw = fopen($save_matched_dir.
                     time().
                     '.'.
                     rand(1000, 99999).
                     '.'.
                     gethostname().
                     'S='.
                     strlen($mail).
                     ':2', 'w');
         fwrite($fw, $mail);
         fclose($fw);
         exit(99);
         // exit(0);
       }
    }
  }
} catch (Exception $e) {
}


// default, continue normal processing
exit(0);

?>

save, add the php hash bang, mark it executable and vpopmail owned. Also, the $save_matched_dir is not created and should already be present in your system.

Thats it. After this setup you should start seeing a steady flow of matched emails being saved in the matched directory and not delivered in your Inbox. As usually this works like a charm to me but can work incredible bad for you, so use at your own peril.