Jumat, 11 Oktober 2013

How to Archive and Backup Emails with Postfix and Dovecot Subfolders on Ubuntu Servers

I've been thinking recently that as well as an off-site compressed backup of all of the emails within my system, I would also like a (relatively) easy way to recover an email that one of my users has accidentally deleted from their their trash folder (i.e. it's gone forever).

So how do we archive and backup emails with Dovecot? It's actually quite simple, and if I say so myself.. clever :-)

Step One:

Blind Carbon Copy, always_bcc

Ok, so first off, create a new user for your main domain. I used "archive@domain.com" for simplicity. You then need to configure postfix to always blind carbon copy every email sent and received to that email address...

Copy and paste this line into your terminal command line (substitute your domain in)


sudo -i
cp /etc/postfix/main.cf /etc/postfix/main.cf.bak
sudo echo "always_bcc = archive@domain.com" >> /etc/postfix/main.cf

What are we doing here? Well, first off we are becoming the super user, backing up our main.cf configuration file for safety and then we are echoing into the file /etc/postfix/main.cf the always_bcc variable. Notice the use of TWO pointy brackets, this appends the line to the file, if you use one, you will replace the contents, so make sure you don't do that here :)

Ok, so with that done, if you log into Roundcube or with your email client to check the archive@domain.com account, you will now start receiving every email that goes through your system, in and outgoing.

Step Two:

Adding Dovecot Maildir Subfolders from Command Line

That's a great start, but a year down the line, this is going to be very unorganised. Yes, we could sort by date, but if you have a large system or lots of active users, that isn't going to be particularly realistic. What we want to do is create Dovecot subfolders for each domain.

Now this is where we need to think a little bit. We cannot simply take the email address "to" field and sort by that, because what happens if there is more than one email address? No, we need to use the sender's email address, as there can only ever be one of those.

We will grab the sender's domain and check to see if we have a Dovecot subfolder for that domain. If we do, move the email in to that subfolder, if not, we will first create the folder and then move the email.

Two things for consideration are, we need to not only create the correct subfolder, but we also have to automatically subscribe archive@domain.com to the new folder so that when you check the emails, the new subfolders are automatically added for us.

So, here is the code that will do all this for you, that can be run however often you want (I do it once daily with cron, leaving it any longer could mean it using a lot of resources whilst everything is sorted.



STORE=/var/vmail/vmail1/domain.com/a/r/c/archive-2013.10.11.16.04.29/Maildir/cur
KEEP=/var/vmail/vmail1/domain.com/a/r/c/archive-2013.10.11.16.04.29/Maildir


for x in `find $STORE -type f`
do
echo "--==~~==--"
RSLT=`cat $x | grep "Return-Path"`
PERSONTMP=`echo $RSLT | cut -f 2 -d "<"`
PERSON=`echo $PERSONTMP | cut -f 1 -d ">"`
echo "..get email $PERSON"
NAMETMP=`echo $PERSON | cut -f 2 -d "@"`
NAME=`echo $NAMETMP | tr '.' '_'`
echo "..senders domain is $NAME"
if [ -d $KEEP/.INBOX.$NAME ]
then
echo "..archive subfolder already exists.."
else
echo "..archive folder does not exist .. we will create it.."
mkdir -p $KEEP/.INBOX.$NAME/cur
echo `chown vmail:vmail -R $KEEP/.INBOX.$NAME`
echo `chmod 0700 $KEEP/.INBOX.$NAME`
echo "..adding .INBOX.$NAME to subscriptions"
echo "INBOX.$NAME" >> $KEEP/subscriptions
fi
echo "..Moving email.."
mv -uv $x $KEEP/.INBOX.$NAME/cur
done


Notes:

  • You will need to first check through your vmail directory to find the correct path for your archive email as Maildir uses dates, time and categorisation in the path so yours will be different.
  • Notice that the actual subfolders your emails are stored in is .INBOX.NAME/cur.
  • For the sake of tidyness, we are replacing any periods (.) in the domains with underscores (_) such as gmail.com -> gmail_com This is because the way MailDir folders work, is that rather than directories inside directories, subfolders are denoted by periods. SO with some email addresses, you would end up with two or three subdirectories before you get to the actual emails. For example, twitter's email is twitter@bounce.twitter.com so the folder tree would be bounce/twitter/com/ which is annoying and untidy for navigation through in Email clients, our way, we just have one folder for each domain.

That's it! Now, when someone rings you up worried that they have deleted an important email, all you need to know is the domain it was sent from, and to make your life easier, the approximate date. Log in to your archive email account and find the domain folder.

That's it for another entry, please remember to click an advert if I have helped you :)

Minggu, 15 September 2013

Amavis Mark Email as Spam, Automatically Move Spam to Junk Spam Folder

So I've been doing some experimenting this weekend with the idea of what to do with suspected junk/spam email and whether we should leave it up to the end user to deal with it, or get the server to do something about it. With iRedMail installed, Dovecot, Postfix and Amavis can take care of this for you, to whatever level you wish.

If you want, you could just have suspected email have a special "tag" added to the subject and then your users can use this to create their own rules to do what they want with it. I want to go one step further, I want "tagged" spam mail to automatically get put into the spam folder. This way, users can still check if they want to, but it will keep their main inbox folder clean and tidy.

Ok, so how do we automatically move spam emails to the junk folder? It's a multistep process..

The first thing we need to do is make sure that all of your domains are included in the Amavis list to be checked and tagged (by default, iRedMail will only do this with your initial/main/Fully Qualified Domain Name). So, thanks to the official iRedMail Forum's user "cmjnissen" for this).

Edit the Amavis configuration file (Ubuntu 12.04 location : /etc/amavis/conf.d/50-user)

Find and comment out the line

#@local_domains_maps = ( [".mydomain.com"] );
and add all of the domains that you host

This ensures all of them are checked and tagged for spam. So now we need to make sure that all emails have a tag added to their subject. We need to scroll down (or search with 'CTRL+w') to the line with the variable "$sa_spam_subject_tag" and make sure it is set to "1". By default, Amavis will fall back to the default settings (/etc/amavis/conf.d/20-debian_defaults) and add the tag "***SPAM***" to the subject line. If you want to change this, add a "$sa_spam_subject_ta" variable to the 50-user configuration with a string that you wish to use. (More info can be found here).

With a quick "sudo service amavis restart", you can now check the headers in any (newly) received emails and you should see spam flags such as :


X-Spam-Flag: NO
X-Spam-Score: 0
X-Spam-Level:
X-Spam-Status: No, score=0 tagged_above=-999 required=4
tests=[RCVD_IN_DNSWL_NONE=-0.0001] autolearn=ham

Ok, so now we have tagged spam emails, we need to move them to the junk folder automatically. Firstly, we need to set the Dovecot "sieve" file to be used. So, edit the file "/etc/dovecot/dovecot.conf" and scroll to the "# Global sieve settings." section and edit the variable "sieve_global_path" so it is as follows: "sieve_global_path = /var/vmail/sieve/dovecot.sieve". Before restarting Dovecot, we need to make sure that the sieve file exists, so CD to "/var/vmail/sieve" you will need to do this as root, so "sudo -i" first.

List everything in the directory and you should see a file named "dovecot.sieve.sample" copy it to "dovecot.sieve" (

cp dovecot.sieve.sample dovecot.sieve"
). Now, change the permissions to 0700 (
chmod 0700 dovecot.sieve
) and change the owner and group to vmail (
chown vmail:vmail dovecot.sieve
). (Have a look at the file and you will see and "if" statement that designates anything with the "X-Spam-Flag" header as "Yes", will be moved to the "Junk" folder:


if header :is "X-Spam-Flag" "YES"
{
fileinto "Junk";
stop;
}

Finally, restart dovecot and you should see any new spam emails automatically pushed to the Junk folder. Rather than waiting for a spam email to come through, you can send one to yourself (from an external server, such as gmail), with the GTUBE (Generic Test for Unsolicited Bulk Email) content, found here :


Subject: Test spam mail (GTUBE)

Content:
This is the GTUBE, the
Generic
Test for
Unsolicited
Bulk
Email

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

So there we go, automatically move spam to your junk folder.

Selasa, 28 Mei 2013

Creating Subdomains with Apache Virtual Hosts | Apache Subdomain Virtual Host Example

Set Up Subdomains with Apache Virtual Hosts

Some people are unaware that the "www" part of the URL is actually a subdomain (http://www.example.com), and can (in theory) point to a different location than http://example.com - it is not advised that you use www as an active sudomain, as most people expect both example.com and www.example.com to have the same content.

This is actually the basis for canonicalisation, but that's another blog post altogether

So how do we set up subdomains using Apache virtual hosts? It's actually very simple, we just need to configure two things, follow this guide for a very simple apache subdomain virtual host example:

  1. Apache Virtual Host Configuration File
  2. Your DNS settings
Setting up your Apache Virtual Hosts file for subdomains is very easy. In fact, you can pretty much copy and paste my config file, just change to your domain and point "DocumentRoot" to wherever you store your site files. If you have set up a virtual host on your server already, you will recognise the layout of the following apache subdomain virtual host example.




<VirtualHost *:80>
# Admin email, Server Name (domain name) and any aliases
ServerAdmin contact@randoof.info
ServerName www.randoof.info
ServerAlias randoof.info

# Index file and Document Root (where the public files are located)
DirectoryIndex index.html index.php
DocumentRoot /srv/www/randoof.info/public_html/

# Custom log file locations
LogLevel warn
ErrorLog /srv/www/randoof.info/logs/error.log
CustomLog /srv/www/randoof.info/logs/access.log combined
</VirtualHost>

<VirtualHost *:80>
# Admin email, Server Name (domain name) and any aliases
ServerAdmin contact@randoof.info
ServerName dev.randoof.info
DocumentRoot /srv/www/randoof.info/dev/

# Custom log file locations
LogLevel warn
ErrorLog /srv/www/randoof.info/logs/dev.error.log
CustomLog /srv/www/randoof.info/logs/dev.access.log common
</VirtualHost>

Depending on how who your domain is set up with, the actual way you change your DNS settings will vary, but the underlying principle to create subdomains with Apache Virtual Hosts is simple. Create a new "A Record" with the left side ("host name") set to "dev" and the right side pointing to your server's IP address.

Don't forget to reload Apache after setting up your virtual host subdomain (sudo service apache2 reload).

That's basically it, there's not much to the apache subdomain virtual host example, probably the hardest part is the DNS configuration, but generally even that is simple.

Jumat, 25 Januari 2013

A Free and Easy Way to Host Email on Your VPS - iRedMail

Woops .. I think it's been far too long since I have last updated this blog, so apologies about that.
I thought it would be prudent to provide a simple and brilliant way to host and serve emails on your VPS. The advantage of the procedure i will be covering means it makes it easy to host websites on a VPS as well.
So, it is possible - and advised by many - to manually set up an email server, here's a quick list of one common and possible collections of applications you will need for an Ubuntu email server:




Postfix + Courier IMAP + MySQL + Amavisd-new + SpamAssassin
 + ClamAV + SASL + TLS + Roundcube + Postgrey

That will cover every aspect you require to set up your email server, including a databse for users, web-, IMAP- and POP-based mail access and antivirus. That's fine and more power to you if you can set that up manually and getting working perfectly. I honestly did go through the entire process in order to manually configure an email server but I got stuck trying to correctly configure DKIM (DomainKeys Identified Mail) and SPF (Sender Policy Framework) validation.
Here's a lovely overview, courtesy of http://workaround.org/ispmail/lenny/bigpicture - side note: Bookmark workaround.org, it's another very helpful site which is great for geeking out and Linux Tutorials

Simple huh?!
So, long story short, after much research, tearing my (virtual) hair out and many server resets, I finally came across a fantastic and free (although there is a paid version) package called iRedMail. I honestly cannot recommend this package enough. At its core, it is a script that downloads and configures a number of software packages:

Postfix, Dovecot, Apache, MySQL, OpenLDAP, Policyd, Amavisd, ClamAV,
Roundcube, Awstats and Fail2ban

You literally answer a couple of questions and the entire system is set up for you. Brilliant. Not that you need it, but for a guide to install IredMail on Ubuntu, try here.
If, by some chance, you have followed my blog up to this point, now is the perfect time to install iRedMail, as you want a basically clean VPS to install it on as it will install everything you need to host and serve multiple websites, using what is known as "Virtual Hosts" with Apache2. Setting up Virtual Hosts will feauter as my next blog entry.