If you are an experienced system administrator you may want to setup your own email server. Setting up a custom email server is a challenging task, but you gain a better understanding of how email works and its inner mechanism by completing it. In this tutorial, you will learn how to setup a full-featured Linux mail server.
Requirements
To be able to go through this tutorial you need a Debian/Ubuntu Linux server and a domain name. For other Linux distributions you need to edit some of the commands. The server you have should be able to use port 25. If you have netcat-traditional package installed, you can do this to check:
nc smtp.google.com 25
If you see something like 220 mx.google.com ESMTP n25-2002...1wmi.102 - gsmtp, then everything is fine.
Also, you need to figure out if your server hosting provider gives you the ability to set PTR / Reverse DNS record. In DigitalOcean, this is done by setting droplet name to domain. With many other providers, to point your IP to the domain you should contact tech support and ask them to do it. Here is a recap of what you need:
- Domain name, preferably on CloudFlare pointed to your server IP address
- Debian/Ubuntu Linux server
- Non-blocked port 25
- Ability to set PTR for server IP address
Objective
In the end of the tutorial you will get:
- Webmail interface using Roundcube running on subdomain, but sending mail from main domain
- Mailu Docker Compose instance
- SMTP relay compatible with all mail clients
- IMAP+POP3 server for mail retrieval
- Encryption available for all protocols above
- DKIM, DMARC, SPF, PTR records set correctly
Getting started
If you just rented your server, follow the server setup tutorial to secure it from bots. Otherwise, you will have a vulnerable system. Also, you should create a non-root user (it is in the guide above too) for security reasons.
Install Docker and Docker Compose
To make sure you are not using old versions of Docker, uninstall them (if you have previously installed):
sudo apt remove docker docker-engine docker.io containerd runc
Install required packages:
sudo apt install ca-certificates curl gnupg lsb-release
Create keyrings directory for apt and install Docker PGP key there:
sudo mkdir -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
Add Docker repository to sources.list.d directory:
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Update again to fetch newly added repository. Then install Docker and Docker Compose:
sudo apt update sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin
Add yourself to group docker to be able to run it as non-root:
sudo gpasswd -a $(whoami) docker
Log out from your server and login again to become docker group user.
Configure Mailu
Create a directory for Mailu data store and configuration. We will use /opt/mailu. Create the directory and set proper permissions on it.
sudo mkdir -p /opt/mailu sudo chown -R $(whoami):docker /opt/mailu
Use official Mailu setup tool to generate initial configuration. Go there and follow these steps:
- Select Compose flavor and click Next.
- Set “Mailu storage path” as
/opt/mailu. - Specify your sending domain in “Main mail domain” field (i.e.
domain.com). - Use “letsencrypt” option for encryption.
- You can opt out of statistics if you want.
- Set any Website name and Linked Website URL you want.
- Enable admin UI using the checkbox.
- Select Roundcube as Web email client.
- Set IPv4 listen address to your server external IP (or
0.0.0.0, but it is not supported well). - Under “Public hostnames” specify the subdomain like
mail.domain.com. - For the sake of simplicity, leave SQLite as database engine.
- Click “Setup Mailu”.
The website will redirect you to the page with configuration UUID in the URL. In /opt/mailu directory use wget command to download generated files (see the exact URL on the page):
cd /opt/mailu wget 'https://setup.mailu.io/1.9/file/5e4ad6b0-c928-...b3273/docker-compose.yml' wget 'https://setup.mailu.io/1.9/file/5e4ad6b0-c928-...b3273/mailu.env'
Review the configuration. If everything is OK, go ahead and launch the system.
Setup Mailu
Make sure that mail.domain.com have an A record which contains your server IP address. In /opt/mailu directory, run Mailu:
docker compose -p mailu up -d
It will download the images and run them. You should see no errors. If everything is fine, set admin password (enter your domain and password inline):
docker compose -p mailu exec admin flask mailu admin admin domain.com YOUR-PASSWORD
If you configured postmaster name other than admin, replace last occurrence of admin with your postmaster name.
DNS setup
Go to mail.domain.com and login as admin@domain.com and enter Admin interface. Go to Mail domains section and click the first icon as shown below:
Click Generate/Regenerate keys button to generate DKIM key pair. It should generate successfully and show it on the page.
Create a .txt file on your computer and copy all of the entries there (i.e. “DNS DKIM entry” but not “DKIM public key”). There is a copy button on the left for each entry. Navigate to CloudFlare DNS dashboard, delete old mail-related DNS entries, click “Advanced” button, select a file and import it.
Also, make sure your mail server IP reverse DNS entry (PTR) points to the domain. You can check it with Virbox PTR and Reverse DNS checker tool. Querying your server IP should your domain name. You can setup PTR entry using your server hosting provider, not domain registrar.
If everything is done correctly, you should be able to enter webmail interface from mail.domain.com/webmail and send and receive email.