How to Set Up Your Own Email Server

A mail server is a system that manages the correct delivery of emails from the sender to the recipient. For instance, when you send an email via Gmail, you’re using Google’s mail server.

A mail client, on the other hand, is a program used to read, send, and store emails. Examples of mail clients include Microsoft Outlook, Thunderbird, and others.

Protocols for Receiving and Sending Emails

SMTP (Simple Mail Transfer Protocol)

The outgoing mail server uses the SMTP protocol, which stands for "Simple Mail Transfer Protocol." Its primary job is to relay messages between the sender and recipient. SMTP has two main functions:

1. Verifying the sender's configuration and allowing the sending device to transmit the email.

2. Sending the message and receiving the response code.

SMTP servers use ports 25 (unencrypted) and 465 (encrypted) for sending emails.

POP3 (Post Office Protocol)

POP3 is a protocol for receiving emails. It allows a mail client to connect to the server and download the email to the local device, making it available for offline access. However, the email is typically removed from the remote server after downloading (though there's an option to keep a copy on the server).

POP3 operates on ports 110 (unencrypted) and 995 (SSL/TLS encrypted).

IMAP (Internet Message Access Protocol)

Like POP3, IMAP is used to receive emails, but it allows you to manage the messages directly on the server without downloading them locally. This is useful for accessing your email from multiple devices.

IMAP uses ports 143 (unencrypted) and 993 (SSL/TLS encrypted).

Why Set Up Your Own Email Server?

The most common answer is: “To use your own domain in your email address.” While this is true, it's not the full picture.

You can use your custom domain without setting up a dedicated mail server. Many services allow you to connect your domain to their mail servers. All you need to do is buy a domain and link it to their servers in the settings. The main advantage of this approach is that they already have a configured SMTP server. This means you don't need to deal with complicated tasks like configuring forwarding, setting up anti-spam filters, or managing other advanced settings. You simply need to correctly set the NS records (Name Server records), which is much easier than setting up your own mail server.

However, using third-party services comes with limitations:

1. Limited control: You won’t have full control over how your email operates.

2. Sending limits: Every company has its own limits on how many emails you can send. If you send a large volume of emails (e.g., for corporate use or newsletters), these limits might be restrictive.

If you decide to create your own mail server, you’ll need to manage:

• Email quotas and sending limits.

• Backups to avoid losing data.

• Avoiding spam blacklists.

• Setting up access permissions if multiple people are using the same mail system.

Creating and maintaining your own email server gives you more control but also requires a higher level of technical expertise and responsibility.

Setting Up an Email Server

In this guide, we'll explore how to create your own email server. For this purpose, we'll use a Hostman cloud server, selecting Ubuntu 20.04 as the operating system.

The mail server we configure will support encryption, have anti-spam protection, and include a web-based admin panel for managing emails.

Preparing the Server

The first step is to switch to the superuser root mode:

sudo su

Before installing the necessary software, update the server packages:

apt update && apt upgrade

Next, check whether the server's hostname matches the domain of your email:

hostname

If the response shows something different from your desired server name, use the following command:

hostnamectl set-hostname mail.hostname.com

Here, replace mail.hostname.com with your actual hostname.

For the mail server and schedulers to work correctly, we need to configure the time zone by installing the time synchronization package chrony:

apt install chrony
timedatectl set-timezone Europe/Athens

You can choose the appropriate timezone using the command:

timedatectl list-timezones

Start the chrony service:

systemctl enable chrony

Next, we need to open the necessary ports for the mail server using the iptables utility. The required ports are:

• 25, 465 — SMTP (for sending mail)

• 110, 995 — POP3 (for receiving mail)

• 143, 993 — IMAP (for receiving mail)

• 80, 443 — HTTP (for web services)

The first port in each pair is for standard connections, and the second is for secure connections. Use iptables to open these ports:

iptables -I INPUT 1 -p tcp --match multiport --dports 25,110,143,465,587,993,995,80,443 -j ACCEPT

Since iptables rules are only stored for the current session and will reset after a reboot, we need to make them persistent:

netfilter-persistent save

Now, with the server properly set up, we can proceed to install the necessary software to create a fully functional mail server.

Installing and Configuring Postfix

Postfix is an open-source mail transfer agent (MTA). It has a modular architecture, which means it doesn't require running as the root user. Let's install Postfix and the postfix-mysql package for database integration:

apt install postfix postfix-mysql

During installation, select Internet Site when prompted. This assumes you have access to edit DNS records and can specify an FQDN (Fully Qualified Domain Name). In the next window, keep the server name as is and continue.

After installation, create a new user account to handle mail operations. First, create a group called vmail using groupadd:

addgroup -gid 1080 vmail

Next, create the vmail user and assign the home directory to /home/mail:

adduser --home /home/mail -gid 1080 -uid 1080 vmail

Where 1080 is the group ID (GID) and user ID (UID). If 1080 is already taken, you can choose a different value. Verify that the /home/mail directory belongs to the vmail user and group:

ll /home

Configuring Postfix

With the user created, proceed to configure Postfix by editing the main configuration file /etc/postfix/main.cf:

nano /etc/postfix/main.cf

Make the following edits:

# Domains we accept mail for
mydestination = localhost.$mydomain, localhost, localhost.localdomain
# Postfix protocol
inet_protocols = ipv4
# Path to the public certificate
smtpd_tls_cert_file = /etc/ssl/mail/public.pem
# Path to the private certificate
smtpd_tls_key_file = /etc/ssl/mail/private.key
Then, add additional options required for Postfix to function correctly:
# Mail storage location
virtual_mailbox_base = /home/mail
# Path to alias maps
virtual_alias_maps = proxy:mysql:/etc/postfix/mysql_virtual_alias_maps.cf
# Domain storage format
virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql_virtual_domains_maps.cf
# Mailbox storage format
virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf
# Minimum virtual user ID
virtual_minimum_uid = 1080
# UID for the main user handling mail
virtual_uid_maps = static:1080
# GID for the group handling mail
virtual_gid_maps = static:1080
# Register Dovecot as the mail delivery agent
virtual_transport = dovecot
# Enable secure authentication
smtpd_sasl_auth_enable = yes
smtpd_sasl_exceptions_networks = $mynetworks
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
# Enable TLS encryption for outgoing SMTP connections
smtp_use_tls = yes
# Enable TLS support for incoming connections
smtpd_use_tls = yes
smtpd_tls_auth_only = yes
smtpd_helo_required = yes

Now, create the files referenced in the Postfix configuration. Start with the alias maps:

nano /etc/postfix/mysql_virtual_alias_maps.cf

Add the following content:

user = postfix
password = postfixPa$$w0rd
hosts = localhost
dbname = postfix
query = SELECT goto FROM alias WHERE address='%s' AND active = '1'

Similarly, configure the domain maps:

nano /etc/postfix/mysql_virtual_domains_maps.cf

Add:

user = postfix
password = postfixPa$$w0rd
hosts = localhost
dbname = postfix
query = SELECT domain FROM domain WHERE domain='%u'

Finally, configure the mailbox maps:

nano /etc/postfix/mysql_virtual_mailbox_maps.cf

Add:

user = postfix
password = postfixPa$$w0rd
hosts = localhost
dbname = postfix
query = SELECT CONCAT(domain,'/',maildir) FROM mailbox WHERE username='%s' AND active = '1'

Edit the master.cf file:

nano /etc/postfix/master.cf

Add the following settings:

submission   inet  n  -  n  -  -  smtpd
-o smtpd_tls_security_level=may
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_type=dovecot
-o smtpd_sasl_path=/var/spool/postfix/private/auth
-o smtpd_sasl_security_options=noanonymous
-o smtpd_sasl_local_domain=$myhostname
smtps   inet  n  -  n  -  -  smtpd
-o syslog_name=postfix/smtps
-o smtpd_tls_wrappermode=yes
-o smtpd_sasl_auth_enable=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
dovecot   unix  -  n  n  -  -  pipe
flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -d ${recipient}

Generating Security Certificates

To ensure secure email transactions, generate security certificates using openssl. First, create the directory where the certificates will be stored (as specified in the main.cf file):

mkdir -p /etc/ssl/mail

Generate the certificates:

openssl req -new -x509 -days 1000 -nodes -out /etc/ssl/mail/public.pem -keyout /etc/ssl/mail/private.key -subj "/C=CY/ST=Limassol/L=Limassol/O=Global Security/OU=IT Department/CN=mail.devnullhost.com"

Finally, enable and restart Postfix:

systemctl enable postfix && systemctl restart postfix

Installing and Configuring Dovecot

Dovecot is an open-source IMAP and POP3 server. We'll install it along with the required modules for database integration:

apt install dovecot-imapd dovecot-pop3d dovecot-mysql

Next, we configure the mail storage method. Open the Dovecot configuration file:

nano /etc/dovecot/conf.d/10-mail.conf

In the file, specify the directory structure for storing mail. We'll use a hierarchy of domain → user:

mail_location = maildir:/home/mail/%d/%u/

In the same file, configure the authentication method:

service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0666
user = postfix
group = postfix
}
unix_listener auth-userdb {
mode = 0600
user = vmail
group = vmail
}
}
service stats {
unix_listener stats-reader {
user = vmail
group = vmail
mode = 0660
}
unix_listener stats-writer {
user = vmail
group = vmail
mode = 0660
}
}

Edit the Dovecot authentication configuration file:

nano /etc/dovecot/conf.d/10-auth.conf

Replace the line !include auth-system.conf.ext with !include auth-sql.conf.ext, indicating that SQL-based authentication should be used.

Next, configure SSL encryption in Dovecot:

nano /etc/dovecot/conf.d/10-ssl.conf

In this file, add the following:

ssl = required
ssl_cert = </etc/ssl/mail/public.pem
ssl_key = </etc/ssl/mail/private.key

When users connect for the first time, we want their mailboxes to be automatically created. To enable this, open the following file:

nano /etc/dovecot/conf.d/15-lda.conf

Add the following line:

lda_mailbox_autocreate = yes

Now, configure Dovecot to connect to the database. Open the SQL configuration file:

nano /etc/dovecot/dovecot-sql.conf.ext

Add the following lines to configure MySQL:

driver = mysql
connect = host=localhost dbname=postfix user=postfix password=postfixPa$$w0rd
default_pass_scheme = MD5-CRYPT
password_query = SELECT password FROM mailbox WHERE username = '%u'
user_query = SELECT maildir, 1080 AS uid, 1080 AS gid FROM mailbox WHERE username = '%u'
user_query = SELECT CONCAT('/home/mail/',LCASE(`domain`),'/',LCASE(`maildir`)), 1080 AS uid, 1080 AS gid FROM mailbox WHERE username = '%u'

Open the main Dovecot configuration file to set up the server interface:

nano /etc/dovecot/dovecot.conf

In this file, add the following line to listen on all available network interfaces:

listen = *

Finally, enable and restart Dovecot to apply the configuration:

systemctl enable dovecot && systemctl restart dovecot

Installing and Configuring PostfixAdmin

For PostfixAdmin to work correctly, you need a configured web server, PHP, and a MySQL database (either a LAMP or LEMP stack). In this guide, we will skip the web server setup and go directly to installing PostfixAdmin.

First, install the necessary PHP extensions:

apt install php-mysql php-mbstring php-imap

Download PostfixAdmin to the web server's root directory using wget:

wget https://sourceforge.net/projects/postfixadmin/files/latest/download -O postfixadmin.tar.gz

Create a directory for PostfixAdmin and extract the archive contents:

mkdir -p /var/www/html/postfixadmin && tar -C /var/www/html/postfixadmin -xvf postfixadmin.tar.gz --strip-components 1

Then, create a directory for storing template caches:

mkdir /var/www/html/postfixadmin/templates_c

Set the correct permissions for the web server to access the PostfixAdmin directory:

chown -R www-data:www-data /var/www/html/postfixadmin

Create the database and a user for PostfixAdmin:

mysql -u root
CREATE DATABASE postfix DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON postfix.* TO 'postfix'@'localhost' IDENTIFIED BY 'postfixPa$$w0rd';
exit;

In the local configuration file of PostfixAdmin, add the database configuration:

nano /var/www/html/postfixadmin/config.local.php

The file contents should be:

<?php
$CONF['configured'] = true;
$CONF['default_language'] = 'en';
$CONF['database_password'] = 'postfixPa$$w0rd';
$CONF['emailcheck_resolve_domain']='NO';
?>

Open the PostfixAdmin setup page in your browser by going to /postfixadmin/public/setup.php. You will be prompted to generate a password hash for authentication.

Enter the password and click the button. A message containing the hash will appear below the form. Copy this hash and insert it into the config.local.php file:

nano /var/www/html/postfixadmin/config.local.php

Refresh the /postfixadmin/public/setup.php page and log in with the password you used to generate the hash. If everything is configured correctly, you should see a configuration check page.

At the bottom of this page, there is a form to create an admin account. After successfully creating the admin, go to /postfixadmin/public/login.php and log in with the credentials you just set up.

You will be redirected to the PostfixAdmin administration panel.

Creating a Mailbox in PostfixAdmin

In your browser, go to /postfixadmin/public/. In the top menu, choose "Domain List → New Domain."

Then, in the "Overview → Create Mailbox" section, enter the details for a test email account.

You can now test the connection using email clients. Use the following connection parameters:

• Server: Your server's hostname

• IMAP: Port 143, STARTTLS

• POP3: Port 110, STARTTLS

• SMTP: Port 25, STARTTLS

• Login and Password: The credentials you specified when creating the mailbox

Conclusion

In this guide, we explored what a self-hosted email server is and the technologies and software used to configure it.

The main advantage of having your own mail server is the ability to fully customize it: create unlimited mailboxes, assign aliases, and manage users. All these features can be implemented by renting a cloud server from Hostman and configuring the server following this guide.