EMAIL INFRASTRUCTURE GUIDE

How to Set Up an Email Server

Learn how to build a secure, reliable, and professional email server using a practical Linux-based stack. This guide covers SMTP, IMAP, DNS, encryption, authentication, and the core best practices required for stable email delivery.

IDEAL FOR

Sysadmins, hosting providers, advanced users

CORE STACK

Postfix, Dovecot, TLS, SPF, DKIM, DMARC

MAIN GOAL

Reliable mail delivery with strong security

Why Run Your Own Email Server?

Running your own email server gives you direct control over your domain’s communication infrastructure. You manage user mailboxes, delivery rules, server security, storage policies, and authentication settings without being fully dependent on an external provider. For businesses, agencies, and technical teams, this level of control can be a major advantage.

That said, email hosting is one of the more demanding server administration tasks. A successful deployment requires much more than simply installing a mail package. You need proper DNS records, TLS certificates, secure client authentication, anti-spam defenses, and a clean server reputation. Without those elements, your email may be rejected, delayed, or sent directly to spam folders.

This guide walks through the essential setup process so you can build a modern Linux email server with a more professional, hosting-grade approach.

Requirements Before You Begin

Domain name

A domain you control, such as example.com, with editable DNS records.

Linux server

A VPS or dedicated server with a static public IP and root or sudo access.

Open ports

SMTP, submission, IMAP, and secure mail ports must be allowed through your firewall.

Basic admin skills

You should be comfortable editing config files and managing services on Linux.

Step-by-Step Setup

1) Choose the Right Email Stack

For most Linux-based email servers, a common and reliable combination is Postfix for SMTP delivery and Dovecot for mailbox access. This pairing is widely used because it is stable, well-documented, and flexible enough for both small and large deployments.

Postfix handles outbound and inbound SMTP mail flow
Dovecot provides IMAP and POP3 access
Rspamd or SpamAssassin helps filter spam
Roundcube can be added later for browser-based webmail

2) Update the Server and Install Packages

Begin by updating your operating system and installing the core mail packages.

sudo apt update
sudo apt upgrade -y
sudo apt install postfix dovecot-imapd dovecot-pop3d -y

During installation, you may be asked to define the system mail name. Use your domain or intended mail hostname, such as mail.example.com.

3) Set a Proper Mail Hostname

Your email server should use a fully qualified domain name that matches your DNS configuration. A common format is:

hostnamectl set-hostname mail.example.com

This hostname should align with your A record, MX record, and ideally reverse DNS as well.

4) Configure DNS Records Correctly

DNS is one of the most critical parts of a successful email deployment. A professionally configured email server needs more than an MX record. It also needs authentication and policy records.

A record:    mail.example.com   -> your.server.ip
MX record:   example.com        -> mail.example.com
SPF record:  v=spf1 mx ip4:YOUR.SERVER.IP ~all
DKIM record: generated by your mail server
DMARC:       v=DMARC1; p=quarantine; rua=mailto:admin@example.com

These records help improve trust, reduce spoofing, and increase inbox delivery rates.

5) Configure Postfix for SMTP Delivery

Postfix uses /etc/postfix/main.cf as its main configuration file. A simple starting point may look like this:

myhostname = mail.example.com
mydomain = example.com
myorigin = /etc/mailname
mydestination = localhost
inet_interfaces = all
home_mailbox = Maildir/

This tells Postfix how to identify the server and where user mail should be stored.

6) Configure Dovecot for Mailbox Access

Dovecot allows users to connect through IMAP or POP3 using standard mail clients.

sudo nano /etc/dovecot/dovecot.conf

protocols = imap pop3
mail_location = maildir:~/Maildir

Keep mailbox paths consistent between Dovecot and Postfix to avoid delivery issues.

7) Enable TLS Encryption

Modern mail services should always encrypt authentication and mailbox traffic. A Let’s Encrypt certificate is a practical starting option.

sudo apt install certbot -y
sudo certbot certonly --standalone -d mail.example.com

Then configure the certificate paths inside your mail service settings:

ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem

8) Open Required Mail Ports

Make sure your firewall allows the ports used by SMTP submission and mailbox protocols.

25   SMTP
465  SMTPS
587  Submission
143  IMAP
993  IMAPS
110  POP3
995  POP3S

sudo ufw allow 25
sudo ufw allow 465
sudo ufw allow 587
sudo ufw allow 993
sudo ufw allow 995

9) Create Mail Users

Small deployments can use standard Linux user accounts, while larger environments often use virtual users and separate mail storage.

sudo adduser alice

After user creation, verify mailbox permissions and test login from an IMAP-capable mail client.

10) Add Spam Filtering and Login Protection

A hosting-grade email server should include anti-spam and intrusion defenses from the start.

sudo apt install spamassassin fail2ban -y

This helps block brute-force login attempts and reduces junk mail before it reaches user inboxes.

Testing and Validation

Check	Why It Matters	Example
MX record	Confirms where your domain receives mail	`dig MX example.com`
SMTP response	Confirms Postfix is reachable	`telnet mail.example.com 25`
IMAP TLS	Verifies secure mailbox access	`openssl s_client -connect mail.example.com:993`
Real inbox delivery	Checks spam placement and acceptance	Send to Gmail or Outlook

Common Problems to Watch For

Mail goes to spam: usually caused by weak authentication, poor IP reputation, or missing DNS records
Clients cannot connect: often linked to firewall issues, certificate errors, or incorrect Dovecot settings
Mail is not received: check MX records, hostname consistency, and service status
TLS handshake fails: review certificate paths and renew certificates on time
Relay errors appear: inspect Postfix restrictions and SMTP authentication rules

Final Thoughts

A self-hosted email server can be powerful, flexible, and professional — but only when it is built with the right technical foundation. The most important parts are a clean hostname, strong DNS authentication, secure TLS, and properly configured SMTP and IMAP services.

Once the basics are working, you can improve the platform with webmail, monitoring, mail queues, backups, advanced filtering, and virtual mailbox management. That is where a simple server begins to look and perform like a premium hosting-grade mail platform.

Best starter stack: Postfix + Dovecot
Most important task: DNS authentication
Best upgrade path: webmail + anti-spam + backups