How to set up an email server on a VPS

Key takeaways

• A VPS mail server needs a domain, dedicated IP, DNS access, and the right email ports.
• A, MX, and PTR records route email, while SPF, DKIM, and DMARC verify it.
• Postfix and Dovecot support Linux setups, while bundled or Windows tools reduce manual work.
• Self-hosting gives you more control, but you must manage security, deliverability, backups, and maintenance.

Hosting your own email server on a VPS gives you control over your software, storage, privacy, and domain configuration.

You can set up email for a business, personal domain, or group of clients. The process requires more than installing mail software, though. You also need to configure DNS, encrypt connections, authenticate outgoing messages, and protect the reputation of your server’s IP address.

This guide walks through Linux and Windows setup options and what to consider before you begin.

What is an email server?

An email server is a computer that sends, receives, and stores email messages.

If you’ve used Gmail or Outlook, their servers handled that work for you. With your own VPS, you can run and manage the mail services yourself.

An email server usually relies on several protocols:

• SMTP sends messages between mail servers and accepts outgoing messages from authenticated users
• IMAP lets users view and manage email while keeping it stored on the server
• POP3 downloads messages from the server to an email client

Most self-hosted Linux setups use one service for sending and routing messages and another for mailbox access. For example, Postfix can handle SMTP while Dovecot provides IMAP or POP3 access.

Benefits of hosting your own email server

Running your own email server comes with a handful of major advantages:

• Cost savings: No monthly per-user fees or escalating prices as your team grows.
• Full control: You choose your software, set your limits, and manage access.
• Better privacy: You decide how the server stores email and who can access it.
• More storage flexibility: Allocate VPS disk space as you want, with no arbitrary mailbox limits.
• Custom security: Configure firewalls, authentication, encryption, spam filtering, and access rules around your requirements.
• Scalability: Add domains, addresses, storage, or computing resources as your needs grow.
• Custom branding: Use your own domain for professional, on-brand communication.

Self-hosting gives you more control over your email environment, but that control also comes with responsibility for maintaining the server and mail stack.

What you need before you set up the server

Before installing a mail stack, confirm that your VPS provider and domain support the configuration you need.

A VPS with the right resources

Your VPS needs enough CPU, RAM, and storage for the number of mailboxes you plan to host. Webmail, antivirus scanning, spam filtering, large attachments, and longer retention periods can increase resource requirements.

You’ll also need:

• Root or administrator access
• A static, dedicated public IP address
• A supported Linux or Windows Server operating system
• SSD storage sized for your mailbox retention needs
• A backup or snapshot option

A domain and DNS access

You need a registered domain and permission to edit its DNS records. The setup requires an A record and MX record, along with TXT records for SPF, DKIM, and DMARC.

Your VPS provider must also let you set reverse DNS for the server’s IP address.

Permission to send email

Ask your VPS provider whether it permits self-hosted email and whether it blocks outbound traffic on port 25. Many providers restrict port 25 to reduce spam and network abuse.

Some providers remove the restriction after reviewing the customer’s intended use. Others require customers to use an external SMTP relay. Confirm the policy before you build the server.

How to set up a self-hosted email server on a Linux VPS

Setting up an email server on a Linux VPS is totally doable with some command line comfort and the right software. Here’s a step-by-step breakdown.

The commands and file paths in this section use a typical Ubuntu or Debian-based setup. Check the documentation for your operating system and software versions before making changes to a production server.

1. Set the server hostname

Assign a fully qualified domain name, or FQDN, such as:

On many Linux distributions, you can set it with:

Verify the hostname:

Use the same hostname for your mail server, A record, and reverse DNS. A mismatch between these values can affect trust and deliverability.

2. Point your domain’s DNS to the VPS

To use your custom domain, you’ll need to update the DNS.

Start with these records:

Replace the sample domain and IP address with your own values.

The MX record should point to a hostname with a valid A record. Don’t point the MX record directly to an IP address.

You usually configure the PTR record through the VPS provider rather than your regular DNS editor. Receiving mail services often check reverse DNS, and a missing or mismatched PTR record can lead to rejected messages or spam placement.

3. Choose a Linux mail server stack

You can piece this together manually or use a bundled solution.

Common options include:

• Postfix and Dovecot: A flexible manual setup for administrators who want detailed control
• Mail-in-a-Box: An all-in-one platform designed to automate much of the setup
• iRedMail: A mail server installer with web-based administration options
• Modoboa: An open-source mail hosting platform with webmail and administrative tools
• Mailcow: A Docker-based mail server suite with a web interface

Most include SMTP, IMAP/POP3, webmail, spam filtering, and TLS support.

A bundled platform can reduce the amount of manual configuration, but it doesn’t remove the need to install updates, monitor logs, protect accounts, and back up the system.

The remaining Linux instructions focus on Postfix and Dovecot because they give you direct control over mail transfer and mailbox services.

4. Install Postfix and Dovecot

Update the server packages:

Install Postfix:

When the installer asks for the mail configuration type, choose Internet Site for a standard server that sends and receives mail directly. Enter your domain as the system mail name.

Install Dovecot with IMAP support:

Install POP3 support only when you need it:

IMAP usually works better for people who check the same mailbox from multiple devices because it keeps messages and folder changes on the server.

5. Configure Postfix and Dovecot

Open the main Postfix configuration file:

Confirm that its core settings match your hostname and domain:

Restart Postfix after saving the changes:

Next, configure Dovecot to use IMAP and the same Maildir storage format. Depending on the installed Dovecot version, you may need to update its protocol, mailbox location, and authentication files.

The exact configuration syntax can differ between Dovecot releases, so use the documentation that matches your installed version.

Restart Dovecot after saving the changes:

Check both services:

6. Set up SSL/TLS encryption

Use Let’s Encrypt to generate a free SSL certificate so mail clients trust your server.

Install Certbot:

Request a certificate for your mail hostname:

Configure Postfix and Dovecot to use the certificate for encrypted connections.

Test the renewal process:

Keep certificate renewal automated and monitor it for failures. An expired certificate can prevent users from connecting securely.

7. Configure SPF, DKIM, and DMARC

SPF, DKIM, and DMARC help receiving providers verify your messages and protect your domain from spoofing.

Add an SPF record

SPF identifies the servers allowed to send mail for your domain. A basic TXT record for a single VPS may look like this:

Replace the example IP with your VPS address.

Keep all authorized senders in one SPF record. Publishing multiple SPF records for the same domain can cause authentication failures.

Configure DKIM

DKIM adds a digital signature to outgoing messages. Your server stores the private key, while the corresponding public key appears in a DNS TXT record.

A manual Postfix setup commonly uses a DKIM signing service. Bundled mail platforms often generate the key pair and provide the DNS value through their control panel.

Add a DMARC record

DMARC tells receiving servers how to handle messages that fail authentication. A monitoring record can look like this:

A p=none policy collects reports without asking receiving systems to quarantine or reject messages. After confirming that legitimate messages pass SPF or DKIM, you can consider a stricter policy.

8. Secure authentication and prevent an open relay

Make sure your users authenticate before sending mail.

Postfix can use SASL authentication to confirm that a remote client has permission to send mail through the server.

A common Postfix configuration includes:

You must also configure the matching authentication socket in Dovecot.

Test the server from an outside network and confirm that it rejects unauthenticated attempts to send messages between unrelated domains. Otherwise, spammers may use the VPS as an open relay and damage its IP reputation.

9. Configure the firewall and mail ports

Allow only the necessary ports:

Port 25 and port 587 have different jobs. Mail servers use port 25 to exchange messages, while users and applications generally send authenticated mail through port 587.

Open only the ports your configuration uses. For example, you do not need POP3 ports when everyone connects through IMAP.

Use UFW or iptables to block everything else. You can also install Fail2Ban for brute-force protection.

Keep your SSH port open before enabling a new firewall configuration so you don’t lose administrative access.

10. Add spam and malware protection

Add spam filtering tools like SpamAssassin or Rspamd.

You can also add antivirus scanning for attachments. Account for the extra RAM and CPU these services use when choosing your VPS plan.

Filtering software cannot compensate for weak account security. Use strong passwords, limit repeated login attempts, remove old accounts, and install operating system and mail software updates promptly.

11. Test and monitor the server

Send a message from your new mailbox to an external account, then reply to it. Confirm that the server can send and receive email and that mail clients don’t show certificate warnings.

Review the message headers and check that SPF, DKIM, and DMARC report PASS.

You should also check:

• Reverse DNS
• SMTP connectivity
• TLS configuration
• IP blocklist status
• Mail queue activity
• Authentication failures

View the Postfix queue with:

On many Ubuntu and Debian systems, you can follow the mail log with:

Email servers can get blacklisted fast if they’re misconfigured. Watch your logs, monitor your IP reputation, and set up alerts for mail queue spikes or failures.

How to set up a self-hosted email server on a Windows VPS

Prefer Windows? The process is similar, but the software stack changes a bit.

1. Choose a Windows VPS

Look for a VPS that comes with a Windows Server license, RDP access, and at least 2 GB of RAM. You’ll want similar specs as Linux, but note that Windows tends to use more memory.

Add more RAM when you plan to host many inboxes or run spam filtering and antivirus tools.

2. Configure your domain and reverse DNS

Just like with Linux, point your domain’s MX and A records to your Windows VPS IP. Don’t forget SPF, DKIM, and DMARC.

Set the PTR record through your VPS provider and make sure it matches the Windows server hostname.

3. Install a Windows-compatible mail server

Choose software that supports your Windows Server version and continues to receive security updates.

Windows mail server options may include:

• MailEnable
• IceWarp
• Other actively maintained SMTP and IMAP platforms designed for Windows Server

Review current software support, licensing, authentication features, and update policies before installing a platform. Don’t rely on an older recommendation simply because the software remains available for download.

4. Set up encryption and authentication

Windows tools like Certify The Web or Win-ACME can handle Let’s Encrypt certificates. Then configure your mail server to require authentication and encrypt mail connections.

Confirm that users submit outgoing mail through an authenticated port and that the server doesn’t permit open relay activity.

5. Configure Windows Defender Firewall

Use Windows Defender Firewall to open the same ports as on Linux: 25, 587, 993, and any other ports required by your setup.

Avoid opening POP3 or unencrypted protocols unless users need them.

6. Test mail flow

Send test emails using Outlook or Thunderbird. Check that messages are sending, receiving, and not ending up in spam. Use Event Viewer to troubleshoot any issues.

Review message headers to confirm that SPF, DKIM, and DMARC pass.

Common VPS mail server problems

Messages go to spam

Check the message headers before changing the server. Confirm that reverse DNS matches the hostname and that SPF, DKIM, and DMARC pass.

A new IP address has little or no sending history. Avoid sending a sudden burst of email from it, and monitor its reputation as normal use increases.

Authentication fails

Confirm that the user entered the right username, password, hostname, port, and encryption method. On Linux, also verify that Postfix can reach the Dovecot authentication socket.

The IP appears on a blocklist

Check for compromised accounts, an open relay, unexpected sending volume, or malware. Correct the cause before requesting removal from the blocklist.

What is the best VPS for an email server?

The best VPS for an email server provides enough resources for your mailbox volume and gives you room to grow as storage and processing needs increase.

When comparing VPS plans, focus on:

• CPU and RAM
• SSD storage
• Backups or snapshots
• Upgrade options
• Reliable network availability
• Management and support scope

Use these resource ranges as starting points:

Actual requirements depend on mailbox size, message volume, attachments, webmail, filtering, antivirus scanning, and backup retention.

A managed VPS plan may cover parts of the operating system or control panel, but it may not include custom mail server configuration or deliverability work. Check the support scope before choosing a plan.

Do you really need self-hosted email?

A VPS mail server makes the most sense when you need direct control over storage, routing, access, or domain policies and have someone who can maintain the operating system, review mail logs, manage DNS, and respond to deliverability problems.

A managed email service may make more sense when your team doesn’t have server administration experience or when email downtime would create a serious business problem. Consider whether your team needs control over the mail stack itself or simply needs an email service that someone else maintains.

Self-hosting business email and sending bulk marketing campaigns also require different tools. A basic mail server doesn’t automatically provide subscriber consent management, one-click unsubscribe, bounce handling, complaint processing, or campaign reporting.

Ongoing email VPS maintenance

After launch, keep the server secure and test it regularly. Your routine should include software updates, certificate renewal checks, log reviews, mail queue monitoring, blocklist checks, storage monitoring, and DMARC report reviews.

Back up mailbox data, configuration files, certificates, and DKIM keys. Test the restoration process instead of assuming the backups will work when you need them.

A single VPS also creates a single point of failure. Businesses that cannot tolerate an extended email outage should plan for recovery, secondary mail routing, or a redundant architecture.

Email VPS FAQs

Email VPS next steps

A VPS mail server gives you control over your software, mailbox storage, domain settings, and security policies. That flexibility also makes you responsible for authentication, maintenance, deliverability, and recovery.

Start by checking whether your domain, dedicated IP, reverse DNS access, server resources, and provider policies support the mail stack you plan to use.

Liquid Web’s self-managed VPS hosting gives experienced administrators root access, operating system choice, SSD storage, and room to scale their mail server resources. Explore the available VPS options to find a plan that fits your mailbox volume, preferred software stack, and storage requirements.