You’ve got mail!
If you follow r/selfhosted, you know that one of the most common questions asked is, “can I self-host e-mail?” This is usually because most people bought into the Google (Hotmail and Yahoo!) promise of free e-mail only to find out that there are some trade-offs.
There is the privacy trade-off that Google is scanning your e-mail to build a profile about you to show you ads or to sell to data aggregators which is used for the purposes of marketing. This includes selling data for all sorts of business to use, not just online for online ads. (How they track you inside of stores using the signals your phone emits and marry it up with the data they collect from online sources is a whole other concerning subject. But I digress).
Does anyone remember the original promise of “Never delete another e-mail, we’ll give you more storage as you need it”? That went the way of “Don’t be evil” I guess. GMail recently stopped letting me send e-mail unless I paid for more storage or deleted some e-mail. Oof.
What about alternatives to Google? There are other free options, but I don’t see that I can trust them any more than Google. Less, in fact if you consider that Google is generally transparent about what they are doing. They also show a lot more obtrusive ads than Google. If they can’t monetize through advertising (where you are the product) and they don’t have the scale and customer base that Gogole has, then it must be a paid service. I have paid for e-mail hosting in the past and I’m willing to do it again, but I priced a lot of different options and none of them came close to giving me the same capacity and capabilities as Gmail for a price that I consider reasonable.
Self-Hosted E-mail
Back to the original question. Can I self-host e-mail? Yes and no. No, you probably aren’t going to be able to host e-mail on your home network using a residential Internet Service Provider. Spam and abuse closed down that option almost immediately as soon as always-on home Internet became a thing. There are options involving a VPN to a cloud server and TCP proxy-forwarding, but that seems way more kludgy than I’d like.
You can, however, self-host e-mail if you are willing to pay for the server resources from an IaaS (Infratructure as a Service) provider. This could be a baremetal server such as Hetzner or a Virtual Private Server (VPS) from any number of cloud providers like Cloud at Cost, Digital Ocean, OVH, Amazon, Google, etc).
Before we get started, there are a couple of caveats to this and it will require a little bit of luck. Spam fighters are very aggressive and they use their tools like an atom bomb - nuke everything that even looks like spam. Using a shared infrastructure provider like any of the above has the risk that the IP address you are assigned was used previously for spam or one of your neighbors does something suspicious and the whole subnet gets added to a blacklist.
It pays to do some reasearch and check the blacklists before you set anything up then do everything you can to maintain the reputation of your server and its IP. If you need to send a moderate volume of e-mail to a lot of different recipients, you are better off signing up with a service which knows how to do this without getting put on the lists (or pays to stay off said lists). That being said, blacklists affect your ability to send e-mail without it going to spam folders so if your primary purpose is to receive e-mail then it doesn’t matter as much.
You also need to do some reasearch in how to set up DNS for e-mail servers. In the early days, all you needed to do was create an A record for your mail server(s) and add on an MX record for the domain to point to it. Spam and abuse now requires other DNS records to verify that a server is authorized to send e-mail on behalf of a domain or account:
- Reverse DNS is an absolute requirement. You must be able to set the PTR record for the IP address of your server back to your domain. Many IaaS providers have a way to do this if you get a static IP address.
- Autodiscover entries are needed to point different services/protocols within a domain to the correct server. With the proliferation of protocols for e-mail (SMTP/IMAP/POP3) and their secure (SSL/TLS) variants this makes setting up e-mail clients much easier.
- Domain-based Message Authentication, Reporting and Conformance (DMARC) consists of two mechanisms - Sender Policy Framework (SPF) and DomainKeys Identified Mail (DKIM) which attempt to prevent or reduce the ability of unauthorized users to impersonate or send e-mail. There are several guides available on how to set this up.
Fortunately, there are tools such as mail-tester which you can use to tweak your settings before you start using it extensively. You send an e-mail to the address provided and then visit the link provided to see report. Keep tweaking the configration and retesting until you receive the best possible score. I was able to achieve a perfect 10/10 score pretty easily.
Let’s Do This
I decided to go with Digital Ocean as the VPS provider for my e-mail server. They have a relatively good reputation for good customer service, reasonable prices, and understand the importance of maintaining the reputation of their IP blocks. I also decided to go with a solution that could be deployed using Docker and Ansible relatively easily. There are several options, but Mailcow was highly recommended and it has worked out well for me.
I originally spun up a 1 vCPU droplet with 1GB of memory and 25GB of SSD disk, but that was only because I didn’t read the Mailcow requirements ahead of time. 1GB would have been fine, but it meant disabling some features such as anti-spam, anti-virus, and search. I had to up that to 3GB which means it will cost about $20/month. That might seem like a lot for e-mail considering there are paid services available for a lot less, but I can host a whole lot of accounts and domains before I outgrow that. I plan to offset this cost by self-hosting other services on my own infrastructure at home for which I’m currently paying a monthly cost.
I picked a basic Debian 10 template for my new droplet, checked the IP address against blacklists, and used my Ansible bootstrap role to setup an Ansible user with sudo privileges and ssh keys. Next I needed a new Ansible role for Mailcow.
The new role builds on the previous roles I created for installing and updating packages, setting up ssh keys, setting up restic for backups to Backblaze, basic firewall rules, fail2ban to block repeated failed attempts to access the system, a user under which to deploy apps, and of course, Docker itself.
Here is the exerpt from the prod.yml Ansible playbook:
- name: Mail server
hosts:
- lachlanmail
remote_user: ansible
become: yes
roles:
- mailcow
roles/mailcow/meta/main.yml:
dependencies:
- role: common
- role: debian
- role: sshkeys-root
- role: paulfantom.restic
- role: ufw
- role: user
- role: fail2ban
- role: docker
roles/mailcow/tasks/main.yml:
---
- name: Deploy UFW Application Profile
template:
src: "templates/ufw-mailcow-app.j2"
dest: "/etc/ufw/applications.d/mailcow"
owner: "root"
group: "root"
mode: "0644"
- name: Allow Mailcow HTTP(s) ports
ufw:
rule: allow
app: HTTP
- name: Allow Mailcow IMAP port
ufw:
rule: allow
app: IMAP
- name: Allow Mailcow IMAPS port
ufw:
rule: allow
app: IMAPS
- name: Allow Mailcow ManageSieve port
ufw:
rule: allow
app: ManageSieve
- name: Allow Mailcow POP3 port
ufw:
rule: allow
app: POP3
- name: Allow Mailcow POP3S port
ufw:
rule: allow
app: POP3S
- name: Allow Mailcow SMTP port
ufw:
rule: allow
app: SMTP
- name: Allow Mailcow SMTPS port
ufw:
rule: allow
app: SMTPS
- name: Check out Mailcow from github
become: yes
become_user: deploy
become_method: sudo
git:
repo: 'https://github.com/mailcow/mailcow-dockerized'
dest: /home/deploy/mailcow-dockerized
clone: yes
update: no
- name: Deploy Mailcow configure response file
template:
src: "templates/mailcow-response.j2"
dest: "/home/deploy/mailcow-dockerized/mailcow-response"
owner: "deploy"
group: "deploy"
mode: "0644"
- name: Run Mailcow configuration
become: yes
become_user: deploy
become_method: sudo
shell: ./generate_config.sh < mailcow-response
args:
chdir: /home/deploy/mailcow-dockerized
creates: mailcow.conf
- name: Pull latest containers for Mailcow
become: yes
become_method: sudo
docker_compose:
project_src: /home/deploy/mailcow-dockerized
build: no
pull: yes
recreate: smart
- name: Deploy cron script
template:
src: 'mailcow.cron.j2'
dest: '/etc/cron.d/mailcow'
mode: '0640'
no_log: true
templates/mailcow-response.j2 (answers the configuration questions):
{{ansible_host}}
n
templates/mailcow.cron.j2 (save backups to the backup directory where restic is configured to do its backups):
vi: ft=jinja.crontab
{{ ansible_managed | comment }} MAILCOW_BACKUP_LOCATION=/home/deploy/backup/mailcow
Do the backup
0 6 * * * root /home/deploy/mailcow-dockerized/helper-scripts/backup_and_restore.sh backup all –delete-days 7
templates/ufw-mailcow-app.j2 (define the ufw application profile for the ports which need to be opened) :
[SMTP] title=Postfix SMTP description=SMTP port to postfix-mailcow container ports=25/tcp
[SMTPS] title=Postfix Secure SMTP description=SMTPS port to postfix-mailcow container ports=465/tcp
[Submission] title=Postfix Submission description=Postfix Submission port to postfix-mailcow container ports=587/tcp
[IMAP] title=Dovecot IMAP description=IMAP port to dovecot-mailcow container ports=143/tcp
[IMAPS] title=Dovecot Secure IMAP description=IMAPS port to dovecot-mailcow container ports=993/tcp
[POP3] title=Dovecot POP3 description=POP3 port to dovecot-mailcow container ports=110/tcp
[POP3S] title=Dovcecot Secure POP3 description=POP3S port to dovecot-mailcow container ports=995/tcp
[ManageSieve] title=Dovecot ManageSieve description=Dovecot ManageSieve port to dovecot-mailcow container ports=4190/tcp
[HTTP] title=NGINX description=HTTP and HTTPS ports to nginx-mailcow container ports=80,443/tcp
Run 'ansible-playbook prod.yml' and you are good to go.
### Configuring Mailcow
The next part involves logging into the Mailcow web-based administration interface and configuring the e-mail domains, setting up accounts, and other preferences. There is already some [very nice documentation](https://mailcow.github.io/mailcow-dockerized-docs/) availbable from the Mailcow folks, so I won't recreate those steps here. I would, however, recommend that once you've configured your domain in Configuration>Mail Setup, you click on the DNS button and that will open a window which queries the various DNS records for that domain and tells you if they are configured properly. You don't have to set up every record it checks for, but definitely configure the ones I mentioned earlier.
### Mail Clients
Once you have everything configured server side, proceed with configuring your mail client of choice. I prefer Mozilla Thunderbird on Linux and Windows. They have been around a long time and work very well. On Android, K-9 was highly recommended but I didn't like the interface so I ended up with the standard Samsung mail client which supports ActiveSync.
### Email Encryption
If you use the web mail client provided by Mailcow, SoGo, and the secure version of the IMAP/POP3/SMTP protocols you will have encryption in transit for reading and writing mail as it has support for Let's Encrypt right out of the box. However, this does not provent your mail from being only read by the intended recipient.
For this, you need to support for the OpenPGP (Pretty Good Privacy) encryption standard which uses a private and public key to provide an assymetical encryption. If used correctly within the [web of trust](https://en.wikipedia.org/wiki/Web_of_trust), it ensures that the sender is the person they claim to be and the recipient is the only person that can read the contents of the e-mail. It does not, however, protect the metadata of the e-mail such as from, to, subject, and other headers.
I will cover this in a bit more detail in my next post which will be about Keybase.io, GNU Privacy Guard (GPG), and FlowCrypt.