My Profile Photo

AndrewCz


Using liberty-minded opensource tools, and using them well


Home email service


I call it 'Service' because it'll end up being more than one server. Hopefully this can give me multiple email accounts with varying degrees of security and accessibility.

For the record, this will have several programs working together - namely Postfix, Dovecot, Fetchmail, NeoMutt, Thunderbird, OpenSMTP, SpamAssassin, ClamAV, and notmuch.

MX Record

The MX record for this domain will be set for a VPS that’s running a MTA - namely postfix, seeing as I’m familiar with it. That will only be a forwarding/proxy/loadbalance server that will forward on requests to one of two MTA servers. One will be running in my DMZ - that is my main server that will recieve my mail. I will also maintain a backup that’s running on a VPS in the event of interrupted service of my hosted box.

IMAP and POP

There are a couple ways to deliver the mail, and that’s where my privacy concerns and accessibility concerns deviate. I will have IMAP access to the outside internet for when I’m away and need to access my email, however, I will also have a MDA that is not accessible from the internet.

While I will have Dovecot running on the DMZ email machine, I will have a second server running dovecot and fetchmail on the inside LAN that’s behind my DMZ. Fetchmail will poll the two servers (the main DMZ server, and the VPS backup) to see if new email has arrived that needs to be taken into shelter there and retrieve it. It will not use imap, but rather POP3 in order to remove the sensative emails from the original server.

The way the emails are addressed (aka user@domain) will be completely arbitrarily chosen by myself - most likely via different subdomains. That’s then the privacy side - a ‘pull’ mechanism to not have anything pushed into the LAN without a request that initiated from outside of it. Then later, when I get home, I will be able to retrieve that mail by physically connecting to that network and retrieving that mail which had subsequently been put into the MDA - in this case dovecot.

Accessing the mailbox with MUA’s

I’ve decided to try out NeoMutt, since I liked Mutt so much. I would be very excited to see what it offers over vanilla mutt, and also it’s the most dead-simple MUA in existance. Also, for outside access, I’ll be using thunderbird (icedove in debian) most likely as it’s already pre-installed, and evolution doesn’t really tickle my fancy if you know what I mean.

Backups and notmuch

Backups will be easy too as they’re a pull model and can subsequently use fetchmail on any server with pop capabilities. I’ll have to set up a script to run that for old email - the U.S. has set a legal turn-over-your-email date of 180 days of sitting on a server, so I’ll have to make it any email that has reached ~150 days, just to be safe.

When they’re retrieved, they will all be put into a Maildir backup and indexed for later retrieval if necessary. I’ll not have anyone accusing me of losing emails e.g. Hillary Clinton.

Security and ports

I doubt that my DMZ port 25 is open, but luckily enough, my MX record is pointing to a proxy that I can have send all incoming email to 587. IMAP on 993 and POP3 on 995. That is with all of the SSL/STARTTLS ports as default because the incoming mail is from my proxy. Now that proxy, that can have ports 25, 465 (depricated), 587, 143/993, and 110/995 all open to accept connections. I may limit it down the road, but I’m not too concerned about that box.

The most internal server will only have an imap service running. Of course, all appropriate precautions should be taken, but there’ll be little on that box to expose, as I only need it for one purpose.

Spam and Anti-Virus

I’m not quite sure how much I’m going to benefit from anti-virus, being an all-linux shop, but it’s still worthwhile to enable, ya know, justin case. Anyways, I’ll have that as well as spam filtering out of my inboxes. Those chores will be handled by ClamAV and SpamAssassin respectively, as they’re the cream of their crop for this type of stuff.

Postfix

Master.cf is such a weird name for a file. There’s also over 1000 lines in that file with all the comments. All told it’s just over 120 lines of pure config options. So I won’t be going through them here, but trying to figure out how I’m going to set my variables.

Server Names

Well, I think I’ve come to the conclusion that there are going to be three `` variables that I’ll be using:

  • home
  • relay
  • backup

And this should cover all eventualities. I’ll see how this works in the config file. Keep in mind the internal server is only going to be a MDA at most, so there’s no reason to use postfix there. Sendmail at the most. With fetchmail at the most.

Second read-through

Well, I read through the ISPmail guide for Debian Jessie which had tons of great advice, and I re-read Ars Technica’s Taking Email Back, Part 2 (which itself is four parts long) which was a simpler explanation of the setup.

What I took away from ISPmail’s write-up were the following. I really do want to use a MySQL database, but don’t want to have an apache instance running on this server. So I’ll have to learn MySQL commands. Luckily, the walk-through does a good job at giving me the correct commands (it seems like). This way I can have all my authentication going through dovecot - except for postfix’s login maps for protecting against forged sender attacks that use his email2email database lookup.

Secondly, he talks about self-signing a cert, and then in the comments says:

If several MX servers point to the same IP that’s no problem. If one server sends email to another then nowadays it will accept any certificate and initiate an encrypted connection.

I guess it’s relying on DNS for identity so I should theoretically be able to self-sign my server’s cert - but I’ll definitely want to get something from let’sencrypt.

To top that off, for the communication between my relay and my backup and home servers, I should be able to use PKI. That means I’ll have to install the root certificateson the servers, but that shouldn’t be a problem.

Dovecot will be handling all of the authentication, but will only be present on the home and backup servers. No need for it to be installled in the relay.

The worrysome part is, in a setup like this, I would want to bounce the emails at the relay server, and not the home server. So the SQL databases would have to be replicated at each end - or at least accessible to each. Now, since the config files used to access these databases have passwords, we have to treat them as compromised. So, even if the attacker has root on my relay, he won’t get a foothold in my local machine. Therefore, there must necessarily be two instances of the database - one on the relay and the other on the main servers (home/backup). Both instances must have different passwords, however, they must be replicatable. Ansible should be able to do this no problem.

The reason I say that it that I can configure the group_vars directory to hold variables by server, so that hell! each server can have it’s own password with the same database information.

Now, it’s not great that I have to put a database in a potentially compromised server - aka they have the password (being root themselves) to the database and therefore can know my users, but at least the passwords are hashed, and all of the emails are under my domain as well - it’s not like they couldn’t’ve brute-forced their way to this information, or socially engineered it out of someone if they knew. Also, for this case, the worst thing that could happen would be a vulnerability in the setup - which would be a bug due to be reported to the devs - or they take it to a journalist that I redirect everyone’s emails to my own inbox - whatever, I’m the admin - I can do that shit. (unless I contracted with them not to, in which case I’m boned - Rothbard would be disappointed)

Another thing that Ars’ article didn’t get into (besides MySQL) was the ability for the newest version of postfix - 2.10 - to use the LMTP protocol for delivering mail locally. I believe this works on a LAN as well, but I am not 100% sure. For the time being, I just need it to work for postfix -> dovecot. It seems very simple as well, so I’m not concerned, I just need to make sure to remember that all LDA protocol variables are deprecated and should be taken out.

Also, this is a good time to bring up sieve! Sieve is a tool that will sort email that’s coming into a Maildir format into separate folders. This is not spam-checking or AV scanning, but rather taking the header info (To/From/X-Spam/etc) into account when placing the email into folders. The rest is done earlier, actually - back during the postfix phase of sorting. That happens via it’s milters function. I’m sure I’ll have a hell of a good time getting the two to work together, and let the record show that if that’s impossible, I would much rather have spam filtering than AV scanning, as once again, I’m a linux shop here.

The hardest part will be getting MX records in the first place for my domain. I don’t immediately want to jump to hosting my own DNS servers, but I will it I have to. Either way, the pitfalls of big ESP’s not accepting dynamic IP addresses as valid servers and redundancy issues would prevent me from hosting them as well as the primary (relay) email servers at my own place - without getting a static IP that is.

Apart from that, it’s just about setting up conf files and one MySQL database. How hard could that be.

Thank $deity for Ansible

Postfix

So I’m demo-ing everything on my vmlab box, and I almost forgot how to make mx records. Here’s one for future reference that is in Services -> DNS Resolver under “Custom Options”:

local-data: "gitserver.vmlab. IN MX 10 email.vmlab."

Bad boolean configuration - soft_bounce = True

So I had in my variables postfix_soft_bounce = yes, which apparently returned a boolean of True - which is fine for python, but not for postfix. I put it in single quotes, and it still returned a boolean. It wasn’t until I put it in double quotes that it was passed as a string. Weird.

Single-quotes this time - right foot let’s stomp

To continue on about quotes, in Jinja2, if you are comparing to a string, you better make sure that you single quote the value, like so:


{% if variable == 'value' %}

Otherwise, it tries to find the variable value, which of course is undefined. But that’s an easy fix.

Address rewriting

So when I echo test | mail john@example.com in order to test the lmtp functionality, I see something weird. I have tail -f /var/log/maillog running in one tmux pane, and see the following spit out, with the hostname mail2 and the domain vmlab.

Oct 14 01:25:26 mail2 postfix/qmgr[14712]: 9F43B400A5DB: from=<user@vmlab.vmlab>

So, that’s weird. I took a look at a couple variables at my main.cf file, and saw myorigin = $mydomain. I then looked up qmgr’s docs and saw that it rewrites address to a standard form. The variable append_dot_domain “Rewrites user@host to user@host.$mydomain”. Yet already, there’s a variable append_at_myorigin that “Rewrites user to user@$myorigin.

So, first off, if sumbitting an email as user, it gets rewritten to user@$mydomain - translating in my case to user@vmlab. Then next, it calls the append_dot_domain to translate it into user@vmlab.vmlab by appending $mydomain…again. So we end up with user@$mydomain.$mydomain which is ridiculous. So our goal here is to have email in the typical format of user@domain.

So what I did was to specify append_dot_domain = no into my main.cf file. In fact, looking into the postfix docs, the default in Postfix < 3.0 the default is ‘yes’, while in Postfix > 3.0, the default is ‘no’. So it seems that postfix has seen the problem this creates and has switched the default the a saner one in future releases.

Damnit Jim, I’m a Sysadmin, not a DBA!

So I had some trouble putting together the SQL commands. Actually it wasn’t that bad, but I patted myself on the shoulder a bit too early. I had set up a for loop for the VALUES field using the Jinja2 templating that ansible uses. Ya know what, just take a look here:


VALUES
{% for domain in mysql_virtual_domains %}
    ( '{{ domain.domain_id }}', '{{ domain.domain_name }}')
    {% if loop.last %};{% else %},{% endif %}
{% endfor %}

First off, I had initially attempted to do an for i, j in ... style for referencing iterator’s values, as the actual variables are dictionary lists. Duh of course that didn’t work. It’s much cleaner this way. The worst part was the syntax. I didn’t know it had to be comma-separated parenthesis between each table entry with the last one being a semicolon. I had to wade through so many bogus error codes to get that, but by god I got it. And now it’s extensible as all hell. I was especially pleased when I found the loop.last variable. That saved me putting placeholder as the last static entry. Ugh - ‘Johnny DROP TABLES’ much?

HELO darkness my old friend

Since this will not be a strictly in-house/enterprise server, I’ll have to let go of the HELO restrictions and make it accessible to everyone - well, at least everyone who doesn’t attempt to HELO as the server itself. This is done by commenting out the smtpd_helo_restrictions variable in main.cf and setting smtpd_helo_required = no. I may be able to tweak it for the relay server, but I’m not even sure about that.

Trust fall-back_relay

Just in case my primary relay isn’t available, I want to still get my mail accepted and pushed out, so I’m setting up a backup relay. In order to do that, my main server has an extra setting of smtp_fallback_relay in it’s main.cf. This assures that even if my primary MX host is down, my second one can pick up the slack. I plan on getting a good VPS provider though, so I don’t think this’ll be a problem.

Dovecot

Postmaster

Missing in the ISPmail tutorial is the detail that postmaster_address must be specified in the file /etc/dovecot/conf.d/20-lmtp.conf or else you’ll be getting an error trying to send mail locally.

Mutt

When specifying -o smtpd_sender_restrictions=reject_sender_login_mismatch in master.cf, I didn’t realize that mutt would take that so personally. I attempted to send mail from my other host user2@client.vmlab, and kept getting the error:

mail2 postfix/submission/smtpd[25755]: NOQUEUE: reject: RCPT from unknown[192.168.3.111]:\
    453 4.7.1 <user2@client>: Sender address rejected: not owned by user user2@client.vmlab;\
    from=<user2@client> to=<root@mail2.vmlab> proto=ESMTP helo=<client>

I assume that the from address wasn’t specified by Mutt, but rather by postfix because of my client machine’s helo=<client>. In my .muttrc, I appended set from="user2@client.vmlab and it worked like a charm.

Ansible

Retry

All I really need retry to do is start over from the point of failure. It does the antithesis of that. I starts over from the beginning and goes up until the error occurred and stops after (if) that task completes. A total hogwash.

Other hosts’ vars

For this setup, I need to get other hosts’ variables like their IP addresses for instance - this for the mynetworks parameter in postfix for “trusted” networks.

So there’s a special variable that I remembered ansible had called hostvars. This variable can be used to reference, well, host variables. Paired with some pythonic flair, it’s really flexible. For instance, here I am specifying the FQDN of my relay server:

hostvars[groups['relay'][0]]['ansible_fqdn']

Beautiful, ain’t it? Well, it takes a little work to get to.

See, the information has to be already gathered or cached by ansible. Meaning, you’d either have to run through all of your hosts that you reference in this way and gather their facts before they’re called, or have that info cached somewhere. I did both.

As a new default of mine, I’ll be running through my common role in the beginning of every site.yml so that I can gather those facts. It looks like:

- name: Fact-gathering with 'common'
  hosts: all
  remote_user: root

  roles:
  - { role: common, tags[""] }

As an aside, this may be the perfect place to set up management_user when I get around to including that in common.

Also, there are some settings in ansible.cfg where you can specify that information like this can be cached for a specifyable amount of time.

gathering = smart
fact_caching = jsonfile
fact_caching_connection = ./.jsonfile_caching
fact_caching_timeout = 86400

86400 is in seconds if anyone thought that was in minutes…no

And this caching disables the fact gathering for the specified timeout. Not that it really speeds up the overall time it takes to run a playbook in any meaningful way, but whatever. It’s nice to reference the variables as well - every hostname has a specific json file in that directory containing the cached variables.

Hyphens vs underscores

Hyphens are just the worst. Ansible hates them, and have been the source of many a confusion. Just use underscores everywhere. The only excuse for a hyphen is if the destination filename on the remote machine has a hyphen in it. That’s it. Just don’t bother.

That also has the side-effect that everything in your playbook looks uniform. Which is never a bad thing.

Jinja 2 evaluation errors

For some host-specific and general evaluation errors, the error won’t spring up on you until you run the playbook. That means that you’ll be running and re-running the playbook after you discover that something’s not quite right and fixed it. Which is exactly why I say that the relay functionality is bass-ackwards than how it should be. The best way to go back to where you were is to have very good tags (I’ll talk about my methods later) and use them. For instance:

ansible-playbook -i dev site.yml -l relay -t postfix_config

I have four tasks in the YAML file that controls and is tagged with postfix_config. Also this command specifies to only run it on relay hosts - paring down the selection of hosts it will be running on.

I still say that’s too much effort that could be aliviated with a sane ‘retry’ functionality.

F*cking NetworkManager

'dict object' has no attribute 'ansible_fqdn`

This error popped up when my other two servers went offline and the cache timed out. No big deal since I can access them with virt-manager since they’re on ‘virtualizor’, but they’re erroring out weird, and I’ll have to troubleshoot that soon, but for the time being, I’ll just restart them until I can write the role to install network instead of NetworkManager. Doing that remotely will be interesting at best.

DNS

Ok…so there’s a lot here. I’ll break it up by records

NOTE: Bups is a copy of gateway, so that can just be copied from gateway.

A

An A record maps a domain name to the IP address (IPv4) of the computer hosting the domain. Simply put, an A record is used to find the IP address of a computer connected to the internet from a name.

I’ll need a couple of A records for this. One for gateway and internal.

HOST                      |   TYPE    |   Points To   |   TTL
gateway.hobbithole.blue.         A         <IP ADDR>       1h
internal.hobbithole.blue.        A         <IP ADDR>       1h

CNAME

A Canonical Name record (abbreviated as CNAME record) is a type of resource record in the Domain Name System (DNS) used to specify that a domain name is an alias for another domain, which is the “canonical” domain.

This makes www.hobbithole.blue redirect transparently to gateway.hobbithole.blue. This means that the url in the web browser’s address bar will not change when others navigate to my site, but they get redirected to the correct IP address. Also for shits and gigs I redirected client.mail.hobbithole.blue to internal.hobbithole.blue and server.mail.hobbithole.blue to gateway.hobbithole.blue.

HOST                            |   TYPE    |       Points To            |   TTL
www.hobbithole.blue.               CNAME       gateway.hobbithole.blue       1h
server.mail.hobbithole.blue.       CNAME       gateway.hobbithole.blue       1h
client.mail.hobbithole.blue.       CNAME      internal.hobbithole.blue       1h

MX

A mail exchanger record (MX record) is a type of resource record in the Domain Name System that specifies a mail server responsible for accepting email messages on behalf of a recipient’s domain, and a preference value used to prioritize mail delivery if multiple mail servers are available. The set of MX records of a domain name specifies how email should be routed with the Simple Mail Transfer Protocol (SMTP).

Since only gateway will be sending or receiving mail, we’ll only need an MX record for that, not internal

HOST                      |   TYPE    |   Points To   |   TTL
gateway.hobbithole.blue.        MX         <IP ADDR>       1h

PTR

Pointer to a canonical name. Unlike a CNAME, DNS processing stops and just the name is returned. The most common use is for implementing reverse DNS lookups, but other uses include such things as DNS-SD.

My home cable provider is giving me grief that they cannot change my PTR record, but I’ll set them straight soon enough. This is really only pertanant for my Ansible script as, for my home server, dig -x $(myip) now returns <their internal naming scheme>.try.wideopenwest.com.

NOTE: I have myip aliased to return my public ip via the command dig +short @resolver1.opendns.net myip.opendns.net

Technically, the PTR record is registered in a separate Master Zone. That is actually a Reverse Master Zone if we’re being technically correct. But I’ll just stick to divvying it up by records here. But it is worth mentioning that my DNS Zones are each labled <Reversed IP Address>.in-addr.arpa.

HOST                                |   TYPE    |       Points To            |   TTL
<Reversed IP Address>.in-addr.arpa.       MX         gateway.hobbithole.blue       1h

And

HOST                                |   TYPE    |       Points To            |    TTL
<Reversed IP Address>.in-addr.arpa.       MX         internal.hobbithole.blue       1h

I hope this way to override them, but we’ll see.

CrownCloud

My VPS provider CrownCloud responded to my inquiry on #crowncloud and said that as long as I put in a ticket, I should be able to change the PTR record. That’s one down!

SPF

(Sender Policy Framework) is a DNS text entry which shows a list of servers that should be considered allowed to send mail for a specific domain. Incidentally the fact that SPF is a DNS entry can also considered a way to enforce the fact that the list is authoritative for the domain, since the owners/administrators are the only people allowed to add/change that main domain zone.

HOST             |   TYPE    |       Points To                         |    TTL
hobbithole.blue.      TXT         v=spf1 include:hobbithole.blue -all*      1h

DKIM

DKIM (DomainKeys Identified Mail) should be instead considered a method to verify that the messages’ content are trustworthy, meaning that they weren’t changed from the moment the message left the initial mail server. This additional layer of trustability is achieved by an implementation of the standard public/private key signing process. Once again the owners of the domain add a DNS entry with the public DKIM key which will be used by receivers to verify that the message DKIM signature is correct, while on the sender side the server will sign the entitled mail messages with the corresponding private key.

HOST                       |   TYPE    |       Points To            |    TTL
_domainkey.hobbithole.blue      TXT         DKIM-specific-text          1h

DMARC

DMARC (Domain-based Message Authentication, Reporting and Conformance) empowers SPF and DKIM by stating a clear policy which should be used about both the aforementioned tools and allows to set an address which can be used to send reports about the mail messages statistics gathered by receivers against the specific domain

HOST                       |   TYPE    |                            Points To                              |    TTL
_dmarc.hobbithole.blue         TXT         v=DMARC1;p=reject;pct=100;rua=mailto:postmaster@hobbithole.blue      1h

Something to tune here would be the pct and the p values. For pct, I would initially start it off at ~20 so that only 20% of mail is subject to DMARC processing. At that point, any problems should be able to be progressively eliminated from the system before turning DMARC on for all mail (pct=100). Then for p, I would set it to quarantine first, and then reject after a bit of working out the kinks.

Getting Public Suffix List

If in /etc/opendmarc.conf, the PublicSuffixList option is specified, the file has to be there, otherwise it will refuse to start with opendmarc_policy_library_init() failed. This file has to be readable by the user opendmarc.

check-auth@verfier.port25.com for SPF and DKIM

Databases

Here’s a quick reference if you’re ever debugging a database and want to manually inspect the database just to make sure you know what’s there:

# Typically there is a `-p` parameter passed that precedes the password you
# would typically give this command. In dev, I don't use passwords until I
# need them.
Login: mysql -u root

view Databases: show databases;

Show tables in `database`: use database; show tables;

Show users: select host, user, password, from mysql.user;

Delete user: drop user 'user'@'host';

Check Permissions of current user: show grants;

Check permissions of other user: show grants for 'user'@'host';

https://ssl-tools.net/mailservers

Unbound DNS TXT records

For my vmlab, I’m using pfSense’s Unbound DNS Resolver. This has the ability to specify custom records using the key local-data: in the “Custom Options” section of the configuration. If you’re sshing into the box, just issue a find / -name "unbound.conf", I think I remember that there was one under /var/run, but it’s easier to do it in the GUI.

Anyways, since TXT records need to be quoted with double quotes in the raw text file, and the local-data: setting requires them to be quoted as the value in the key-value store that it uses, you need to surround the entire thing in single quotes, and then surround just the value to put in that TXT record in double quotes.

local-data: '_dmarc.mail-relay.vmlab. IN TXT "v=DMARC1; p=none; pct=100; rua=mailto:thing@vmlab.com"'

References: