I'm on 12 years of self hosting email and counting. Once every so often, I do end up being blocked, usually by Outlook and once by Yahoo. I'm in their 'sender program' and they still don't actually bother to contact postmaster@, but a few emails is usually enough to unblock the block within 24h.
Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do.
Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.
I completly agree with you. While Hetzner is my actual neighbor as their headquarters is right in the neighboring town, I still use a server at very small scale provider. I have no problem with my email server. I receive some spam here and there, mostly from Russia. But I immediatly block the according IP addresses for some time.
For years I avoided to use any external service to decide whether its Spam or not. But about 2 years ago I started to rely on some of the external Blocklists.
Till today I have no problem sending Email. Even as I don't use DKIM or DMARC.
If you are using external blocklists, you are perpetuating the problem here. Small mail operators end up on blocklists through no fault of their own, and it sounds like your server would reject them just like the big servers do.
To those that still persist, there is a page I found recently that helps you make sure that your outgoing mail is configured correctly: https://appmaildev.com/en/dkim. They generate an e-mail address, you send a mail to them, they do the check and display results (not affiliated, just a happy user).
I've also been self-hosting email for years, and the only deliverability problem I've ever had has been with AT&T. If I try to send something to an AT&T customer, I get an automated "your message has been eaten" notice, and following its directions accomplishes precisely nothing. At this point, I can only guess they're hellbanning the IP block in which my VPS resides, because it does not show up on any public DNSBLs.
Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.
I think we are currently in a phase because of phishing problems. Corporate filters are currently turned up to eleven, but hosting your own server is still very well possible. SPF/DKIM/DMARC helps, but isn't required. You really think the corps that do host their own servers have set that up? It is far more widely spread by enthusiasts that host their own servers.
We also should keep mail for pseudonymous and anonymous user authentication. There are a lot of threats to the free internet right now and user account consolidation is one of it. I agree that people should keep hosting themselves. Someone blocked you? Their loss, let them rant to their internal IT about it. Corporations or institutions that use Google as a provider should be seen with scepticism.
I use a small datacentre in my country, actually not far from where I live. DKIM/SPF are independent of the provider. The easiest way to understand is to consider how receiving works. If I'm getting an email from hnemail.example, the first thing I do is consider the IP address. Oh, 257.257.257.1? Ok. So I then ask DNS "what is the SPF record for hnemail.example?" and it returns
v=spf1 mx -all
This tells me only to accept emails from 'MX' entries for that domain. So I query 'MX' against the DNS server and I get a list of A records, which I can get IPs from. If the IP is in the list, spf passes. Otherwise it fails, mark as spam.
For DKIM, when the email was sent it was signed with a key by the sending server. It is identified by a UUID in the incoming email. So the receiving server again queries DNS for TXT <UUID>._domainkey.hnemail.example and receives the public key as a response. Signature verification passes? Accept email. It fails? Mark as spam.
This doesn't have a lot to do with IP reputation. This is different. If you are a very large email provider, you might develop custom spam filters. IPs are allocated to 'autonomous systems' i.e. who actually uses them and hands them out to users, and depending on the business you might make some decisions about reputation. For example, if the IP address is part of a consumer ISP block that is handed out to users of broadband, chances are high that if they're sending email, it is probably a Windows PC compromised by malware.
Similarly, you might decide some ASNs are better than others. Some hosters are more liberal in what they will accept, such as VPN endpoints, tor nodes and such and as a consequence of this more spam comes from these ranges.
Rightly or wrongly, larger email providers try to add these extra filters to the process to protect their users from spam. This obviously sucks if you are genuinely trying to run an email server on your symmetric home fibre connection with a dedicated IP, but that's the world we live in.
I can't make any general statement on which providers might be best, and some people will have no issue whereas others will find themselves unable to send anything. I don't work for Outlook/Microsoft or Google and never have, so I don't know exactly what rules they use, and in all likeliness they shift constantly depending on spammer patterns. I can only say I've found running from a small DC to work pretty well.
Find a clueful small provider, local to you if possible. On huge providers like Hetzner and DO, you are guaranteed to have spammers as neighbours some of the time, even if the provider rapidly shuts them down. On the other hand, a good-quality small provider may rarely if ever host spammers.
I've been hosting my mail for 20+ years now, with minor issues. I guess I've been lucky.
Reading the comments here makes me incredibly sad. Every answer that tells me to use a provider misses the point. The Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers. I should be able to run my own E-mail.
The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Either we solve the spam problem, or things will get worse. The big tech corps won't solve it for us.
I have also been self-hosting email for 15 years and only had couple of problems at the beginning, mainly until my IP got enough reputation. I have been hosting it on a bare metal Supermicro server in a proper datacenter, though. It has reverse-DNS, SPF, DKIM, TLS, MTA-STS and even DANE with DNSSEC (on a self-hosted BIND but that's another story). It is implemented using Exim, Dovecot, SpamAssasin, DNSBL and Roundcube with OpenLDAP auth. I can recommend this awesome hand-on guide provided by Netherlands Domain Registration Foundation as a basis of a nice configuration https://www.sidn.nl/en/news-and-blogs/hands-on-implementing-...
I had some troubles with IMAP search. I set up CLucene, it was easy and enough for me (no need for Java Lucene). It just took me a long time to figure out why it wouldn't search a domain part of email addresses. It just required to set up the tokenizations in such a way to split words also on @ character, i.e. don't consider a full email address as a word. :P I also had some troubles with OpenLDAP until I finally decided to read the docs and examples there properly. Since then I have been using this setup happily and it appears I will continue to do so! I also share the LDAP with NextCloud btw.
> I have been hosting it on a bare metal Supermicro server in a proper datacenter, though.
This is a key difference. Many people who have a bad time self-hosting mail set up with a minimum-effort major provider like Hetzner or DO — you're guaranteed to have spammers as neighbours some of the time, and other networks will behave accordingly when deciding whether to accept your mail.
A real server in a proper datacentre is great — I do the same — but as a lower-cost option a virtual server with a small, clueful provider who knows their customers also works just as well.
The problem is that collectively we love 'free' (at the point of sale) so much that we'll gladly allow gmail to just walk in and own almost the entirety of the email infrastructure. Then later we realize this gives them the ability to unilaterally make the rules, and we complain. But it's too late.
Also 20+ years, only very minor issues. The spam situation has gotten much better over that time, too. This topic comes up every so often on HN and I feel like the "never self-host, it's guaranteed to be a disaster, you have to use a centralised provider" crowd are just louder. Self-hosting my mail isn't something I think about much or talk about much. It's so obvious to me that it's worth doing, and it's extremely low effort.
I think the problem was this guy was providing email accounts for other people. Other people who probably reused their passwords and had their email in the same DB in the apps they used. That DB was compromised. Hackers got an account to send spam. Then the domain was blacklisted.
He alluded to this in
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned
I understood that sentence as something that can happen to you even if you believe it won't.
> Please believe me. My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.
Wow, good catch! He really buried the lede there -- "I kept getting blacklisted because my servers kept sending out spam." I don't know what to tell you pal -- spam is a serious problem, and if your users are sending it out, you're contributing to that. If you can't keep your users in line, other providers don't have any choice but to blacklist you.
> The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Is this true? Even for consumer stuff, the only Google product besides search that seems widely popular is Gmail.
For enterprise, Microsoft is pulling in billions from office products. Without digging into their statements I know from my own experience that enterprise email services constitute a core piece of this.
For consumers, switching costs are low (mostly just inconvenient to go through all of your accounts) but not relative to the drawbacks of using corporate email providers which is zero unless you value your privacy.
>the Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers.
any community that grows large enough needs some mechanism to manage trust, this is a universal issue. The early internet was more permissive and less differentiated simply because it was smaller.
The big corps do an alright job at managing spam given the sheer size of the problem, and more importantly you don't just need to solve spam, you need to do so economically, because for your system to stay distributed the nodes need to do the job competitively.
Given that there's intrinsic benefits to managing these things at scale that's not really realistic, in large systems you're always going to have division of labor and stratification for that reason.
gmail silently drops emails (while reports smtp 250 accepted) - they could just as easily report that it's blackholed. spammers already do get through their fancy AI filters.
microsoft proactively blocks half of the world, rejects the incoming mail, and sends the sysadmin on a wild goose chase to get the IP/domain/whatever allowed. then you diligently register, and wait. and then no signal from them, and the problem still persists.
so big corps, small shops, everyone and their dog flocks to the good old microsoft/google duopoly.
and at this point if someone asks what to use for email knowing ... well, it's hard to not recommend folks to just use google workspace and have some kind of backup ready for when G bans their whole account just because.
1. Require the email receiver to add the sender to their contact list, then put all other emails in "Junk". The UX of this could be quite good nowadays with smartphones and QR codes. The changes to email apps are minimal:
- 1) When a user opens a "mailto:" URL, the email program shows the normal "send email" screen with a "Just add to Contacts" button at the top.
- 2) Email program has a function to show a QR code with the user's "mailto:" URL.
2. Add support to the SMTP protocol to let the receiving server demand that the sender make a small "postage" payment. This will require a privacy-preserving micro-transaction system. I worked on one that doesn't use crypto [0].
Here are some ideas for improving the current system, but not actually solving spam:
3. Create a protocol for automatically reporting emails marked as spam back to their admins. Servers will only accept signed emails. Then servers can do lots of things like:
- Mark sent emails as "Receiver flagged this as spam" and inform the user.
- When an account begins sending spam, rate limit it, disable it pending password reset, or alert the admin.
4. Add SMTP responses that mean "rejecting because your server sends spam" and "rejecting because that user sends spam". Servers can mark sent emails as "Receiver rejected this email" and inform the user.
5. Build a shared spam reporting system that accepts only signed emails and supports searching by email address. Receivers can use it to identify compromised user accounts and reject their email. Senders use it to identify receivers acting in bad faith (reporting ham as spam). A centralized version would be straightforward. A decentralized version would be a challenging project.
6. Add support to the SMTP protocol for reporting rate limits and cooling-off times to email senders. Admins of large shared email systems can feed these metrics into a monitoring system and receive alerts of problems early.
I think the real problem is how bad most e-mail interfaces are (makes you want to use it less) or that there is not a dead simple way to do it other than a provider. It should be easy enough that I can download an app on my phone and be sending messages.
I think matrix might get rid of a lot of the infrastructure barriers, but I don't have much experience with it.
Maybe there needs to be a self-hosting association/union that self-hosters can join? It could advocate for adherence to open standards, and an equal standing for small servers. It could also be a repository of advice for current best practise in small server administration and configuration. Should it be under the auspices of an existing group such as FreedomBox?
G'Day Femto
Over the past 15+ years I have joined a number of groups like a web site 'web hosting talk' and all of them jumped all over my privacy and passed on my details to spammers / hackers.
I know what you are thinking -> how would he know that ?
Well its really simple I use DedicatedEmailAddressing ( DEA ) and our system tells me when a 3rd party tried to deliver a spam message using one of these DEAs.
Some of our customers charge suppliers for a new replacement DEA when they find the supplier has 'leaked' :)
Yeah, the host I use sells on webhostingtalk and I'm 99% sure they're selling my info on. My cPanel username there is the only place that I have that username, and that's what a massive chunk of my spam is directed to.
The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.
Same philosophy for exposing a your personal blog of html files or content like mp4 videos. The sweet spot is to focus on buying a domain name you control. Then let Amazon S3, or Cloudflare, Hezner etc, host your html or mp4 files.
I quit self-hosting email at home over 15 years ago. It's just not something I want to babysit anymore because I have other things to focus on. As long as I control the MX record on my own domain, that's really all that's necessary.
There is also a happy medium. Host your own MX servers but use someone else's SMTP servers. You have complete control over the incoming mail but dodge the filters by using the established business for sending mail.
> Host your own MX servers but use someone else's SMTP servers.
Yes. My outgoing email goes out via Sonic's SMTP server, with the SPF records to allow it to have a source address of my own domain. Incoming email goes to my own domains and gets forwarded.
This seems to be trouble-free. The domain is on a cheap shared hosting account. I'm not running a server. I own the domain, and not though the hosting company, so I can switch to another provider if necessary. In 27 years, I've had to do that twice, because the hosting provider went out of business.
This is easy to do, and I don't have to deal with Google. I don't even get much spam. All the spammers seem to be targeting the big services now.
I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer.
It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads.
The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
Third party email providers, the so-called "established businesses", get a free pass as sending SMTP servers that are accepted by almost all receiving STMP servers. Everyone just assumes everything coming from those SMTPs is legit.
Establishing this "legitimacy" and getting the "free pass" is difficult and some have suggested, the third parties may employ anticompetive tactics.
However, IMHO the receiving SMTPs is a different issue. Why do we let these third parties receive and store our email. (Why do our homes have their own mailboxes. Why not use a "P.O. Boxes" instead.) Eventually we could move away from letting third parties control the receipt of our mail. Neither "POP3" nor "webmail" was part of the original concept of email.
Today, it is easier than ever to set up overlay networks where we can assign our own IP addresses and run our own SMTP servers that can communicate directly with other SMTP servers on the overlay network. These networks are not open to the world, they may only be open to people we know. Much of our mail is between people who know each other, e.g., friends, family, colleagues. Or businesses that we contact first. We can separate different social and business networks on different overlay networks.
Anyway, the sending and receiving of mail can be separated. We do not need to let a third party control both.
Adding to the happy medium is to teach your friends and family to use Thunderbird so they can easily GPG encrypt [1] their emails keeping the nosey email providers off the email body. Also teach them to use the IMAPS (TLS) endpoint for their mail provider, usually port 993. There are probably simpler how-to's with pictures, I just do not have any of them handy.
You are totally missing the simple fact that the number of blessed email providers to choose from is slowly going down. I've seen ISPs with thousands of clients to give up and move the mailboxes to large players simply because their clients' email was ending up in the spam so often that running the support has gotten too expensive.
Indeed. I host my domains with dreamhost and it turns out that outgoing mail from their servers will get marked as spam by Google. The exact same mail sent through a gmail address (whether under gmail.com or a custom domain) will be delivered no problem (although after the sudden closing of legacy free email, I found that emails that I had been sending via gmail with my own domain that had been getting blocked by spam filters were being delivered when they came from a gmail.com address).
I just randomly looked at 8 different emails in my inbox. All of them were from different email providers (except google which was there twice). There's hundreds or thousands of email providers you can chose from.
iphmx 1
google 2
kornet 1
linkedin 1
secureserver 1
amazonses 1
self hosted university email 1
Or maybe it's an overly competitive protocol? Like playing Monopoly. Even a neophyte can end sweeping the game despite not understanding any of the underlying mechanics that drive the game's outcome. Those remaining mail providers are also fighting back the insanity. They 'just' won.
Yeah, I've been doing exactly this for over 20 years. The only problem I can recall is related to the fact that the hosting provider uses a single SSL cert for the machine that hosts my domain (and many others, presumably), so of course the cert doesn't match my domain name. It's pretty easy to work around, and I only have to deal with it every few years when they do a hardware upgrade, which sometimes means moving my domain to a different machine.
Certs for MX servers are supposed to have the MX as subject or SAN, not your email domain. It's important when the sender enforces encryption with a valid cert (e.g. MTA-STS, or config in the mail server, or many hosted solutions like Google Workspace also support enforcing this for selected or all domains).
Example:
example.com. MX aspmx.l.google.com.
Cert should have aspmx.l.google.com as subject or SAN.
I agree with this assessment, and it's what I do. There is still the single point of failure of losing your domain due to a hostile registrar or mismanagement e.g. allowing registration to lapse.
Ideally there would be some a decentralized permanent domain registry keyed on certs (I know these exist but have not been adopted), or at least a fallback domain you could configure somehow in case you lost control of your main domain.
For me, the issue wasn't that I had to fix it frequently, but that when I did it was urgent, stressful and disruptive. Eventually the VDS I was renting had a fatal HD crash and I gave up.
> simply own your domain name and point the MX record to whatever hosting provider you want
That's not necessarily a sure cure, depending on the hosting provider. RoadRunner (Spectrum / Charter) in the US and Shaw in Canada won't deliver emails from my domain hosted at Runbox.com (or sent directly from the runbox.com domain.) Spectrum's bounce message references an error code that translates to "Spectrum limits the number of concurrent connections from a sender, as well as the total number of connections allowed. Limits vary based on the reputation of the IP address. Reduce your number of connections and try again later."
> point the MX record to whatever hosting provider you want
You can even have a hybrid solution where incoming mail goes directly to your self-hosted server and (some) outgoing mail is relayed through a third party.
"The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name"
You would think. The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life(at least). Otherwise, as soon as you lose your domain you lose ownership of your email. Any one that has control of the domain has control of your e-mail.
This dawned on me after a place I worked at reactivated the email I used for work when I worked there. It has my name but I have 0 control over it. Lucky for me I never used the email for things other than work so it's not a big deal. It is still bothersome that they can do that and I have no say so on its use.
> The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life
But that's true for any delivery endpoint in any medium, including physical mail and phone numbers.
It's pretty easy to set up auto-pay for domain names so that all you really need to do is keep the billing info up to date. After that it all runs on autopilot.
I like the idea of having a node I can just plug into my network. I run my Urbit on a Mac mini with tailscale (which works great).
The core of what he writes about is correct though, email failed to be truly peer to peer (as imo all non-urbit-like federated systems will) because of the incentives that lead to centralization (spam, difficulty of running nodes, etc.)
We’re suffering the consequences of the local max we’re trapped in currently because of this. The promise of the 90s internet was a bunch of people using decentralized services they controlled - instead we're primarily thin clients connecting to a small handful of powerful ad companies. We're mostly serfs [0] allowed access if we give up our data for ad targeting, follow EULAs nobody reads, and don't say anything the company earls disagrees with.
Controlling your domain/mx is the most valuable thing.
Email reception has not been a problem for me so I enjoy having my mx pointing to my own mail server. It gives me more control and it requires very little maintenance.
If I was going to outsource anything the first choice would be outbound. My email system does everything right for reputation protection. All senders are authenticated on secure connections and the senders are people I trust. Nothing bad gets sent and my static IP is on a reputable server host and is not on any public black lists. I maintain SPF, DKIM etc. If someone decides to block my IP for no reason by accident or on purpose there is very little I can do or care to do anymore.
I have an alternate path for emails setup via a server I host elsewhere ready to go. If I run into a widespread delivery issue due to massive indiscriminate ip blacklisting of my provider I can enable it. That has happened once in 20 years. If delivery gets too hard I will change that policy to send all outgoing emails through a commercial smtp delivery service and let them deal with the problems.
This is what I do. The downside is that a lot of email providers make it really difficult to set up 3rd party clients outside of their own clients. I've struggled to get mutt to work with a lot of email providers because they use their own auth mechanisms.
I am ashamed to admit that I have no idea how email works. Is there a dumb down explanation of what are the moving parts and how you can achieve that sweet spot?
You send your email through a client. That client then sends (transfers / SMTPs) that email through an MTA either bundled with it or provided by your mail server.
The MTA parses the message, figures out who it needs to go to (To, CC, BCC headers), figures out what servers receives mail for those recipients, and then transfers (SMTPs) it to the server.
What OP is referring to is that the MTA essentially does a DNS lookup for the recipients domain for a record of type MX (Mail eXchanger).
If you own the domain you have complete control over where that mail goes: you own the MX record.
Email is complicated but the jist is there are relays and mailbox/mail-exchanger servers (and email clients). And a million different anti-spam measures that make making sure people actually get you email difficult.
When you send an email you mail client contacts you mail-exchanger (MX) and drops the mail in your outbox. Then the MX will look up the MX record for the domains in the TO field and attempt to send the email to it using SMTP.
The first thing receiving server ussally does and look up the IP of the sending server and see if it's on a spam black list. If it is it will probably just drop the connection. Then it will look up the SPF record for the domain in the FROM address and see if the sending server is allowed to send that mail.
Larger email services will have an internal 'reputation' scoring system that will use data from reported spam to figure out what IP's and domains are sending spam emails and filter them out. They'll also look at if it's a residential IP, in an IP pool for a major cloud provider etc. Each provider has a different system and it can be really difficult to get a provider to trust your IP or whitelist your IP so you can make sure that your mail actually gets to who you're trying to send it to.
Relays are pretty simple they'll take email from one place and send it to the recipient. The mail exchanger usually has a built-in relay. A lot of people will use a third party relay service so they don't have to worry about managing the reputation of the IP of their sending mail server. They'll just add an SPF record for the relays service to their domain. And then configure their mail exchanger to send all outbound mail to the relay and then the relay will do the MX lookup.
A lot of mail services will also have their MX records pointed at relays. These inbound relays will often have a lot of those anti-spam services bolted onto them and they can also be used for load balancing to make sure that the service is always able to accept mail even if it doesn't make it in the mailbox immediately.
Typically, if you sign up for an email account, you get an email address like skywal@gmail.com or skywal@yahoo.com. Alternatively, if you own/host skywal.com, you can have an email address like skywal@skywal.com served from a computer in your home.
The "sweet spot" is combining the two, where you own skywal.com, and have your email send/receive through Google or whoever. Then, if Google decides to ban you, you just register skywal.com with another company who provides that same service, and you keep your same email address.
Let's imagine I'm sending from user@zahllos.example to user@skywall.example. I'm doing it from say Thunderbird or Outlook, and you're using the same.
I need to send, and to do this I typically use an SMTP server. This is something configured, probably on zahllos.example, maybe with the domain smtp.zahllos.example. My mail client contacts this and 'logs in' with my details, then transmits the email message I want to send to this server. The server says 'right fine' and closes the connection.
At this point, the message is in a mail queue ready to go. The SMTP server then does a DNS request for the MX record of skywall.example. Let's say this is 'mail.skywall.example'. My server, smtp.zahllos.example, then connects to `mail.skywall.example', also speaking SMTP, and says "hey, I have this message to deliver".
It is at this point that mail.skywall.example can decide to do some things. It might check SPF, so it will query the SPF record of zahllos.example to find the list of servers that may send email for that domain. In this example, let's assume smtp.zahllos.example is in the list. Great.
It may then also check the mail headers for a signature, called a dkim signature. My server signed the message before sending; mail.skywall.example can query <uid>._domainkey.zahllos.example and find a public key (or not) and check that this signature matches (or not). Again, let's assume it matches.
It might also check something like TXT _dmarc.zahllos.example to see what my DMARC policies are. If I have something like v=DMARC1;p=reject;sp=reject;pct=100;ri=86400;fo=1;aspf=s;adkim=s;rua=mailto:postmaster@zahllos.example;ruf=mailto:postmaster@zahllos.example;" this tells you I'd like you to outright reject anything not matching policy, that I expect everything to match SPF and have DNS signatures, and you can send reports if you support that to postmaster@zahllos.example. Your server can then enforce these checks as it likes.
One of the first things that will happen is that my server will announce itself via an EHLO statement. An obvious check to do is to check that the sending IP actually matches smtp.zahllos.example. by querying 'reverse ptr' records.
Your receiving server will also likely hand the message over to various spam-checking tools for analysis, such as against DNS blocklists and so on. Larger providers likely have much more sophisticated infrastructure here. Ultimately, you're going to do one of a few things: 1) deliver to inbox or apply user-specified rules and deliver to a folder; 2) deliver to junk (which is typically just another folder, but treated specially by clients), 3) reject, and tell smtp.zahllos.example you don't want the email.
Once the email passes through the smtp dameon, assuming either 1 or 2, it then gets stored somehow and in some way. I'm being a little bit vague here, because 'it depends', but in the simplest scenario, the smtp daemon will write the message to an mbox or maildir-style format. More complex setups definitely exist, indeed, there can be multiple layers of servers doing analysis on separate machines, but for simplicity, mail.skywall.example is one VM that makes its decisions and the result ends up in /var/mail/user@domain/ or some such.
A key aspect of this step that makes email very nice is that if smtp.zahllos.example cannot, for some reason, reach your server now it will queue the message and try again at set intervals. You can reasonably safely turn off mail.skywall.example for a couple of hours.
Another aspect is that you can have multiple MX records where you are prepared to accept email, with priorities. So if you can't accept at one address because the server is down for maintenance, another will accept.
So, now you've technically got an email, but you don't know it. So you open thunderoutlook, and you connect to an IMAP server imap.skywall.example - in our example let us assume this is really the same thing as mail.skywall.example. The server checks you are really you with your credentials. At the backend, this is just another daemon that knows to read /var/mail/... and find new messages; it finds one, downloads its headers and displays it on your screen.
Since we're in a slightly more modern world now, in the case your client was already open you might have an "idle" connection with the server at all times, in which case it can push the message down to you.
In the case of webmail, it is really all the same thing, except you point your browser at a webpage, and that webpage communicates with the servers instead of your client communicating directly. Open source webmail might even use IMAP underneath; things like Zimbra use their own java mail agent, while Google is entirely custom.
That might seem complicated, but in the end it isn't: between two domains at the 'edge', in the end, there's an SMTP conversation. One sending server tells a receiving server it has mail for delivery, and it finds that server by asking DNS where it is. The receiving server may do a bunch of checks against DNS also before making a decision on what to do with the email.
> The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.
So, mandatory third-party doctrine/warrantless surveillance.
It’s a misnomer to say you can “own” a domain name. You cannot. The better word would be “rent”. Because of that you should think about what could happen if your domain name registration lapses and someone else registers it.
I think everyone knows this who owns a domain. Probably people say this because (like I do too) they have all the control over the name. If I rent an apartment, I do have to consult the owner to make major changes.
Most emails are still going through servers owned by Microsoft or Google, so what does self-hosting email accomplish in reality? Some government entity likely has warrant-less access to most of your emails regardless. I like email as a way to communicate, but speaking pragmatically, it’s just not a secure means of communication in 2022.
I support the author but let me tell you a counterargument I don’t think he devotes enough to:
Spam is a real issue.
The amount of spam emails which get sent are absurd and likely orders of magnitude more than non-spam. And spammers do a lot to mimic real emails, including just hacking legitimate addresses and adding them to botnets.
Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
And even if it really isn’t that bad and never will be, the big companies and those arguing against self-hosting will claim it is. They don’t want to allow a relative few self-hosted email servers in exchange for much more difficult and less effective spam detection. Forget Gmail and Outlook, why not just use Fastmail or Protonmail?
If you want a legitimate argument for self-hosted emails you need to address the spam. It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
But this article doesn’t mention that org, in fact doesn’t say much at all about spam besides “keep existing spam-prevention because it already works”. But you should at least explain why. Because spam is a legitimate argument for big-co forming an oligarchy that’s not just “so they can make more money”, and it’s the main argument that big-co uses.
email spam is not a real issue. big uncooperative gigacorps fucking it up for everyone is.
responsibility for spam (and other kinds of abuse) can be delegated via simple reputation scoring for netblocks, sender authentication and a proper feedback mechanism along the chain.
when the user clicks "spam" gmail already uses that to train their fancy AI and if you are not a small nobody, then they already alert you that ooops you sent something spammy via a feedback loop [1]. (see also how Mailchimp proudly claims they work with big integration partners like gmail ... https://mailchimp.com/help/how-mailchimp-prevents-and-handle... )
whitelist clearinghouses exist [2] but they are not terribly useful, because there's most of the signal to use for reputation is hidden :/
> responsibility for spam (and other kinds of abuse) can be delegated via simple reputation scoring for netblocks, sender authentication and a proper feedback mechanism along the chain.
Are there any real-world examples of this? I know other decentralized networks like Tor and BitTorrent have some sort of reputation and feedback system. What type of "spam" do they deal with and how well do they deal with it? Are there any systems more similar to mail with mechanisms to prevent spam?
Spam is a serious issue, not just in email, not just in decentralized systems - it's one of the main issues in technology today. If there's a solution, even a proof-of-concept in a smaller system would be great
> And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
> Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
It is as good as it is exactly because of the requirements author finds tedious.
> It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
They do exist but there are a bunch of problems with that. People have varying definitions of spam, getting paid to whitelist someone creates a perverse incentive or not getting paid will quickly overwhelm the organisation.
Things could be better if we could enforce sender authentication (SPF/DKIM). It would assign a direct cost to getting your domain blacklisted. But if nobody is taking away rest of the spammer's domains (or keeps selling them new ones), they'll continue.
Spam is an issue, but it's not the one that impacts me the most. I rarely get true 'spam' in my inbox.
What impacts me the most is a never-ending problem with Gmail classifying messages as spam that were actually important to me. Time sensitive announcements of meetings or events, for example. Many are coming from senders I've been receiving email from for years.
Gmail is convinced that almost every technical-related email mailing list I'm on is a spam source, despite my constantly going into my spam folder and telling it dozens of messages from those mailing lists are not actually spam.
Meanwhile, what I do get is a barrage of promotional emails - what I consider very much to be spam - from corporations that at some point have had my email address and now email me multiple times a week. Those sail right past the spam filter into my "promotions" folder and accumulate...
o/t: Could you add a close button to that ethical ads element? It's obscuring a decent chunk of the screen on mobile and I can't figure out how to dismiss it
The ecosystem can definitely use more work, but I've found maddy to be really good, because it's all in one, makes very reasonable choices, and is easy to configure.
I put maddy first because it's what I personally use (without dovecot) -- I make sure to back it up as well, and that's enough for me/most people, I think. It's the closest to the whole package which doesn't rely on.
Oh you know what, I completely forgot about iRedMail:
It was a huge mistake for email receivers to take on the cost of filtering spam. Of course given the evolution of the internet and email it is easy to see how that mistake happened. Nobody had a crystal ball. But the only solution here is to raise the cost of sending email to the point where spam is no longer profitable.
It seems like one solution is to bcrypt hash (or some similarly expensive algorithm) the email and include the hash in a header. Of course you need to hash per receiver or a spammer can just hash it once and spam away.
The receiving client hashes the email and compares the result with the value in the header and discards emails that don't match.
You'll never get industry buy in though - the FAANG companies don't want to pay that cost for their semi-legitimate email. They prefer to keep that cost externalized.
I believe there have been attempts at something like this, but it clearly never went anywhere.
This indeed looks like a good direction. A decade ago Freenet (not sure if it still exists?) had a problem with spam on its equivalent of USENET. It was pretty bad, until they changed the protocol so that it's the sending node that keeps the message, which is then pulled (or not) by the recipients. It made a lot of sense to me: I'm the one sending you the message, I want you to see it, while you don't even know whether you want to get that message. So it's me that should pay for storage and propagation of the message in terms of bandwidth and disk space. Not sure how it turned out in the end, but the approach seems right to me. It's like phone calls: it's the caller that pays the cost, not the one receiving the call.
Wow that's ingenious - and obvious when you think about it. I guess evolving the email standard is a much bigger obstacle though, no matter how clever the proposal.
Something different: a hash which is expensive to calculate but cheap to verify. E.g. calculating a string of bytes to append to a hash stream on order to produce a hash with a certain number of leading zeros; you provide the hash and the bytes, and it's trivial to verify.
Like cryptocurrency this will be a moving target. You would need a difficulty setting - but where as cryptocurrency only has to track one thing "total hash rate" this would need to track two things "minimum supported hardware for legit sender" and "hardware threshold for attacker to be successful with a mining rig they can afford, based on the income from the spam".
Each email provider might come up with different difficulty levels based on what they thing this is. So some handshaking might be required. And less computer literate people would be stressed why their email is taking 6 minutes to send. I think it would be hard to implement.
That is a clever idea but I think it'll still fail so long as email (SMTP) is a fire-and-forget architecture. As long as you have that asymmetry, your SNR is going to suck.
If it were a back-and-forth protocol, more like TCP, then you have way more options for congestion control, error reporting, load balancing, and the like. The server can choose to accept the incoming request, ask for more verification, or interrogate the client in various ways. This could be something just like DKIM / DMARC / SPF, or even something more exotic, like making the client do proof-of-work with difficulty tied to how suspicious that client is to the server, and also the delivery scope/scale. Or forcing the client to wait for ACK for valid delivery while slow-walking it.
This gets around some of the issues in cousin comments, with respect to punishing botnets and rewarding lawful players. Established, high-trust players pay no cost. Suspicious players can still get through, albeit with a tax (that should be trivial for low-volume personal MX, but expensive for high-volume spam). Furthermore, it's adaptable.
> If it were a back-and-forth protocol, more like TCP, then you have way more options for congestion control, error reporting, load balancing, and the like. The server can choose to accept the incoming request, ask for more verification, or interrogate the client in various ways.
That's basically what graylisting aims to achieve.
The problem to adding a cost to email is that it affects everyone. The amount of CPU power you need to waste to make most spam not viable is so much that it isn't worth it.
It definitely does not - you can allow different work loads for different senders. Mailing lists you actually want can be dropped to zero for instance.
Most spam comes from new address pairs, not existing ones. Requiring high cost to get past a first-contact filter, then near zero forever after, is completely reasonable and would practically eliminate unsolicited spam.
The address 929@homeaffairs.gov.au, which must be used by Australian permanent residents to update their personal details, refuses to accept email messages unless they are from big tech.
Shame on you, Australian Department of Home Affairs.
And shame on Telstra, which provides the service.
---
Remote-MTA: dns; dibp-ibmail2.msng.telstra.com.au
Diagnostic-Code: smtp; 554-mx.msng.telstra.com.au 554 Your access to this mail
system has been rejected due to the sending MTA's poor reputation. If you
believe that this failure is in error, please contact the intended
recipient via alternate means.
929 is the form. The email address is for completed copies of the form.
Permanent residents are a class of non citizens who have the right to permanently reside in Australia. Most cannot vote.
All of it can be done online through their portal. The Australian government spies on everyone anyway so using the portal presents no increased attack surface for anyone who would care.
Really it's there as a legacy technology. Most people under 35 would certainly use a portal. Probably most of them over 35 as well at this point.
Readit News
Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do.
Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.
For years I avoided to use any external service to decide whether its Spam or not. But about 2 years ago I started to rely on some of the external Blocklists.
Till today I have no problem sending Email. Even as I don't use DKIM or DMARC.
I wouldn’t expect sending email to be the problem. But I’d be surprised if it’s delivered.
To those that still persist, there is a page I found recently that helps you make sure that your outgoing mail is configured correctly: https://appmaildev.com/en/dkim. They generate an e-mail address, you send a mail to them, they do the check and display results (not affiliated, just a happy user).
I will check out your recommendations. Thanks
Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.
We also should keep mail for pseudonymous and anonymous user authentication. There are a lot of threats to the free internet right now and user account consolidation is one of it. I agree that people should keep hosting themselves. Someone blocked you? Their loss, let them rant to their internal IT about it. Corporations or institutions that use Google as a provider should be seen with scepticism.
To host anything beyond those protocols and/or on more powerful hardware is often counter productive.
The problem is getting the ports opened, you need to fight for that right even if it makes spam worse in the short term.
Fight for external IP, ports and static IP in that order.
Also does it help to use Google/Azure to host with regard to IP reputation with Gmail or Outlook?
For DKIM, when the email was sent it was signed with a key by the sending server. It is identified by a UUID in the incoming email. So the receiving server again queries DNS for TXT <UUID>._domainkey.hnemail.example and receives the public key as a response. Signature verification passes? Accept email. It fails? Mark as spam.
This doesn't have a lot to do with IP reputation. This is different. If you are a very large email provider, you might develop custom spam filters. IPs are allocated to 'autonomous systems' i.e. who actually uses them and hands them out to users, and depending on the business you might make some decisions about reputation. For example, if the IP address is part of a consumer ISP block that is handed out to users of broadband, chances are high that if they're sending email, it is probably a Windows PC compromised by malware.
Similarly, you might decide some ASNs are better than others. Some hosters are more liberal in what they will accept, such as VPN endpoints, tor nodes and such and as a consequence of this more spam comes from these ranges.
Rightly or wrongly, larger email providers try to add these extra filters to the process to protect their users from spam. This obviously sucks if you are genuinely trying to run an email server on your symmetric home fibre connection with a dedicated IP, but that's the world we live in.
I can't make any general statement on which providers might be best, and some people will have no issue whereas others will find themselves unable to send anything. I don't work for Outlook/Microsoft or Google and never have, so I don't know exactly what rules they use, and in all likeliness they shift constantly depending on spammer patterns. I can only say I've found running from a small DC to work pretty well.
this can't exactly be policed
Reading the comments here makes me incredibly sad. Every answer that tells me to use a provider misses the point. The Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers. I should be able to run my own E-mail.
The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Either we solve the spam problem, or things will get worse. The big tech corps won't solve it for us.
I had some troubles with IMAP search. I set up CLucene, it was easy and enough for me (no need for Java Lucene). It just took me a long time to figure out why it wouldn't search a domain part of email addresses. It just required to set up the tokenizations in such a way to split words also on @ character, i.e. don't consider a full email address as a word. :P I also had some troubles with OpenLDAP until I finally decided to read the docs and examples there properly. Since then I have been using this setup happily and it appears I will continue to do so! I also share the LDAP with NextCloud btw.
This is a key difference. Many people who have a bad time self-hosting mail set up with a minimum-effort major provider like Hetzner or DO — you're guaranteed to have spammers as neighbours some of the time, and other networks will behave accordingly when deciding whether to accept your mail.
A real server in a proper datacentre is great — I do the same — but as a lower-cost option a virtual server with a small, clueful provider who knows their customers also works just as well.
DMARC, IP reputation and spams are not obvious.
He alluded to this in
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned
> Please believe me. My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.
Wow, good catch! He really buried the lede there -- "I kept getting blacklisted because my servers kept sending out spam." I don't know what to tell you pal -- spam is a serious problem, and if your users are sending it out, you're contributing to that. If you can't keep your users in line, other providers don't have any choice but to blacklist you.
Is this true? Even for consumer stuff, the only Google product besides search that seems widely popular is Gmail.
For enterprise, Microsoft is pulling in billions from office products. Without digging into their statements I know from my own experience that enterprise email services constitute a core piece of this.
For consumers, switching costs are low (mostly just inconvenient to go through all of your accounts) but not relative to the drawbacks of using corporate email providers which is zero unless you value your privacy.
any community that grows large enough needs some mechanism to manage trust, this is a universal issue. The early internet was more permissive and less differentiated simply because it was smaller.
The big corps do an alright job at managing spam given the sheer size of the problem, and more importantly you don't just need to solve spam, you need to do so economically, because for your system to stay distributed the nodes need to do the job competitively.
Given that there's intrinsic benefits to managing these things at scale that's not really realistic, in large systems you're always going to have division of labor and stratification for that reason.
https://en.wikipedia.org/wiki/Feedback_loop_(email)
gmail silently drops emails (while reports smtp 250 accepted) - they could just as easily report that it's blackholed. spammers already do get through their fancy AI filters.
microsoft proactively blocks half of the world, rejects the incoming mail, and sends the sysadmin on a wild goose chase to get the IP/domain/whatever allowed. then you diligently register, and wait. and then no signal from them, and the problem still persists.
so big corps, small shops, everyone and their dog flocks to the good old microsoft/google duopoly.
and at this point if someone asks what to use for email knowing ... well, it's hard to not recommend folks to just use google workspace and have some kind of backup ready for when G bans their whole account just because.
Email is your identity in many cases (for lack of a better solution).
Email -> identity -> tracking -> advertising revenue
1. Require the email receiver to add the sender to their contact list, then put all other emails in "Junk". The UX of this could be quite good nowadays with smartphones and QR codes. The changes to email apps are minimal:
2. Add support to the SMTP protocol to let the receiving server demand that the sender make a small "postage" payment. This will require a privacy-preserving micro-transaction system. I worked on one that doesn't use crypto [0].Here are some ideas for improving the current system, but not actually solving spam:
3. Create a protocol for automatically reporting emails marked as spam back to their admins. Servers will only accept signed emails. Then servers can do lots of things like:
4. Add SMTP responses that mean "rejecting because your server sends spam" and "rejecting because that user sends spam". Servers can mark sent emails as "Receiver rejected this email" and inform the user.5. Build a shared spam reporting system that accepts only signed emails and supports searching by email address. Receivers can use it to identify compromised user accounts and reject their email. Senders use it to identify receivers acting in bad faith (reporting ham as spam). A centralized version would be straightforward. A decentralized version would be a challenging project.
6. Add support to the SMTP protocol for reporting rate limits and cooling-off times to email senders. Admins of large shared email systems can feed these metrics into a monitoring system and receive alerts of problems early.
[0] https://github.com/mleonhard/hipp
EDIT: tone
I think matrix might get rid of a lot of the infrastructure barriers, but I don't have much experience with it.
As you say though, there are existing groups. EFF[0] comes to mind. They advocated for net neutrality back in the day.
[0] https://www.eff.org/
This would require certain rules to avoid SPAM, but the outcome is totally worth for me.
Same philosophy for exposing a your personal blog of html files or content like mp4 videos. The sweet spot is to focus on buying a domain name you control. Then let Amazon S3, or Cloudflare, Hezner etc, host your html or mp4 files.
I quit self-hosting email at home over 15 years ago. It's just not something I want to babysit anymore because I have other things to focus on. As long as I control the MX record on my own domain, that's really all that's necessary.
Yes. My outgoing email goes out via Sonic's SMTP server, with the SPF records to allow it to have a source address of my own domain. Incoming email goes to my own domains and gets forwarded.
This seems to be trouble-free. The domain is on a cheap shared hosting account. I'm not running a server. I own the domain, and not though the hosting company, so I can switch to another provider if necessary. In 27 years, I've had to do that twice, because the hosting provider went out of business.
This is easy to do, and I don't have to deal with Google. I don't even get much spam. All the spammers seem to be targeting the big services now.
I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer. It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads. The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
Third party email providers, the so-called "established businesses", get a free pass as sending SMTP servers that are accepted by almost all receiving STMP servers. Everyone just assumes everything coming from those SMTPs is legit. Establishing this "legitimacy" and getting the "free pass" is difficult and some have suggested, the third parties may employ anticompetive tactics.
However, IMHO the receiving SMTPs is a different issue. Why do we let these third parties receive and store our email. (Why do our homes have their own mailboxes. Why not use a "P.O. Boxes" instead.) Eventually we could move away from letting third parties control the receipt of our mail. Neither "POP3" nor "webmail" was part of the original concept of email.
Today, it is easier than ever to set up overlay networks where we can assign our own IP addresses and run our own SMTP servers that can communicate directly with other SMTP servers on the overlay network. These networks are not open to the world, they may only be open to people we know. Much of our mail is between people who know each other, e.g., friends, family, colleagues. Or businesses that we contact first. We can separate different social and business networks on different overlay networks.
Anyway, the sending and receiving of mail can be separated. We do not need to let a third party control both.
[1] - https://support.mozilla.org/en-US/kb/openpgp-thunderbird-how...
You can also even keep Gmail as your MX server! Just move messages off of it as soon as they arrive. It's just a mailbox, after all
It's definitely an anticompetitive practice.
Please make your substantive points without swipes. Your comment would be fine without that bit.
https://news.ycombinator.com/newsguidelines.html
yeah, but in this climate what are you gonna do? It's not like there's any kinda recourse for monopolistic behavior that has any teeth to it.
iphmx 1 google 2 kornet 1 linkedin 1 secureserver 1 amazonses 1 self hosted university email 1
Or maybe it's an overly competitive protocol? Like playing Monopoly. Even a neophyte can end sweeping the game despite not understanding any of the underlying mechanics that drive the game's outcome. Those remaining mail providers are also fighting back the insanity. They 'just' won.
Don't hate the player, hate the game.
Example:
example.com. MX aspmx.l.google.com.
Cert should have aspmx.l.google.com as subject or SAN.
Ideally there would be some a decentralized permanent domain registry keyed on certs (I know these exist but have not been adopted), or at least a fallback domain you could configure somehow in case you lost control of your main domain.
Dont know about you, but I have setup my mailserver years ago, and outside of regular OS updates, havent had to touch it.
That's not necessarily a sure cure, depending on the hosting provider. RoadRunner (Spectrum / Charter) in the US and Shaw in Canada won't deliver emails from my domain hosted at Runbox.com (or sent directly from the runbox.com domain.) Spectrum's bounce message references an error code that translates to "Spectrum limits the number of concurrent connections from a sender, as well as the total number of connections allowed. Limits vary based on the reputation of the IP address. Reduce your number of connections and try again later."
You can even have a hybrid solution where incoming mail goes directly to your self-hosted server and (some) outgoing mail is relayed through a third party.
You would think. The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life(at least). Otherwise, as soon as you lose your domain you lose ownership of your email. Any one that has control of the domain has control of your e-mail.
This dawned on me after a place I worked at reactivated the email I used for work when I worked there. It has my name but I have 0 control over it. Lucky for me I never used the email for things other than work so it's not a big deal. It is still bothersome that they can do that and I have no say so on its use.
But that's true for any delivery endpoint in any medium, including physical mail and phone numbers.
It's pretty easy to set up auto-pay for domain names so that all you really need to do is keep the billing info up to date. After that it all runs on autopilot.
Is there a registrar that will allow you to do this? Most of the ones I’ve looked at have an upper limit of around 15 years or so
Deleted Comment
I like the idea of having a node I can just plug into my network. I run my Urbit on a Mac mini with tailscale (which works great).
The core of what he writes about is correct though, email failed to be truly peer to peer (as imo all non-urbit-like federated systems will) because of the incentives that lead to centralization (spam, difficulty of running nodes, etc.)
We’re suffering the consequences of the local max we’re trapped in currently because of this. The promise of the 90s internet was a bunch of people using decentralized services they controlled - instead we're primarily thin clients connecting to a small handful of powerful ad companies. We're mostly serfs [0] allowed access if we give up our data for ad targeting, follow EULAs nobody reads, and don't say anything the company earls disagrees with.
[0]: https://zalberico.com/essay/2020/07/14/the-serfs-of-facebook...
Email reception has not been a problem for me so I enjoy having my mx pointing to my own mail server. It gives me more control and it requires very little maintenance.
If I was going to outsource anything the first choice would be outbound. My email system does everything right for reputation protection. All senders are authenticated on secure connections and the senders are people I trust. Nothing bad gets sent and my static IP is on a reputable server host and is not on any public black lists. I maintain SPF, DKIM etc. If someone decides to block my IP for no reason by accident or on purpose there is very little I can do or care to do anymore.
I have an alternate path for emails setup via a server I host elsewhere ready to go. If I run into a widespread delivery issue due to massive indiscriminate ip blacklisting of my provider I can enable it. That has happened once in 20 years. If delivery gets too hard I will change that policy to send all outgoing emails through a commercial smtp delivery service and let them deal with the problems.
The MTA parses the message, figures out who it needs to go to (To, CC, BCC headers), figures out what servers receives mail for those recipients, and then transfers (SMTPs) it to the server.
What OP is referring to is that the MTA essentially does a DNS lookup for the recipients domain for a record of type MX (Mail eXchanger).
If you own the domain you have complete control over where that mail goes: you own the MX record.
When you send an email you mail client contacts you mail-exchanger (MX) and drops the mail in your outbox. Then the MX will look up the MX record for the domains in the TO field and attempt to send the email to it using SMTP.
The first thing receiving server ussally does and look up the IP of the sending server and see if it's on a spam black list. If it is it will probably just drop the connection. Then it will look up the SPF record for the domain in the FROM address and see if the sending server is allowed to send that mail.
Larger email services will have an internal 'reputation' scoring system that will use data from reported spam to figure out what IP's and domains are sending spam emails and filter them out. They'll also look at if it's a residential IP, in an IP pool for a major cloud provider etc. Each provider has a different system and it can be really difficult to get a provider to trust your IP or whitelist your IP so you can make sure that your mail actually gets to who you're trying to send it to.
Relays are pretty simple they'll take email from one place and send it to the recipient. The mail exchanger usually has a built-in relay. A lot of people will use a third party relay service so they don't have to worry about managing the reputation of the IP of their sending mail server. They'll just add an SPF record for the relays service to their domain. And then configure their mail exchanger to send all outbound mail to the relay and then the relay will do the MX lookup.
A lot of mail services will also have their MX records pointed at relays. These inbound relays will often have a lot of those anti-spam services bolted onto them and they can also be used for load balancing to make sure that the service is always able to accept mail even if it doesn't make it in the mailbox immediately.
The "sweet spot" is combining the two, where you own skywal.com, and have your email send/receive through Google or whoever. Then, if Google decides to ban you, you just register skywal.com with another company who provides that same service, and you keep your same email address.
That's the broad strokes anyway.
Let's imagine I'm sending from user@zahllos.example to user@skywall.example. I'm doing it from say Thunderbird or Outlook, and you're using the same.
I need to send, and to do this I typically use an SMTP server. This is something configured, probably on zahllos.example, maybe with the domain smtp.zahllos.example. My mail client contacts this and 'logs in' with my details, then transmits the email message I want to send to this server. The server says 'right fine' and closes the connection.
At this point, the message is in a mail queue ready to go. The SMTP server then does a DNS request for the MX record of skywall.example. Let's say this is 'mail.skywall.example'. My server, smtp.zahllos.example, then connects to `mail.skywall.example', also speaking SMTP, and says "hey, I have this message to deliver".
It is at this point that mail.skywall.example can decide to do some things. It might check SPF, so it will query the SPF record of zahllos.example to find the list of servers that may send email for that domain. In this example, let's assume smtp.zahllos.example is in the list. Great.
It may then also check the mail headers for a signature, called a dkim signature. My server signed the message before sending; mail.skywall.example can query <uid>._domainkey.zahllos.example and find a public key (or not) and check that this signature matches (or not). Again, let's assume it matches.
It might also check something like TXT _dmarc.zahllos.example to see what my DMARC policies are. If I have something like v=DMARC1;p=reject;sp=reject;pct=100;ri=86400;fo=1;aspf=s;adkim=s;rua=mailto:postmaster@zahllos.example;ruf=mailto:postmaster@zahllos.example;" this tells you I'd like you to outright reject anything not matching policy, that I expect everything to match SPF and have DNS signatures, and you can send reports if you support that to postmaster@zahllos.example. Your server can then enforce these checks as it likes.
One of the first things that will happen is that my server will announce itself via an EHLO statement. An obvious check to do is to check that the sending IP actually matches smtp.zahllos.example. by querying 'reverse ptr' records.
Your receiving server will also likely hand the message over to various spam-checking tools for analysis, such as against DNS blocklists and so on. Larger providers likely have much more sophisticated infrastructure here. Ultimately, you're going to do one of a few things: 1) deliver to inbox or apply user-specified rules and deliver to a folder; 2) deliver to junk (which is typically just another folder, but treated specially by clients), 3) reject, and tell smtp.zahllos.example you don't want the email.
Once the email passes through the smtp dameon, assuming either 1 or 2, it then gets stored somehow and in some way. I'm being a little bit vague here, because 'it depends', but in the simplest scenario, the smtp daemon will write the message to an mbox or maildir-style format. More complex setups definitely exist, indeed, there can be multiple layers of servers doing analysis on separate machines, but for simplicity, mail.skywall.example is one VM that makes its decisions and the result ends up in /var/mail/user@domain/ or some such.
A key aspect of this step that makes email very nice is that if smtp.zahllos.example cannot, for some reason, reach your server now it will queue the message and try again at set intervals. You can reasonably safely turn off mail.skywall.example for a couple of hours.
Another aspect is that you can have multiple MX records where you are prepared to accept email, with priorities. So if you can't accept at one address because the server is down for maintenance, another will accept.
So, now you've technically got an email, but you don't know it. So you open thunderoutlook, and you connect to an IMAP server imap.skywall.example - in our example let us assume this is really the same thing as mail.skywall.example. The server checks you are really you with your credentials. At the backend, this is just another daemon that knows to read /var/mail/... and find new messages; it finds one, downloads its headers and displays it on your screen.
Since we're in a slightly more modern world now, in the case your client was already open you might have an "idle" connection with the server at all times, in which case it can push the message down to you.
In the case of webmail, it is really all the same thing, except you point your browser at a webpage, and that webpage communicates with the servers instead of your client communicating directly. Open source webmail might even use IMAP underneath; things like Zimbra use their own java mail agent, while Google is entirely custom.
That might seem complicated, but in the end it isn't: between two domains at the 'edge', in the end, there's an SMTP conversation. One sending server tells a receiving server it has mail for delivery, and it finds that server by asking DNS where it is. The receiving server may do a bunch of checks against DNS also before making a decision on what to do with the email.
So, mandatory third-party doctrine/warrantless surveillance.
no forth amendment protection for your email because its stored by a third party.
Spam is a real issue.
The amount of spam emails which get sent are absurd and likely orders of magnitude more than non-spam. And spammers do a lot to mimic real emails, including just hacking legitimate addresses and adding them to botnets.
Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
And even if it really isn’t that bad and never will be, the big companies and those arguing against self-hosting will claim it is. They don’t want to allow a relative few self-hosted email servers in exchange for much more difficult and less effective spam detection. Forget Gmail and Outlook, why not just use Fastmail or Protonmail?
If you want a legitimate argument for self-hosted emails you need to address the spam. It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
But this article doesn’t mention that org, in fact doesn’t say much at all about spam besides “keep existing spam-prevention because it already works”. But you should at least explain why. Because spam is a legitimate argument for big-co forming an oligarchy that’s not just “so they can make more money”, and it’s the main argument that big-co uses.
responsibility for spam (and other kinds of abuse) can be delegated via simple reputation scoring for netblocks, sender authentication and a proper feedback mechanism along the chain.
when the user clicks "spam" gmail already uses that to train their fancy AI and if you are not a small nobody, then they already alert you that ooops you sent something spammy via a feedback loop [1]. (see also how Mailchimp proudly claims they work with big integration partners like gmail ... https://mailchimp.com/help/how-mailchimp-prevents-and-handle... )
whitelist clearinghouses exist [2] but they are not terribly useful, because there's most of the signal to use for reputation is hidden :/
[1] https://en.wikipedia.org/wiki/Feedback_loop_(email) [2] https://www.dnswl.org/
Are there any real-world examples of this? I know other decentralized networks like Tor and BitTorrent have some sort of reputation and feedback system. What type of "spam" do they deal with and how well do they deal with it? Are there any systems more similar to mail with mechanisms to prevent spam?
Spam is a serious issue, not just in email, not just in decentralized systems - it's one of the main issues in technology today. If there's a solution, even a proof-of-concept in a smaller system would be great
> And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
> Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
It is as good as it is exactly because of the requirements author finds tedious.
> It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
They do exist but there are a bunch of problems with that. People have varying definitions of spam, getting paid to whitelist someone creates a perverse incentive or not getting paid will quickly overwhelm the organisation.
Things could be better if we could enforce sender authentication (SPF/DKIM). It would assign a direct cost to getting your domain blacklisted. But if nobody is taking away rest of the spammer's domains (or keeps selling them new ones), they'll continue.
What impacts me the most is a never-ending problem with Gmail classifying messages as spam that were actually important to me. Time sensitive announcements of meetings or events, for example. Many are coming from senders I've been receiving email from for years.
Gmail is convinced that almost every technical-related email mailing list I'm on is a spam source, despite my constantly going into my spam folder and telling it dozens of messages from those mailing lists are not actually spam.
Meanwhile, what I do get is a barrage of promotional emails - what I consider very much to be spam - from corporations that at some point have had my email address and now email me multiple times a week. Those sail right past the spam filter into my "promotions" folder and accumulate...
https://github.com/foxcpp/maddy
https://blitiri.com.ar/p/chasquid/
These options are much easier to set up, will do things like generate DKIM for you, etc.
I talk about this a lot[0]. There are positively awesome tools for email out there.
[EDIT] - Since I'm repeating myself I've collected all the options into a post[1] I can just link to.
[0]: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
[1]: https://vadosware.io/post/its-never-been-easier-or-harder-to...
Something like Maddy is much easier to set up, comes with workable SMTP + IMAP, saves to object storage.
If you pair maddy with Thunderbird/Outlook/Mail/whatever else, I think we can actually get new people self-hosting without getting discouraged.
[EDIT] I misunderstood the earlier comment as a recommendation -- it's just a statement of fact.
o/t: Could you add a close button to that ethical ads element? It's obscuring a decent chunk of the screen on mobile and I can't figure out how to dismiss it
IE something like: https://blog.tinned-software.net/setup-sieve-mail-filter-wit...
Ed: Does anyone have experience with https://modoboa.org/ ?
I put maddy first because it's what I personally use (without dovecot) -- I make sure to back it up as well, and that's enough for me/most people, I think. It's the closest to the whole package which doesn't rely on.
Oh you know what, I completely forgot about iRedMail:
https://www.iredmail.org/
I haven't run this, but it's another great option when combined with SoGo.
https://mailcow.email/
It seems like one solution is to bcrypt hash (or some similarly expensive algorithm) the email and include the hash in a header. Of course you need to hash per receiver or a spammer can just hash it once and spam away.
The receiving client hashes the email and compares the result with the value in the header and discards emails that don't match.
You'll never get industry buy in though - the FAANG companies don't want to pay that cost for their semi-legitimate email. They prefer to keep that cost externalized.
I believe there have been attempts at something like this, but it clearly never went anywhere.
https://en.wikipedia.org/wiki/Hashcash
Each email provider might come up with different difficulty levels based on what they thing this is. So some handshaking might be required. And less computer literate people would be stressed why their email is taking 6 minutes to send. I think it would be hard to implement.
If it were a back-and-forth protocol, more like TCP, then you have way more options for congestion control, error reporting, load balancing, and the like. The server can choose to accept the incoming request, ask for more verification, or interrogate the client in various ways. This could be something just like DKIM / DMARC / SPF, or even something more exotic, like making the client do proof-of-work with difficulty tied to how suspicious that client is to the server, and also the delivery scope/scale. Or forcing the client to wait for ACK for valid delivery while slow-walking it.
This gets around some of the issues in cousin comments, with respect to punishing botnets and rewarding lawful players. Established, high-trust players pay no cost. Suspicious players can still get through, albeit with a tax (that should be trivial for low-volume personal MX, but expensive for high-volume spam). Furthermore, it's adaptable.
That's basically what graylisting aims to achieve.
Most spam comes from new address pairs, not existing ones. Requiring high cost to get past a first-contact filter, then near zero forever after, is completely reasonable and would practically eliminate unsolicited spam.
Shame on you, Australian Department of Home Affairs.
And shame on Telstra, which provides the service.
---
Remote-MTA: dns; dibp-ibmail2.msng.telstra.com.au
Diagnostic-Code: smtp; 554-mx.msng.telstra.com.au 554 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means.
You update personal details with the government via email? That's interesting. I would have assumed they would use at least a secure web form.
Permanent residents are a class of non citizens who have the right to permanently reside in Australia. Most cannot vote.
All of it can be done online through their portal. The Australian government spies on everyone anyway so using the portal presents no increased attack surface for anyone who would care.
Really it's there as a legacy technology. Most people under 35 would certainly use a portal. Probably most of them over 35 as well at this point.
Deleted Comment
Deleted Comment