For me, as someone with their own mail server, these technologies mostly serve to inform me that Russian IP addresses are still trying to send email in the name of my domain for some stupid reason.
It makes sense that people whose business is sending email know how to set up email correctly. I'm mostly surprised at how many legitimate sysadmins struggle with getting the basics correct. Surely those dozens of DMARC emails you get that your sendgrid email has been refused because of a bad SPF signature should set in motion some kind of plan to ask if maybe marketing is using them legitimately?
Automated signatures are of limited value but I rarely see rejections based on SPF and DKIM that are a mistake. Things are probably worse for big organizations but as a small email server, technical rejections are usually the right call. The only exception is mailing lists, but the dozens of people who still use those can usually figure out how to add an exception for them.
The problems I noticed were, it doesn't matter what the SPF and DKIM look like. If Google or Microsoft refuse to relay your email based on secret internal factors then you're out of business.
The issue here generally boils down to the defining difference between a generalist Admin and a Messaging Admin. The generalist can follow instructions, and nearly all the instructions out there stop at the point where SPF/DKIM/DMARC are successfully implemented. A generalist worth their salt will then fill in the gaps if they can', and knows this isn't where you stop when you want mail deliverability. There's a higher bar.
If you follow instructions written by non-professionals blindly you don't ever reach the point where you get to quality work.
Google, Microsoft, and the other large ESPs don't refuse to relay your email based on secret internal factors. This is what the non-professional people say to falsely justify why they can't do something.
Google and Microsoft publish the internal factors they use in the form of whitepapers at the industry working group. Its not ready made, and there are a lot of them, and they may not release their specific implementation details, but the metrics are there and often are based in weighted form (reputation-based systems).
If you follow them correctly, and set up the appropriate reporting accounts, and maintain those accounts, you won't have these problems. You generally only have problems once you've violated guidelines continuously, which happens when you rely on, or are unable to discern between qualified and unqualified help.
But there are big problems with mapping from IPs to countries. My IPv6 is detected as Russian, though it is London-located tunnel exit point and I'm in the Netherlands.
Ive got a server hosting a number of things, amd monitoring setup for a lot of stats. Got tired of seeing blips because various countries were beating on my server, not a DoS, but enough requests to notice, and sometimes generate an alert. I blocked 7 countries, in full, and the impact was fantastic. No more 2gb of logs generated every day by countries that have no business accessing my server.
Unless you own a global business, i see no reason to even allow other countries access. The potential for attacks is too great, especially from some very specific countries.
Ok but you can’t block someone else using their own IPs to send email.
If you set DMARC to report, you’ll get notices from remote email systems when they receive noncompliant emails with your domain in the Envelope From field. Those reports are where you’ll see Russian IP addresses show up when they are trying to spoof your emails.
But there is no way to block them because neither the senders nor receivers are on your infrastructure. The best you can do is set a reject DMARC policy and hope everyone follows it.
As a non-email guy, I can tell you that if a system that boils down to having an (optionally certified?) key requires much more than just putting it into a folder with a domain name and running a service, it’s badly designed and has unnecessary complexity. Which will result into abusers having more expertise than legitimate users. The fact that you can “get” DMARC SPF DKIM wrong, while it’s basically a hard requirement for operation, is just screaming something important to the email software.
As a generalist admin, would you say the same about DBA operations or would you say that's just not my specialty?
The reasoning you provide doesn't differentiate, and speaks more of frustration which naturally comes with any area you aren't steeped in, or knowledgeable about.
In most organizations there is no point in a sysadmin to spend the effort in understanding how to set it up correctly as Marketing has got more authority on email. Marketing will simply demand changes to the config that they do not understand and there is nothing you can do to stop it as they will have the CEO on their side.
> Marketing will simply demand changes to the config that they do not understand and there is nothing you can do to stop it as they will have the CEO on their side.
Marketing should get their own (sub)domain for sending their missives, that way the primary corporate domain's reputation is not harmed.
Unless you want to run the risk of outgoing e-mails from Finance / Accounts Receivable to be sent to other companies' Junk folder.
Orgs like that will hire consultants like me when they can't figure out why their stuff isn't landing in the inbox. Then 3 months later their webdev will somehow delete the entire zone when adding their A record.
You mean like the time I had a salesperson demanding that we turn off Cloudflare across our entire domain because he'd read some random article somewhere saying we should?
Which is another reason to strictly enforce SPF and DKIM, in my book. Let marketing break those policies, that way I don't need to bother with reading your company's spam!
even worse when you have even less control than that, if you run some type of hosting and are trying to convince non-technical clients (or even worse, non technical clients who think they are technical) to “please just add this record exactly as it says here to your domain” and they’re somehow unable to for months and months
to be fair here: for a lot of companies, if the mass mailing stops, the money-flow stops then that's no good for anyone... so the CEO will probably err on the side of money, presumably.
As someone who set these up, I can tell you, the answer is rather simple:
- spammers have 1 system to set up in order to spam. They get it right.
- company admins have dozens of projects, of which this is a tiny one, with zero ROI to the bottom line (if people don't consider how critical security is). So they delay.
- companies often have dozens of systems integrated, when I set up DMARC/DKIM the first time for my company, a bunch of email tools broke, we had to do a bunch of leg work, took us a month end-to-end. The value was recognized when we almost lost 20k to a "ceo emails you" scam. But until then it wasn't a priority.
- we didn't even have a full IT, i just stepped in because I cared enough.
- my current company has a dedicated security team. These holes are plugged VERY quickly.
I used to have my policy set to reject, but then I found out some part of an Enterprise Outlook mail filtering chain was rewriting the mail I sent before checking the DKIM signature. I can't fix stupid, especially for other parties, so I changed the policy to quarantine instead.
I doubt Russian spammers will care about the difference to be honest. If they accept that their email will be delivered to spam folders, why would they care that the email gets silently dropped? In neither case anyone is going to fall for them.
I am just having this problem. Actually getting SPF, DKIM and DMARC right and having a domain with a 0 spam score will still land you in the spam directory. It turns out, you need to have a "reputation"? before your email gets accepted into gmail. My head was spinning as to how that reputation will be built if your email just goes straight to spam.
But sure, Linkedin emails are definitively not spam and their dark-patterns at adding you at n+1 emailing list doesn't get them banned from the big (or any?) provider.
It's easy, you just have to have a regular, decently sized volume of non-spam emails, and suddenly your email stops being marked as spam!
The logic isn't even that bad. SPF and DKIM serve to prove to the email who the sender is. That doesn't mean much if the sender is a spammer. Verifying identity claims is only the first part in checking email for spam, the harder part is checking if that identity is someone you trust.
When you email Outlook or Google, you're better sending more than a few every single day, and the recipient better manually drag those emails from their spam folders to their inbox, or they're all being learned as spam.
> you just have to have a regular, decently sized volume of non-spam emails
But if you have a regular decent size of emails coming from your domain, that is more likely to be spam than if you have a small number of intermittent emails coming from a domain.
I worked on this for a while, at a time and in a market where most of our recipients had @hotmail addresses. I discovered that mass email sending was akin to a "pay-to-win" game.
We had/opted to acquire the services of a company "expert in email deliverability" (Return Path), who somehow provided detailed metrics of how our IPs were scored by MSFT. I always wondered why MSFT didn't provide those scores by themselves, and how a 3rd. party could have access to them.
Re. your comment... slow ramp-up is the only way, with constant monitoring of deliverability and consequent adjusting of recipients (i.e. removing those who do not open or hard-bounce). I did also wonder if paying that company perhaps gave us a headstart when adding new IPs...
It's almost like all those bad actors (linkedin) are owned and controlled by the big players (microsoft) that benefit from email being only commodity they can provide.
I think the domain rep is worth less than IP rep. I had occasional issues sending issues when I self hosted on a VPS. When I moved my domain to Fastmail I haven’t ever had my emails go to spam.
As a tip, go to a VPS that's had a history of being very selective of allowing SMTP traffic but still allows after some kind of review. Cheap providers that never did any blocking probably have bad reputations for their entire address range.
I've been successfully using VPSes to send emails for 20 years.
I am sending from SES. Interestingly, I didn't have a problem getting the email delivered to inbox in fastmail despite having an aggressive "protection level".
This isn't a problem for personal emails, as after a request or two friends will unspam you. Google blackholes emails, breaking all mail logic (no bounce), so I assure you the SPAM folder is a good gmail sign.
I would imagine that on the corporate side, your employees could do the same. Beyond that, if you're sending spammy stuff, have unsubscribe headers and links in emails.
If it's a new domain, then your problem isn't reputation exactly, it's having a newly-registered domain. Buying a new domain, setting up the SPF, DKIM, and MARC, and then immediately spamming from it until it's banned everywhere a week later is standard spammer MO.
I've been self-hosting mail for me and my family for about 20 years and don't send nearly enough mail to have a "reputation" with anybody. Still, I don't have any problems with deliverability of mail.
> Actually getting SPF, DKIM and DMARC right and having a domain with a 0 spam score will still land you in the spam directory.
This little bit of wisdom gets passed around all the time, but it's actually not true. You can send email from a brand new domain to Google and Microsoft and whoever just fine. What you can't do is send email from a brand new domain, and a brand new email server--or an email server on a VPS, or an email server on a residential IP. Residential IP blocks are almost completely blocked, because of unsecured devices being used to send spam, and VPS blocks have the same problem. You can get around this by using a mail relay, or building your domains reputation on a server that already has a good reputation.
SPF/DKIM is really about mail server reputation. So it mostly benefits larger servers like the ones run by Google, Microsoft and Yahoo. Unfortunately, that means that attempts by those larger providers to combat spam using such reputation will naturally hurt smaller providers. So the actual effects of SPF/DKIM are on the whole negative.
The root problem is that we don't actually need to keep track of email server reputation. No one says to themselves "Huh, this is from a Gmail address, it must be legit". We really want to keep track of sender reputation. We need to be able to treat anonymous email differently than email from people we actually know. That implies that we have some work to do on the problem of identity. As it is, there is not even a way for a known email sender to securely introduce an unknown email sender. You know, the way that regular human people normally are able to transfer identities from one to the other.
>SPF/DKIM is really about mail server reputation. So it mostly benefits larger servers like the ones run by Google, Microsoft and Yahoo. Unfortunately, that means that attempts by those larger providers to combat span using such reputation will naturally hurt smaller providers. So the actual effects of SPF/DKIM are on the whole negative.
That paragraph is incorrect. SPF/DKIM is not about reputation. The main purpose is preventing domain impersonation from unauthorized senders. E.g. mail servers will reject fake emails from "upofadown@microsoft.com" because you don't control any email servers that's whitelisted in microsoft.com DNS TXT records.
E.g. I was able to register a brand new .com address and then successfully send to gmail and MS Outlook accounts within minutes because I had proper SPF/DKIM in the DNS records for that new domain. That new domain had zero reputation and yet Gmail accepted it because SPF/DKIM was configured correctly -- and -- the underlying ip address of the server it came from had a good reputation.
If SPF/DKIM was truly about "reputation", it would mean I'd have to wait days or months for reputation history to build up before Gmail accepted it.
And it will mysteriously _stop_ being able to send mail to Google despite you doing everything right, because of whatever nonsense they use to determine reputation.
I don't think this is correct? SPF and DKIM are about ensuring that the server actually is who it says it is, not about its reputation. In other words, when you receive an email that claims to be from Gmail, SPF and DKIM help you ensure that's where the letter actually came from, not from a server just pretending to be one of Gmail's servers.
> That implies that we have some work to do on the problem of identity. As it is, there is not even a way for a known email sender to securely introduce an unknown email sender.
There is: gpg/pgp signature, but many people find it complicated, primarily because they are reluctant to read the documentation. And it’s popular to criticize it, especially here on HN, in favor of various half-baked alternatives.
I think everyone can agree that any technology that "isn't complicated if you read the documentation" is by definition complicated. I don't need to read the documentation for Gmail to use Gmail successfully.
Could I, as a trained programmer, use PGP and GPG? I'm sure I could if I spent some time reading about it. Could my 90 year old grandmother, who is otherwise quite comfortable with email and whatsapp? No, not to any meaningful extent.
That's what kills a lot of these "perfect" implementations.
HN members tend to be nerds, and we don't really have an issue with setting stuff up (many HN IDs, for instance, have Keybase auths).
Most non-HN types have no patience for that stuff. Security needs to be made accessible and easy-to-use, before the vast majority of folks will implement it. That's the single biggest conundrum, IMNSHO.
We really need a "Trust on First Use"(TOFU) system for messaging, that can be verifified, or pre-trusted offline, face to face. It'd be awfully nice for your bank to give you some thing that you can later verify that any communication from them (web site, online banking, text message, email, etc) are legit and verified.
Or if we can't trust users to handle TOFU, then some token/unique address/whatever that we can exchange face to face to enable trusted communication.
> We need to be able to treat anonymous email differently than email from people we actually know.
The simplest solution to that would be an "only show me emails from people in my address book" filter. That would mostly echo how we treat user trust on all other platforms. Genuinely surprised this doesn't exist in most email clients (or does it and I have just overlooked it so far?)
Of course that's only a partial solution and wouldn't work for accounts where you expect unsolicited mails from people you don't know. I'd see it more as a "low-hanging fruit" solution. You could also expand the heuristic, e.g. also consider previous conversations, mailing lists, etc.
(Interestingly, the "introduce a friend" functionality would come for free: You can already send contact details as a VCard in an attachment. When receiving such a mail, some email clients will show a button to quickly add the contact to the address book.)
> only a partial solution and wouldn't work for accounts where you expect unsolicited mails from people you don't know.
I actually think this would work fine. Imagine a quarantine inbox for new emailers that the user must scan and approve/block. This is exactly what hey has implemented.
> The root problem is that we don't actually need to keep track of email server reputation.
We actually do. Is the server allowing anyone to sign up so that it's sending 99% spam, or does it have a lot of anti-spam measures so sign-ups can't be automated and it blocks accounts as soon as it detects them sending spam?
> As it is, there is not even a way for a known email sender to securely introduce an unknown email sender. You know, the way that regular human people normally are able to transfer identities from one to the other.
That's exactly what PGP's web of trust model is for. Someone you know, and trust, can sign and send you a public key of someone that they trust.
This new key will be automatically trusted in your trust store because it's signed by someone you already trust, although in a lesser trust level to account for the degree of separation. If you later verify that key out of band you can upgrade it to a higher trust level.
SPF/DKIM, as well as TLS etc., is just stupid shit we do because we're too lazy and/or incompetent to make web of trust work for us. It's not a technology problem, it's a human problem.
Well, yeah, we should use preexisting standards and OpenPGP would be perfectly fine here and is probably the best choice. That is a wheel we do not need to reinvent. But the actual system used to do the signatures and keep track of the reputation is the last thing we should be thinking about at this point. We should instead concentrate on how to create a system that the majority of people can use and understand. We should be concentrating on standardizing concepts...
It's weird so see such a factually incorrect comment so high up on HN.
SPF/DKIM is literally how you establish sender identity instead of relying on the IP address of the email server so it is ironic to claim that they have anything to do with server reputation while lamenting a lack of sender reputation mechanisms.
Currently, SPF/DKIM are mostly used to prevent fraud, but they also provide the best tool we have to build sender based reputation systems.
> Unfortunately, that means that attempts by those larger providers to combat spam using such reputation will naturally hurt smaller providers.
Indeed. I am on a few mailing lists and many people on them host their own email. I use Gmail so that means visiting my spam inbox once a day to click "not spam" upward of a few dozen times. Day in and day out, same people end up in Googles spam folder. It's bullying and sabotage.
> No one says to themselves "Huh, this is from a Gmail address, it must be legit".
Indeed. I get a shit load of spam from google addresses so reputation is not there.
> We really want to keep track of sender reputation.
This is the hard part but I do think it's something to think about. I should be able to get an email from a known sender every time without $emailprovider making that decision for me. Some sort of attached signature or key that is proof of sender identity so I can route that right into my inbox.
To be fair, SPF saves mail.ru and outlook.com users from five, maybe six spam emails per month coming from my domain, based on DMARC reports. If those numbers scale to include every domain on the internet, that's a huge amount of spam being filtered out very easily and very early.
You'd think spammers would've learned to avoid SPF domains at the very least but they haven't, so despite SPF/DMARC/DKIM failing to get anyone out the spam folder, the technology is still catching spam bots.
While it may or may not reduce spam, it has definitely (based on my personal experience) reduced the amount of spoofed phishing emails and backscatter spam emails to nearly nothing.
In the early-to-mid '10s, before SPF/DKIM/DMARC became the law of the email land, one had to be much, much more careful with phishing emails, checking the wording, the logos, etc, because 9 out of 10 of them appeared to come from the actual domain the email purported to be from. In the past several years (I honestly don't know exactly when the change happened; I don't get a huge amount of phishing emails), it's shifted so that the first thing to check is the sender address. Usually that turns out to be some nonsense string @gmail.com or some long garbled domain.
All of these technologies are basically DOA because of how fickle they are and for lack of support across the board. Most policies are set to not to deny.
DMARC is nice though. It won't stop spam. It won't stop spoofing. But you will know that someone somewhere is spamming people using your domain name. How awesome. :)
I never found the DMARC reports actionable, so I quickly turned them off. What do you do with the information?
Of course, even with hard fail spf and dmarc, I still see some bounces from spam where some server accepted the mail to deliver it elsewhere and the next server denies it, so the first server sends me a bounce.
Trouble is, this is simply not true. My SPF and DKIM are fine. This makes me wonder whether the email ingestion system is simply broken for everyone.
—⁂—
2. I got involved in setting up a Google Workspace for someone a few months back, and the entire tool that their own documentation instructs you to use to check things, https://toolbox.googleapps.com/apps/checkmx/, has been laughably broken for years, sometimes not working at all, but mostly producing misleading nonsense results (e.g. claiming domains have no mail server set up when they do).
I observe the same thing. However, that does mean that SPF and DKIM are useless (although DMARC probably is).
It is correct that SPF/DKIM does not really avoid spam, because spammers are not stupid and can read these standards like anyone else. However, before SPF/DKIM, I remember that I got a ton of phishing mails with FROM containing "support@paypal.com" or similar. Then came Bayes spam filtering, and that would move legitimate mail from Paypal to spam, because obviously, the phishing mails are quite similar.
This problem has pretty much vanished, because Paypal clearly denotes which IP addresses are allowed to send mails from that domain via SPF and the client can verify the mail via DKIM. For instance, Spamassassin makes sure that mails with correct DKIM and from paypal.com get a massively reduced spam score so that your Bayes filter will not move it to spam. This is hardcoded for a lot of domains (see *welcomelist_dkim.cf).
I run my own email server. Most spam crap cannot pass spf/dkim. Although this post has caused me to sit up and notice that the trendline is moving in the unfortunate direction, where I'd say 3 years ago the ones that pass were about 1/4, today it feels like 40-60% pass. The amount of mail I get that I expect, passes spf/dkim at around 90-95%
I suspect the delta between their any my results are the very restrictive sender rules I have prior to accept. In addition any_address@domain goes to my default mailbox, so I'm also probably selecting for laziness a bit more than most.
I also publish an email address without obfuscation on my site, which is getting very little spam, (near zero) which makes me wonder if most spam has given up on scraping the Internet for emails these days.
It’s far easier to buy the email addresses of known good people by buying dumps of websites that got breached.
Web scraping gets you a lot of fake emails, company sinkholes, and other low reward stuff. Paying $20 for 100k confirmed real emails with names? That’s gold.
I think the problem isn't necessarily that adding DNS entries is hard (especially compared to the rest of the process of hosting your own email), but that getting a clear overview of what email tools an organization uses is difficult.
You need IT to list all of the reporting tools, customer service to tell you about their support system, marketing to tell you about their mailing list tool of the week, the sales guys to warn you they're using this new AI email enhancer, and somehow get that shady email forwarding service the CEO uses to give up their mail server IP addresses. Then you need to figure out how to get coverage for all of those tools and keep on top of them whenever something changes.
A lot of companies promise to do great things for you if you just enter the email address you'd like to send email from, and a lot of people gloss over the important details because those sound hard and when they tested the tool on their personal email it worked fine so that's probably unnecessary anyway! Managing email for a corporate domain can be like herding cats.
I think pretty much all email providers (and other systems that want to send on your behalf) have this. More or less the same process where they tell you what to add and then a "check my stuff" button to verify. Which is great.
Sounds good. As I said it was my first time, and I'd just glossed over the specs and did not look forward to it (I usually don't enjoy sysadmin work). So, was just pleasantly surprised.
Readit News
It makes sense that people whose business is sending email know how to set up email correctly. I'm mostly surprised at how many legitimate sysadmins struggle with getting the basics correct. Surely those dozens of DMARC emails you get that your sendgrid email has been refused because of a bad SPF signature should set in motion some kind of plan to ask if maybe marketing is using them legitimately?
Automated signatures are of limited value but I rarely see rejections based on SPF and DKIM that are a mistake. Things are probably worse for big organizations but as a small email server, technical rejections are usually the right call. The only exception is mailing lists, but the dozens of people who still use those can usually figure out how to add an exception for them.
The issue here generally boils down to the defining difference between a generalist Admin and a Messaging Admin. The generalist can follow instructions, and nearly all the instructions out there stop at the point where SPF/DKIM/DMARC are successfully implemented. A generalist worth their salt will then fill in the gaps if they can', and knows this isn't where you stop when you want mail deliverability. There's a higher bar.
If you follow instructions written by non-professionals blindly you don't ever reach the point where you get to quality work.
Google, Microsoft, and the other large ESPs don't refuse to relay your email based on secret internal factors. This is what the non-professional people say to falsely justify why they can't do something.
Google and Microsoft publish the internal factors they use in the form of whitepapers at the industry working group. Its not ready made, and there are a lot of them, and they may not release their specific implementation details, but the metrics are there and often are based in weighted form (reputation-based systems).
If you follow them correctly, and set up the appropriate reporting accounts, and maintain those accounts, you won't have these problems. You generally only have problems once you've violated guidelines continuously, which happens when you rely on, or are unable to discern between qualified and unqualified help.
The factors are published at https://www.m3aawg.org/
Every professional that specializes in email or messaging that I know of is well aware of this.
People don't have the same vitriol when it comes to comparing Generalist Admins to DBAs, and this is the same with any specialized niche.
If you need email and messaging to work in a complex environment, you hire a person that specializes in it.
For what it's worth, I've started seeing cybersecurity insurers requiring riders and extra payments if you don't block Russian IPs.
Unless you own a global business, i see no reason to even allow other countries access. The potential for attacks is too great, especially from some very specific countries.
If you set DMARC to report, you’ll get notices from remote email systems when they receive noncompliant emails with your domain in the Envelope From field. Those reports are where you’ll see Russian IP addresses show up when they are trying to spoof your emails.
But there is no way to block them because neither the senders nor receivers are on your infrastructure. The best you can do is set a reject DMARC policy and hope everyone follows it.
The reasoning you provide doesn't differentiate, and speaks more of frustration which naturally comes with any area you aren't steeped in, or knowledgeable about.
Marketing should get their own (sub)domain for sending their missives, that way the primary corporate domain's reputation is not harmed.
Unless you want to run the risk of outgoing e-mails from Finance / Accounts Receivable to be sent to other companies' Junk folder.
- spammers have 1 system to set up in order to spam. They get it right.
- company admins have dozens of projects, of which this is a tiny one, with zero ROI to the bottom line (if people don't consider how critical security is). So they delay.
- companies often have dozens of systems integrated, when I set up DMARC/DKIM the first time for my company, a bunch of email tools broke, we had to do a bunch of leg work, took us a month end-to-end. The value was recognized when we almost lost 20k to a "ceo emails you" scam. But until then it wasn't a priority.
- we didn't even have a full IT, i just stepped in because I cared enough.
- my current company has a dedicated security team. These holes are plugged VERY quickly.
You can set your policy to reject, that will deter the Russians from using your domain.
I doubt Russian spammers will care about the difference to be honest. If they accept that their email will be delivered to spam folders, why would they care that the email gets silently dropped? In neither case anyone is going to fall for them.
But sure, Linkedin emails are definitively not spam and their dark-patterns at adding you at n+1 emailing list doesn't get them banned from the big (or any?) provider.
The logic isn't even that bad. SPF and DKIM serve to prove to the email who the sender is. That doesn't mean much if the sender is a spammer. Verifying identity claims is only the first part in checking email for spam, the harder part is checking if that identity is someone you trust.
When you email Outlook or Google, you're better sending more than a few every single day, and the recipient better manually drag those emails from their spam folders to their inbox, or they're all being learned as spam.
But if you have a regular decent size of emails coming from your domain, that is more likely to be spam than if you have a small number of intermittent emails coming from a domain.
The domain is new and didn't send a single email until I tested it.
Edit: The domain is actually a bit old but was parked/inactive for a while, though the email was used only for receiving.
We had/opted to acquire the services of a company "expert in email deliverability" (Return Path), who somehow provided detailed metrics of how our IPs were scored by MSFT. I always wondered why MSFT didn't provide those scores by themselves, and how a 3rd. party could have access to them.
Re. your comment... slow ramp-up is the only way, with constant monitoring of deliverability and consequent adjusting of recipients (i.e. removing those who do not open or hard-bounce). I did also wonder if paying that company perhaps gave us a headstart when adding new IPs...
Most home and VPS IP ranges have negative rep.
I've been successfully using VPSes to send emails for 20 years.
I would imagine that on the corporate side, your employees could do the same. Beyond that, if you're sending spammy stuff, have unsubscribe headers and links in emails.
I've been self-hosting mail for me and my family for about 20 years and don't send nearly enough mail to have a "reputation" with anybody. Still, I don't have any problems with deliverability of mail.
This little bit of wisdom gets passed around all the time, but it's actually not true. You can send email from a brand new domain to Google and Microsoft and whoever just fine. What you can't do is send email from a brand new domain, and a brand new email server--or an email server on a VPS, or an email server on a residential IP. Residential IP blocks are almost completely blocked, because of unsecured devices being used to send spam, and VPS blocks have the same problem. You can get around this by using a mail relay, or building your domains reputation on a server that already has a good reputation.
So what options are left for a self-hoster. Colo?
The root problem is that we don't actually need to keep track of email server reputation. No one says to themselves "Huh, this is from a Gmail address, it must be legit". We really want to keep track of sender reputation. We need to be able to treat anonymous email differently than email from people we actually know. That implies that we have some work to do on the problem of identity. As it is, there is not even a way for a known email sender to securely introduce an unknown email sender. You know, the way that regular human people normally are able to transfer identities from one to the other.
That paragraph is incorrect. SPF/DKIM is not about reputation. The main purpose is preventing domain impersonation from unauthorized senders. E.g. mail servers will reject fake emails from "upofadown@microsoft.com" because you don't control any email servers that's whitelisted in microsoft.com DNS TXT records.
E.g. I was able to register a brand new .com address and then successfully send to gmail and MS Outlook accounts within minutes because I had proper SPF/DKIM in the DNS records for that new domain. That new domain had zero reputation and yet Gmail accepted it because SPF/DKIM was configured correctly -- and -- the underlying ip address of the server it came from had a good reputation.
If SPF/DKIM was truly about "reputation", it would mean I'd have to wait days or months for reputation history to build up before Gmail accepted it.
Tin-foil hat time, but I've always thought there was nothing unintentional or "unfortunate" (from Google's perspective) about this.
They serve to authenticate that the sender is somewhat connected to the person who controls that sending domain.
There is: gpg/pgp signature, but many people find it complicated, primarily because they are reluctant to read the documentation. And it’s popular to criticize it, especially here on HN, in favor of various half-baked alternatives.
Could I, as a trained programmer, use PGP and GPG? I'm sure I could if I spent some time reading about it. Could my 90 year old grandmother, who is otherwise quite comfortable with email and whatsapp? No, not to any meaningful extent.
That's what kills a lot of these "perfect" implementations.
HN members tend to be nerds, and we don't really have an issue with setting stuff up (many HN IDs, for instance, have Keybase auths).
Most non-HN types have no patience for that stuff. Security needs to be made accessible and easy-to-use, before the vast majority of folks will implement it. That's the single biggest conundrum, IMNSHO.
Deleted Comment
Or if we can't trust users to handle TOFU, then some token/unique address/whatever that we can exchange face to face to enable trusted communication.
The simplest solution to that would be an "only show me emails from people in my address book" filter. That would mostly echo how we treat user trust on all other platforms. Genuinely surprised this doesn't exist in most email clients (or does it and I have just overlooked it so far?)
Of course that's only a partial solution and wouldn't work for accounts where you expect unsolicited mails from people you don't know. I'd see it more as a "low-hanging fruit" solution. You could also expand the heuristic, e.g. also consider previous conversations, mailing lists, etc.
(Interestingly, the "introduce a friend" functionality would come for free: You can already send contact details as a VCard in an attachment. When receiving such a mail, some email clients will show a button to quickly add the contact to the address book.)
I actually think this would work fine. Imagine a quarantine inbox for new emailers that the user must scan and approve/block. This is exactly what hey has implemented.
We actually do. Is the server allowing anyone to sign up so that it's sending 99% spam, or does it have a lot of anti-spam measures so sign-ups can't be automated and it blocks accounts as soon as it detects them sending spam?
That's exactly what PGP's web of trust model is for. Someone you know, and trust, can sign and send you a public key of someone that they trust.
This new key will be automatically trusted in your trust store because it's signed by someone you already trust, although in a lesser trust level to account for the degree of separation. If you later verify that key out of band you can upgrade it to a higher trust level.
SPF/DKIM, as well as TLS etc., is just stupid shit we do because we're too lazy and/or incompetent to make web of trust work for us. It's not a technology problem, it's a human problem.
Having key signing parties for the entire world wide web does not seem scalable to me.
* https://en.wikipedia.org/wiki/Key_signing_party
SPF/DKIM is literally how you establish sender identity instead of relying on the IP address of the email server so it is ironic to claim that they have anything to do with server reputation while lamenting a lack of sender reputation mechanisms.
Currently, SPF/DKIM are mostly used to prevent fraud, but they also provide the best tool we have to build sender based reputation systems.
Indeed. I am on a few mailing lists and many people on them host their own email. I use Gmail so that means visiting my spam inbox once a day to click "not spam" upward of a few dozen times. Day in and day out, same people end up in Googles spam folder. It's bullying and sabotage.
> No one says to themselves "Huh, this is from a Gmail address, it must be legit".
Indeed. I get a shit load of spam from google addresses so reputation is not there.
> We really want to keep track of sender reputation.
This is the hard part but I do think it's something to think about. I should be able to get an email from a known sender every time without $emailprovider making that decision for me. Some sort of attached signature or key that is proof of sender identity so I can route that right into my inbox.
You'd think spammers would've learned to avoid SPF domains at the very least but they haven't, so despite SPF/DMARC/DKIM failing to get anyone out the spam folder, the technology is still catching spam bots.
In the early-to-mid '10s, before SPF/DKIM/DMARC became the law of the email land, one had to be much, much more careful with phishing emails, checking the wording, the logos, etc, because 9 out of 10 of them appeared to come from the actual domain the email purported to be from. In the past several years (I honestly don't know exactly when the change happened; I don't get a huge amount of phishing emails), it's shifted so that the first thing to check is the sender address. Usually that turns out to be some nonsense string @gmail.com or some long garbled domain.
DMARC is nice though. It won't stop spam. It won't stop spoofing. But you will know that someone somewhere is spamming people using your domain name. How awesome. :)
Of course, even with hard fail spf and dmarc, I still see some bounces from spam where some server accepted the mail to deliver it elsewhere and the next server denies it, so the first server sends me a bounce.
—⁂—
1. I tried responding to a Chromium bug tracker message by email a couple of months ago, and it failed me:
> Unfortunately, your email to create/update an issue was not processed.
> Reason: SPF/DKIM check failed. Please ensure your domain supports SPF (https://support.google.com/a/answer/178723) and DKIM (https://support.google.com/a/answer/174124). If your domain does not support them, please use the Google Issue Tracker UI (https://issuetracker.google.com).
Trouble is, this is simply not true. My SPF and DKIM are fine. This makes me wonder whether the email ingestion system is simply broken for everyone.
—⁂—
2. I got involved in setting up a Google Workspace for someone a few months back, and the entire tool that their own documentation instructs you to use to check things, https://toolbox.googleapps.com/apps/checkmx/, has been laughably broken for years, sometimes not working at all, but mostly producing misleading nonsense results (e.g. claiming domains have no mail server set up when they do).
Then, to make it even more absurd, the feedback link they give you, https://toolbox.googleapps.com/apps/main/feedback?toolname=c..., iframes https://docs.google.com/a/google.com/forms/d/e/1FAIpQLSdnlp8..., but you haven’t been allowed to iframe such documents for I don’t know how long so it doesn’t load, and even if it did, it’s a private form that only Googlers, I suppose, can fill in. And there have been plenty of reports about all of this for years, and it’s still broken.
It is correct that SPF/DKIM does not really avoid spam, because spammers are not stupid and can read these standards like anyone else. However, before SPF/DKIM, I remember that I got a ton of phishing mails with FROM containing "support@paypal.com" or similar. Then came Bayes spam filtering, and that would move legitimate mail from Paypal to spam, because obviously, the phishing mails are quite similar.
This problem has pretty much vanished, because Paypal clearly denotes which IP addresses are allowed to send mails from that domain via SPF and the client can verify the mail via DKIM. For instance, Spamassassin makes sure that mails with correct DKIM and from paypal.com get a massively reduced spam score so that your Bayes filter will not move it to spam. This is hardcoded for a lot of domains (see *welcomelist_dkim.cf).
I run my own email server. Most spam crap cannot pass spf/dkim. Although this post has caused me to sit up and notice that the trendline is moving in the unfortunate direction, where I'd say 3 years ago the ones that pass were about 1/4, today it feels like 40-60% pass. The amount of mail I get that I expect, passes spf/dkim at around 90-95%
I suspect the delta between their any my results are the very restrictive sender rules I have prior to accept. In addition any_address@domain goes to my default mailbox, so I'm also probably selecting for laziness a bit more than most.
I also publish an email address without obfuscation on my site, which is getting very little spam, (near zero) which makes me wonder if most spam has given up on scraping the Internet for emails these days.
Web scraping gets you a lot of fake emails, company sinkholes, and other low reward stuff. Paying $20 for 100k confirmed real emails with names? That’s gold.
I was dreading this step as I hadn't done it before but turned out to be a breeze thanks to that.
You need IT to list all of the reporting tools, customer service to tell you about their support system, marketing to tell you about their mailing list tool of the week, the sales guys to warn you they're using this new AI email enhancer, and somehow get that shady email forwarding service the CEO uses to give up their mail server IP addresses. Then you need to figure out how to get coverage for all of those tools and keep on top of them whenever something changes.
A lot of companies promise to do great things for you if you just enter the email address you'd like to send email from, and a lot of people gloss over the important details because those sound hard and when they tested the tool on their personal email it worked fine so that's probably unnecessary anyway! Managing email for a corporate domain can be like herding cats.
https://www.fastmail.help/hc/en-us/articles/360060591153-Man...