Fetchmail grew to be a mail transport swiss army knife and encompassed half baked functionality of a domain-wide MTA rather focusing on single user async email consolidation and collection for a logical single account, which is largely what this rant is about. Half of the half-bakedness is due to lack of MTA protocol support for MUA interactions on the receiving side.
pop3 was never meant as a spool for MTA to MTA email transfers, but was instead meant for an MUA to access. It's not fetchmail's fault that the envelope TO isn't available via pop3. That, at the time, email service providers bolted on entire domain delivery to a single mailbox, and then users of those services put multiple recipients behind that single mailbox. Fetchmail made this work as best it could. Also missing at the time was any kind of serious connecting-client-as-MTA or MTA-to-MTA authentication (rather, relying on DNS and TCP!), which might have made async, receiver controlled MTA initiation more workable.
This was also due to common SMTP MTAs not having good (or at least easy/obvious to configure) store-for-later-pickup functionality where an MTA could query for email destined for itself rather than be always-on connected to the internet waiting for another MTA to initiate delivery. Fetchmail filled a hole that existed in email delivery at the time, but it was really a hole that should have never existed. The quality of the fetchmail code is another, albeit serious, issue in itself.
These days we have a better understanding of the differences between MTAs, MSAs, and MUAs.
These days we have a better understanding of the differences between MTAs, MSAs, and MUAs.
I don't want to be "that guy" but we've known about this distinction since the 1980s. I know, because I was fringing in the discussions in standards ml during the x.400 years.
You are "that guy". I didn't say we didn't know about this distinction, I said we have a better understanding. It used to be the case submitting originating email on port 25 was accepted. Now, if you have that configured, it's wildly considered that You're Doing It Wrong™.
Many email clients, like Eudora and Thunderbird, were submitting email on port 25 for a long time, in the clear, where the only protection was very loose source IP address restrictions for outbound, otherwise you'd be an open relay. Using different servers for inbound vs outbound mail submission was a relatively rare thing too, because the routing can be a bear to set up. SMTPS (SMTP over SSL) on port 465 wasn't official and was locally used where it could be supported (by the client) and where an admin had bothered to configured it. Many email clients didn't have support for even using a non-standard port for any thing mail related, so even once the mail submission port 587 was standardized it was often a problem getting a client to use it or providing authentication on it.
But even the OP is saying that fetchmail is a problem because it's trying to be an a full, transparent MTA when it's really a glorified MUA. The fact that fetchmail (and its ilk) talk MUA protocols to receive email (pop3, imap) and can then turn around and reinject that message into the mail stream using SMTP to implement local delivery is where a lot of the problems with fetchmail come from (such as munging the headers or creating email loops with mailing lists).
I actually learned the difference between MTAs, MUAs, etc because of fetchmail. Fetchmail kept randomly crashing on select spam messages and my bug reports went unheeded (and I wasn't what anyone would call a programmer) so I needed a new way to download my mail. So I wrote a crappy Perl script. All it does is use standard Perl modules to download the mail, and pipes it off to an MDA (postfix, maildrop, etc) to deal with them.
After nearly two decades, no matter what system I plunk it on or mail provider I use, the same stupid script just keeps working.
Once I left Gmail I was interested in something faster and more efficient, because I was using the transfer tool for more than just backup, it became the main way I receive email on my machine. Then the mail gets indexed (for text search) by another program and displayed by a third.
Anyway I found mbsync, which is just this amazing, efficient, very solid transfer program that's been around for many years just quietly doing its job. I almost wrote it off because its homepage is on sourceforge but if you poke around a bit you'll see it is actively developed, there's an active mailing list, etc. I love it. It's kind of interesting how this project I had never heard of just does its job well and works and the program by the semi-famous software pundit whose programming essay went viral really does not (from what I have heard).
fetchmail can run as a daemon and fetch from imap with IDLE. As far as I can tell, mbsync doesn't do this.
I haven't found a single other program that does this well; there are tools to sync entire directory trees that can idle, but they don't seem to support "download everything and delete it" the way fetchmail does.
Many e-mail services throttle new connections, so it's not the difference between getting your mail now and 1 minute from now, it's the difference between getting your mail now and 5-10 minutes from now.
[edit]
Another thing keeping me on fetchmail; I have all my filters setup with maildrop. Most other programs want to deliver to a local folder directly. Maybe there's a way around this?
Same here (like another sibling reply). I really don't get the complaints.
For private email consolidation, I've been running fetchmail for probably 20+ years now without any hiccups or complaints whatsoever. I haven't had to touch my Fetchmail+Postfix+Maildrop setup in like 15+ years, except for the postfix part in the sense that nobody accepts emails from dialup IPs any longer, so everything I _send_ goes through the proper mail server, instead of my own.
My actual email provider (that I've also used for 20+ years, i.e. no, it's definitely not and never will be GMail) does throttle connections via POP3 or IMAP to 3 connections per 5 minutes. That's completely fine. I really don't care whether I get an email now or in 5 minutes from now. I check it way less often than that and I do definitely _not_ have notifications turned on for my email client. And when I do care (coz you know, some "2FA" BS email, I check via my provider's app or log into my server and start a `fetchmail`.
I just got started with mbsync and am having several issues with Gmail and Outlook.com. These are old accounts and I do have well-behaved IMAP emails from smaller hosts, but it would be nice to manage everything with mbsync. Trashing emails in mu4e and then syncing results in them being deleted from my local [Gmail]/Trash while they somehow remain in the remote. I thought Outlook.com would be fine considering they use more standard folders rather than the tagging system which seems to result in duplicates, but I have the same issues there. This is with Expunge None, which I expect would leave emails alone until the server empties them from the Trash.
The bizarre thing is that no one online seems to have these issues? The mbsync configs for Gmail just seem to work for people, or something. I do have auto-expunge turned off in Gmail, so I’m kind of at my wit’s end with it. It’s a shame because it is an amazing program. It synced 10,000 emails without a hitch. I only started getting issues when I tried to sync from my maildir to the IMAP store.
I’ve only run mbsync unidirectionally as a backup utility (in addition to tarsnap on my Mailserver). I wonder if the bidirectional features just aren’t used that much?
I've used fetchmail for years to get email from IMAP inboxes (mainly Exchange server) and never had a single problem with it. Was surprised to see that there are major complaints.
I used to use UUCP to get my email when I was on a dialup, non permanent connection. I was able to host my own mail server. There were a lot of technologies back then helping the internet stay somewhat decentralized. Now I, like most, just use Gmail.
Well, part of this is on us. Let's think on the reasons Gmail is so popular:
1. It's very easy to get to.
2. It has incredibly fast search that has 0 setup.
We have never really even tried to address problem 1 as an open source community. Networks, name lookup, and VPNs remain incredibly complex topics that beginners cannot hope to wrestle with. The best we have is .mdns which either works magically or perversely refuses to work.
Similarly for free text search, the software world simply hasn't delivered a lego-like solution for email search. You CAN rig up any number of open source projects but it is neither easy nor instant. And even other professional products like Apple Mail struggle with a mere gigabyte of email.
Despite the fact that it's 2021 and every successful email provider aggressively solves these problems, the open source world still debates about the utility of ubiquitous search or pretends that local networking isn't a pressing problem.
"We have never really even tried to address problem 1 as an open source community. Networks, name lookup, and VPNs remain incredibly complex topics that beginners cannot hope to wrestle with."
I kinda disagree. It's probably easier than ever to set up your own mail server, in some abstract sense. You can get a virtual machine, use docker, heck, someone can hand you a complete image that you just have to bring up and set up with some config.
The problem is, it literally doesn't matter how much the 'open source' community comes together, it simply can not provide a turn key solution as good as
Desired email account: [________]@gmail.com
Password: [________]
Verify Password: [________]
[X] I agree to have all my data used in arbitrary ways
It's not possible. There is no way to set up a server that easily, even in principle.
Or at least, not in a sane way. I can set up a site where you feed me your credit card number and pick a domain name, and I set up your AWS account for you, register your DNS name for you, configure DNS, and stand up everything you need and set it all up... but then we've got a split ownership interest. I can hand it all back to you, but you don't understand the setup. I can give you root on the system, but when you change anything, my automation stops working.
Another issue that needs to be solved is 'bad actor'. Basically someone gaming the system and overloading it with spam. There is no real nice neat way to fix that. Instead it is a mishmash of blacklists/whitelists/blocklists and sorta intelligent alg filtering. Getting that all filled and working is not trivial either. Oh its 'doable' but kind of a pain for even someone with decent experience at it.
Agree 100% on both of these, and #2 is seldom listed as a big issue. Had IMAP addressed search in a better way, I think it would have made a huge difference.
Of course, it's not simple to set up a mail server that stays clean of spam RBLs, or that is Gmail-acceptable out the gate. But that's just the bar that got set as people went to Gmail because of the lack of other good alternatives.
I use offlineimap/mu4e over Exchange account at work, in parallel to a personal Gmail account. Would not say now that Gmail has a perceptible edge there.
Seriously, mu4e search on modern hw is incredibly fast (and I have a 10 year work email archive).
I used to run personal email server some years ago. The spam was harder and harder to manage - first local SpamAssasin was fine, the I had to tweak rules, then I tried distributed filtering (DCC)... and then I gave up and started paying for Fastmail. This ended up working much better than I could ever do.
Usenet was similar -- I remember looking at the newgroups which were at more than half spam (and some were 95%+ spam) and then just leaving for forums. (plus there were a major UX annoyance: unlike with email, there is no simple way to sync read status across multiple devices)
Our free(digital) land has been killed by a handful of bad actors. I no longer believe in "free for all" systems. The best we can do is many independent systems, each with its own rules and policies.
I run my own mail server with only Spamassassin for filtering, and I get little spam to my inbox in spite of my e-mail address being posted in plaintext across the internet. It has sufficed to 1) simply activate distributed filtering within Spamassassin, and 2) set a slightly more stringent standard than the default for regarding a message as ham, since any legitimately sent message will easily meet that higher standard. I also fed my entire e-mail archive to sa-learn, and what spam I occasionally get I also feed to sa-learn.
In ten years of running this mail server, I have only had one persistent spammer I couldn’t manage to get filtered with these settings, but then I just created a custom rule.
I think worst than the spam problem is the deliverability problem. Worst case I can hit the delete key on spam. (as an aside, most of it is immediately obvious from the subject line alone, which makes it hard to believe that it's such a hard problem). But getting mail delivered from a private server to major email providers like gmail, yahoo, outlook, is difficult.
Sadly, the upsides are just too good. I love decentralization, but I have to recognise that I've not lost a message since I moved to gmail in 2005; before then, losing entire mailbox files was not uncommon, as well as missing messages from unreliable mailservers, spam management, and so on. I'm sure I could have done better etc etc, but the bottom line is that with gmail-like services you don't have to. I enjoy doing sysadmin but not 24/7.
My first access to the web around '93 was via CERNs e-mail to web gateway coupled with UUCP exchange from my Amiga to my local BBS which exchanged with its upstream 4-6 times a day.
Me too! I ran AmigaUUCP on my Amiga, dialed up at 9600 baud to my upstream. This was around 1992 or so. I had Internet email and a Usenet feed of maybe 15 groups or so. Fun times.
> When mail is delivered to a POP3 maildrop, envelope information is destroyed. To combat this, you would need to tunnel the enveleope information in headers.
Isn't that exactly what Envelope-to is supposed to be? Protocol-level data? It's not really part of the e-mail, but added when sending it. If you forward the mail to another machine, there will be a new envelope.
I can see that it's all a gray-ish area with email (because email is a mess (which might be part of why it's been successful)) but I can't see a reason to complain about losing Envelope-to.
Would anyone that's well-versed in UUCP, Fetchmail and related mail protocols and technologies care to explain whether this discussion from 2001 (20 years ago!) is still relevant today?
This doesn't seem so much about UUCP vs fetchmail as about cathedral vs bazaar style of development. In hindsight, it's clear that both of these work, for better or worse, with each having its own disadvantages and benefits.
Think this story is a developer who submitted a patch to they guy who wrote the manifesto for open source. The patch was rejected for non-technical reasons, which led to the developer pointing out the irony of the situation. It's actually funny, and if you ever used fetchmail, you'd know just how accurate Terry Lambert's assessment of fetchmail ("abomination before God") was...
If you have any interest in open source development (which these days we all should given that 95% of the lower parts of our stack are open source projects) it's well worth reading the essay/book.
Readit News
pop3 was never meant as a spool for MTA to MTA email transfers, but was instead meant for an MUA to access. It's not fetchmail's fault that the envelope TO isn't available via pop3. That, at the time, email service providers bolted on entire domain delivery to a single mailbox, and then users of those services put multiple recipients behind that single mailbox. Fetchmail made this work as best it could. Also missing at the time was any kind of serious connecting-client-as-MTA or MTA-to-MTA authentication (rather, relying on DNS and TCP!), which might have made async, receiver controlled MTA initiation more workable.
This was also due to common SMTP MTAs not having good (or at least easy/obvious to configure) store-for-later-pickup functionality where an MTA could query for email destined for itself rather than be always-on connected to the internet waiting for another MTA to initiate delivery. Fetchmail filled a hole that existed in email delivery at the time, but it was really a hole that should have never existed. The quality of the fetchmail code is another, albeit serious, issue in itself.
These days we have a better understanding of the differences between MTAs, MSAs, and MUAs.
I don't want to be "that guy" but we've known about this distinction since the 1980s. I know, because I was fringing in the discussions in standards ml during the x.400 years.
Many email clients, like Eudora and Thunderbird, were submitting email on port 25 for a long time, in the clear, where the only protection was very loose source IP address restrictions for outbound, otherwise you'd be an open relay. Using different servers for inbound vs outbound mail submission was a relatively rare thing too, because the routing can be a bear to set up. SMTPS (SMTP over SSL) on port 465 wasn't official and was locally used where it could be supported (by the client) and where an admin had bothered to configured it. Many email clients didn't have support for even using a non-standard port for any thing mail related, so even once the mail submission port 587 was standardized it was often a problem getting a client to use it or providing authentication on it.
But even the OP is saying that fetchmail is a problem because it's trying to be an a full, transparent MTA when it's really a glorified MUA. The fact that fetchmail (and its ilk) talk MUA protocols to receive email (pop3, imap) and can then turn around and reinject that message into the mail stream using SMTP to implement local delivery is where a lot of the problems with fetchmail come from (such as munging the headers or creating email loops with mailing lists).
After nearly two decades, no matter what system I plunk it on or mail provider I use, the same stupid script just keeps working.
I used getmail for years and it is solid. duckerude posted this elsewhere in the thread, where the author of getmail rants a bit about fetchmail. http://pyropus.ca/software/getmail/faq.html#faq-about-why
Once I left Gmail I was interested in something faster and more efficient, because I was using the transfer tool for more than just backup, it became the main way I receive email on my machine. Then the mail gets indexed (for text search) by another program and displayed by a third.
Anyway I found mbsync, which is just this amazing, efficient, very solid transfer program that's been around for many years just quietly doing its job. I almost wrote it off because its homepage is on sourceforge but if you poke around a bit you'll see it is actively developed, there's an active mailing list, etc. I love it. It's kind of interesting how this project I had never heard of just does its job well and works and the program by the semi-famous software pundit whose programming essay went viral really does not (from what I have heard).
I haven't found a single other program that does this well; there are tools to sync entire directory trees that can idle, but they don't seem to support "download everything and delete it" the way fetchmail does.
Many e-mail services throttle new connections, so it's not the difference between getting your mail now and 1 minute from now, it's the difference between getting your mail now and 5-10 minutes from now.
[edit]
Another thing keeping me on fetchmail; I have all my filters setup with maildrop. Most other programs want to deliver to a local folder directly. Maybe there's a way around this?
For private email consolidation, I've been running fetchmail for probably 20+ years now without any hiccups or complaints whatsoever. I haven't had to touch my Fetchmail+Postfix+Maildrop setup in like 15+ years, except for the postfix part in the sense that nobody accepts emails from dialup IPs any longer, so everything I _send_ goes through the proper mail server, instead of my own.
My actual email provider (that I've also used for 20+ years, i.e. no, it's definitely not and never will be GMail) does throttle connections via POP3 or IMAP to 3 connections per 5 minutes. That's completely fine. I really don't care whether I get an email now or in 5 minutes from now. I check it way less often than that and I do definitely _not_ have notifications turned on for my email client. And when I do care (coz you know, some "2FA" BS email, I check via my provider's app or log into my server and start a `fetchmail`.
The bizarre thing is that no one online seems to have these issues? The mbsync configs for Gmail just seem to work for people, or something. I do have auto-expunge turned off in Gmail, so I’m kind of at my wit’s end with it. It’s a shame because it is an amazing program. It synced 10,000 emails without a hitch. I only started getting issues when I tried to sync from my maildir to the IMAP store.
Also, Gmail’s tags don’t map nicely onto folders.
There are also complaints about it being insecure and of low code quality, but those seem to rarely be the main complaint.
What has become of our beloved free(digital)land?
1. It's very easy to get to.
2. It has incredibly fast search that has 0 setup.
We have never really even tried to address problem 1 as an open source community. Networks, name lookup, and VPNs remain incredibly complex topics that beginners cannot hope to wrestle with. The best we have is .mdns which either works magically or perversely refuses to work.
Similarly for free text search, the software world simply hasn't delivered a lego-like solution for email search. You CAN rig up any number of open source projects but it is neither easy nor instant. And even other professional products like Apple Mail struggle with a mere gigabyte of email.
Despite the fact that it's 2021 and every successful email provider aggressively solves these problems, the open source world still debates about the utility of ubiquitous search or pretends that local networking isn't a pressing problem.
I kinda disagree. It's probably easier than ever to set up your own mail server, in some abstract sense. You can get a virtual machine, use docker, heck, someone can hand you a complete image that you just have to bring up and set up with some config.
The problem is, it literally doesn't matter how much the 'open source' community comes together, it simply can not provide a turn key solution as good as
It's not possible. There is no way to set up a server that easily, even in principle.Or at least, not in a sane way. I can set up a site where you feed me your credit card number and pick a domain name, and I set up your AWS account for you, register your DNS name for you, configure DNS, and stand up everything you need and set it all up... but then we've got a split ownership interest. I can hand it all back to you, but you don't understand the setup. I can give you root on the system, but when you change anything, my automation stops working.
Of course, it's not simple to set up a mail server that stays clean of spam RBLs, or that is Gmail-acceptable out the gate. But that's just the bar that got set as people went to Gmail because of the lack of other good alternatives.
Question is, is Gmail hegemony fixable now?
3. dealing with spam is a massive PITA
Seriously, mu4e search on modern hw is incredibly fast (and I have a 10 year work email archive).
Thunderbird plus global search
Recoll + client of choice
https://www.monperrus.net/martin/recoll-maildir-thunderbird
https://notmuchmail.org/
I used to run personal email server some years ago. The spam was harder and harder to manage - first local SpamAssasin was fine, the I had to tweak rules, then I tried distributed filtering (DCC)... and then I gave up and started paying for Fastmail. This ended up working much better than I could ever do.
Usenet was similar -- I remember looking at the newgroups which were at more than half spam (and some were 95%+ spam) and then just leaving for forums. (plus there were a major UX annoyance: unlike with email, there is no simple way to sync read status across multiple devices)
Our free(digital) land has been killed by a handful of bad actors. I no longer believe in "free for all" systems. The best we can do is many independent systems, each with its own rules and policies.
In ten years of running this mail server, I have only had one persistent spammer I couldn’t manage to get filtered with these settings, but then I just created a custom rule.
And you're right, the centralization is crazy, and opposite of what techno utopians predicted/promised at the time.
Isn't that exactly what Envelope-to is supposed to be? Protocol-level data? It's not really part of the e-mail, but added when sending it. If you forward the mail to another machine, there will be a new envelope.
I can see that it's all a gray-ish area with email (because email is a mess (which might be part of why it's been successful)) but I can't see a reason to complain about losing Envelope-to.
If you don't want to do that, at least read the wikipedia page: https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar