Feel like it's obligatory to say this for people unfamiliar with Usenet, but there are text groups (think text threads like a mailing list or forum) and binary groups (shows/movies/software/etc). The size growth is due to the latter.
I use both types. Text groups almost certainly fell off a cliff earlier this year when Google Groups shut off their spam posting gateway. http://www.eternal-september.org/ is a good free project for the text groups. I've been a newsdemon customer for years but they suck for text groups (their headers got messed up a few years back after a move from highwinds backbone to usenetexpress backbone).
> Serious question: why does anybody use Usenet for pirating rather than torrents?
No need to seed which reduces legal liability and bufferbloat issues on lines with anemic upload speeds. Most torrent clients penalize peers that don't contribute to the swarm. It's also really convenient to plug in automated downloads with a decent index and sabnzbd.
>Most torrent clients penalize peers that don't contribute to the swarm
I agree with you and I also want to add that I've had problems with trackers penalizing me because i can't get my ratios up since hardly anybody is torrenting the same outdated 1980s mecha animes i am. I don't usually have troubles finding or downloading them because there's always at least one guy who has it on a seedbox running 24/7, but my ratio is guaranteed to be stuck at 0.0 forever.
I actually got autobanned from a private tracker once because i downloaded a bunch of comic books which were all miniscule in terms of size and also extremely niche. this series was being distributed as one issue per torrent instead of a big archive torrent, with each torrent being on the order of a few MB. which resulted in me having several dozen torrents with a 0.0 ratio and therefore looking like a leech.
Download speeds, no ratios, very low risk of any action by right holders. Would be things I would consider. Also setting VPN right is not entirely untrivial, you could get appearance it being in use, but traffic actual passing through other ways.
There are many things I've found on soulseek that are simply not on the public (and few private) trackers I check. I can only imagine the same applies to usenet.
similarly, soulseek is limited to 1 peer at a time, which some might consider "ill-suited to the task"
You can still misconfigure or leak your real IP using a VPN if you are not completely right in how you set it up. This leaves you open for large-ish claims in countries like Germany because technically you are uploading and that's illegal. Usenet doesn't have the uploading component so that risk doesn't exist.
> Serious question: why does anybody use Usenet for pirating rather than torrents?
In addition to the other answers, Usenet binary newsgroups are much older than torrents. So another reason is that it simply never died. Torrents didn't replace Usenet, they just complemented it.
If you live in a place where downloading is a gray area legally, it could save you a lot of potential hassle. Even if you win in court, it is far cheaper and better to have never been summoned in the first place.
Usenet, in my experience, is reachable through a variety of easy-to-use web-based providers. Easynews, etc... I agree that it's ill-suited in the sense that often the files are split into parts and often need re-assembing or other such work. But it's fairly trivial stuff that a user may already be familiar with. Other than that it's super easy. Torrents, at least in my experience are not so straightforward due to the required installation of a client. I think users are hesitant to install it on their machines. Just my take based on limited experience.
Honestly there's way more there, and you get consistent solid speeds. Find a provider with a lot of retention and you can find almost all mainstream media regardless of it's age. (Public) torrents tend to track what's popular and quickly fade. The masses seem to favour low size encodes too, so if you're looking for more quality (and again, public trackers) you're usually much more out of luck.
It’s really not that bad unless you really don’t want to be found. Most DMCA/copyright firms just do a basic investigation which stops at the IP AFAIK.
Mullvad and I’m sure other modern VPN clients have “kill switches” built-in that shuts down traffic if the tunnel isn’t on. You can also do a leak check before starting anything up.
Wouldn’t seedboxes be far more dangerous? The provider absolutely can see what you are doing on the box and provide that to the authorities whereas a VPN provider that doesn’t log activity, well there’s no paper trail at least retroactively.
Based on this table there's been a 1100% increase over the last 7 years, with a 60% increase between 2020 and 2021.
I'd be interested to see WHY this is the case. Is it attributable to a larger share of data that cannot be compressed vs more compressible data (e.g., Warez/Movies)?
It just seems highly unlikely this is driven by a growing user base; but, without more details other than this data table, I am at a loss for the reasons why.
It is a growing user base. Streaming services were convenient, now they are not (ads, increased cost, reduced content availability, fragmentation).
> It just seems highly unlikely this is driven by a growing user base;
If you are old enough, there was a time when everyone pirated stuff due to the alternative being rather expensive or unavailable (physical media). Then a golden age of streaming services that were cheap and had high availability basically killed torrenting for the general public. Now people are returning to piracy as the streaming services got worse for reasons I stated above.
One of the most popular types of content is now remuxed (not reencoded or recompressed) 4k blu-ray rips that can be anywhere from 50-150Gb per disc. In the DVD/Blu-ray era movies were often reencoded to lower bitrates the way streaming services now do and had tiny original sources. Those files were orders of magnitude smaller (600Mb-1.5Gb typically).
The people who still care enough to pirate in the era of ubiquitous streaming are often doing it explicitly to get the best possible quality, because no streaming service now offers anything of comparable fidelity for any price.
rn seems to have had it right: ripped from my youth, the pre-posting warning comes to mind:
“This program posts news to thousands of machines throughout the entire civilized world. You message will cost the net hundreds if not thousands of dollars to send everywhere. Please be sure you know what you are doing.”
So 300TB * 365 = ~110PB for a mirror to have 1 year of retention.
That's pretty insane lol. How many mirrors are there that can actually manage that much storage? If you're using 20TB disks that's ~5500 disks per year with zero redundancy. Double or triple that for a bare minimum... not counting the load of actually serving that data to everybody.
How is this economical for anybody at this point? Or are these Usenet mirrors all massive businesses that can support running hundreds of PBs or storage and I'm just naive?
> Or are these Usenet mirrors all massive businesses that can support running hundreds of PBs
ding ding ding. The market consolidated toward only a few large backbone operators. Here's a picture of what the network looks like: https://svgshare.com/s/14tF.svg
I'm also curious -- how come you hear big stories about governments going after torrent sites, which don't even host the pirated content.
But I've never heard big stories about the government going after big Usenet providers. Do they? If not, why not? Or does it just not make the news? Or am I just not paying attention?
Usenet files get taken down all the time. Rights holders send notices to the Usenet providers and they remove enough of the binaries where they can't be repaired by the PAR files.
Indexers are separate entities. Those are the ones who catalog and host the .nzb files. You see those go offline far more often but Usenet is still pretty small compared to torrents so they don't get reported as much.
> But I've never heard big stories about the government going after big Usenet providers.
The most litigious government has a 'safe harbor' provision most of the major Usenet backbones comply with: https://www.copyright.gov/512/
And they do take down a substantial amount of content every day, so while everyone knows piracy is common there, what more are they supposed to do? At this point the main effective thing governments can do to make it annoying to use Usenet for piracy is target the indexers (and they do this, but more always move in to replace the ones that are shut down)
AFAIK, files uploaded to Usenet still need to be ASCII encoded using something like uuencode or base64, which increases file sizes to something like a third larger than their original sizes. So you can decrease the amount of data you need to store quite significantly by only storing the unencoded versions.
After that, deduplication can probably bring you down at least another 50% - how many of these files are just the same things being posted over and over? Probably quite many of them. Store a file once with a database tracking its SHA1 hash, and whenever you see a file with the same hash come in, throw it away and instead store a reference to the first file.
Or are people using Usenet as a way to send encrypted messages in a way that makes traffic analysis more difficult? (If 50,000 people download everything to a group, and post encrypted or steganographic message to that group, then it's easier than seeing that X sent an email blob to Y.)
Readit News
I use both types. Text groups almost certainly fell off a cliff earlier this year when Google Groups shut off their spam posting gateway. http://www.eternal-september.org/ is a good free project for the text groups. I've been a newsdemon customer for years but they suck for text groups (their headers got messed up a few years back after a move from highwinds backbone to usenetexpress backbone).
It seems so fundamentally ill-suited to the task.
And if the answer has something to do with privacy or warnings from your ISP, it seems like VPNs would be the answer.
What am I missing?
No need to seed which reduces legal liability and bufferbloat issues on lines with anemic upload speeds. Most torrent clients penalize peers that don't contribute to the swarm. It's also really convenient to plug in automated downloads with a decent index and sabnzbd.
I agree with you and I also want to add that I've had problems with trackers penalizing me because i can't get my ratios up since hardly anybody is torrenting the same outdated 1980s mecha animes i am. I don't usually have troubles finding or downloading them because there's always at least one guy who has it on a seedbox running 24/7, but my ratio is guaranteed to be stuck at 0.0 forever.
I actually got autobanned from a private tracker once because i downloaded a bunch of comic books which were all miniscule in terms of size and also extremely niche. this series was being distributed as one issue per torrent instead of a big archive torrent, with each torrent being on the order of a few MB. which resulted in me having several dozen torrents with a 0.0 ratio and therefore looking like a leech.
With the exception of the rare, well seeded torrent, a Usenet download will proceed at full speed while the torrent will trickle along at 34kb/sec.
> It seems so fundamentally ill-suited to the task.
It is. But it is there, so....
similarly, soulseek is limited to 1 peer at a time, which some might consider "ill-suited to the task"
anyways I love slsk
clients exist:
- https://github.com/nicotine-plus/nicotine-plus
- https://github.com/slskd/slskd
100% (down) bandwidth saturation, no VPN, no letters from lawyers, and no using my (asymetrical) upload - what is not to like?
You need to find an indexer, the best ones you need to pay for or get an exclusive invite.
Then you need to find a news provider, and pay them too.
On top of that you need software that can plug into the indexer and the server and actually download the stuff.
In addition to the other answers, Usenet binary newsgroups are much older than torrents. So another reason is that it simply never died. Torrents didn't replace Usenet, they just complemented it.
Mullvad and I’m sure other modern VPN clients have “kill switches” built-in that shuts down traffic if the tunnel isn’t on. You can also do a leak check before starting anything up.
I'd be interested to see WHY this is the case. Is it attributable to a larger share of data that cannot be compressed vs more compressible data (e.g., Warez/Movies)?
It just seems highly unlikely this is driven by a growing user base; but, without more details other than this data table, I am at a loss for the reasons why.
> It just seems highly unlikely this is driven by a growing user base;
If you are old enough, there was a time when everyone pirated stuff due to the alternative being rather expensive or unavailable (physical media). Then a golden age of streaming services that were cheap and had high availability basically killed torrenting for the general public. Now people are returning to piracy as the streaming services got worse for reasons I stated above.
The people who still care enough to pirate in the era of ubiquitous streaming are often doing it explicitly to get the best possible quality, because no streaming service now offers anything of comparable fidelity for any price.
gotta hope they decide your connection is capable of 4k (and they decide they want to spare you the bandwidth)
“This program posts news to thousands of machines throughout the entire civilized world. You message will cost the net hundreds if not thousands of dollars to send everywhere. Please be sure you know what you are doing.”
That's pretty insane lol. How many mirrors are there that can actually manage that much storage? If you're using 20TB disks that's ~5500 disks per year with zero redundancy. Double or triple that for a bare minimum... not counting the load of actually serving that data to everybody.
How is this economical for anybody at this point? Or are these Usenet mirrors all massive businesses that can support running hundreds of PBs or storage and I'm just naive?
ding ding ding. The market consolidated toward only a few large backbone operators. Here's a picture of what the network looks like: https://svgshare.com/s/14tF.svg
But I've never heard big stories about the government going after big Usenet providers. Do they? If not, why not? Or does it just not make the news? Or am I just not paying attention?
Indexers are separate entities. Those are the ones who catalog and host the .nzb files. You see those go offline far more often but Usenet is still pretty small compared to torrents so they don't get reported as much.
The most litigious government has a 'safe harbor' provision most of the major Usenet backbones comply with: https://www.copyright.gov/512/
And they do take down a substantial amount of content every day, so while everyone knows piracy is common there, what more are they supposed to do? At this point the main effective thing governments can do to make it annoying to use Usenet for piracy is target the indexers (and they do this, but more always move in to replace the ones that are shut down)
After that, deduplication can probably bring you down at least another 50% - how many of these files are just the same things being posted over and over? Probably quite many of them. Store a file once with a database tracking its SHA1 hash, and whenever you see a file with the same hash come in, throw it away and instead store a reference to the first file.
A backchannel to download papers from sci-hub?
Or are people using Usenet as a way to send encrypted messages in a way that makes traffic analysis more difficult? (If 50,000 people download everything to a group, and post encrypted or steganographic message to that group, then it's easier than seeing that X sent an email blob to Y.)
Or, Usenet as the new numbers station?
Deleted Comment