Readit News logoReadit News
Posted by u/arunharidas 4 years ago
Ask HN: Why don't many video hosting companies adopt PeerTube or P2P?
I recently found the peertube and its technology is very nice. You could deliver video using p2p torrent like streaming if there are more than 1 user watching it. For me it looks like the hosting provider gets a huge savings in terms of bandwidth. But I don't see many companies making use of this technology.

I think Bitchute is using p2p to deliver videos, or is it, really?

freefaler · 4 years ago
I have some experience with large-scale CDN I've cofounded some years ago. The short answer is - the perceived benefits you'd receive doesn't match the decrease in quality of service. So the economics doesn't work. Content distribution is widely skewed towards 1% of content makes 90% of traffic nodes so the cases that p2p may work for very popular content but the long tail is not easily cached/served. Try to watch/download a torrent file with small number of seeders and you'll get the experience you will be providing for the end user. Also internet networking is tricky and connectivity in different parts of the world can differ widely. South America or Asia have almost none of the peering hubs infrastructure that is available in Europe. (I've cofounded a CDN company and have some knowledge of the market)
samwillis · 4 years ago
This is more or less exactly why Spotify dropped P2P from its platform back in 2014.

Good summary here:

https://techcrunch.com/2014/04/17/spotify-removes-peer-to-pe...

"This is when the company took advantage of cached songs. Before today’s change, if you streamed a popular song for the first time, the client would download the song from other users, using peer-to-peer. All of this happened in the background, but it greatly contributed to making the overall user experience snappier."

"Yet, now that the company has many servers, using peer-to-peer in addition to direct downloads actually adds a bit of overhead. Moreover, the company has to maintain the peer-to-peer code base, and update it with each new version."

black_puppydog · 4 years ago
> the company has to maintain the peer-to-peer code base, and update it with each new version.

Really, that's (part of) their excuse? A bit lame, that. :|

sitkack · 4 years ago
So your origin needs to be able to handle the tail and P2P can handle the popular stuff no? It isn't all or nothing. P2P could absolutely be used to peak shave. CDNs also have cold start issues with long tail.

Time to first byte is pretty good, https://webtorrent.io/

To the OP, you should research some other options, are you sure that you are asking the right question?

giantrobot · 4 years ago
Streaming video is not like static files from a torrent. The term "popular content" is a bit of a misnomer. Nominally with streaming a single uploaded video will be encoded into a number of different versions at different frame sizes and bitrates packaged into individual stream segments. So while a particular video URL might be popular, the actual video segments served to viewers will depend heavily on the configuration/environment of all the viewing devices.

So a FHD stream for a video could be super popular while a 480p version might not have any viewers. When some 480p viewers load the stream they don't get any benefit from the P2P/cache popularity of the FHD stream.

This is compounded by the fact that for video time to first byte doesn't matter very much. A decoder can't do anything with a single byte or even a partial stream segment. It's the time to locally buffer a decodable stream segment and a steady reception of segments as the play head reaches their time stamp.

If a segment gets held up (as happens with P2P) it can't be decoded so you either pause playback or drop frames (if you've got later segments available). You want the media segments to cover at minimum a single GOP (Group of Pictures) which is a relatively large I-frame and the following smaller P and B frames. If you were to tune the video segmentation such that each segment was some tiny time slice it would balloon the bitrate since you need way more I-frames. Super long GOPs don't help either because segment drops end up ruining playback until a new I-frame is received.

Streaming video is hard enough with a reliable CDN, adding P2P makes much harder and orders of magnitude less reliable.

zinekeller · 4 years ago
> So your origin needs to be able to handle the tail and P2P can handle the popular stuff no? It isn't all or nothing. P2P could absolutely be used to peak shave. CDNs also have cold start issues with long tail.

No, it'll be genuinely be worse, as you can't really have efficient P2P today.

Let's start with the points; I'm not talking about Europe or US today. P2P or CDN is a coin flip there, and I'll agree that P2P is as reliable as a CDN. I'll also agree that theoretically P2P and CDN will be a coin flip (performance-wise, I'm leaving monetary considerations in this discussion).

Those points break down quickly in Latin America, Asia and Africa. I'll talk about Asia as it is where I'm more familiar of, but the points apply also in Africa and Latin America. Outside of Jio and actual hosters/colos (meaning nearly all of residential and mobile connections), IPv6 in Asia is actually more unreliable than you think (it should not be, but that's peering fights for you) and IPv4 is CGNATTed heavily so P2P is no dice. Even if somehow there's a unique IPv4 for every device, internet connections there is very top-down, unlike with Europe's mesh-like connectivity.

How bad? It'll be better to route Telkom Indonesia's connections to America than to another hoster located in Indonesia or Singapore (nearest regional hub), unless you're Akamai which have agreed to buy colocation space inside Telin just to have good connectivity. And before you comment, OVH in Singapore has the precise issues we're discussing (https://lowendtalk.com/discussion/172659/ovh-routing-issues-...). Thus, it is miles better to use your time to build good CDNs with good connections (or spend money to Akamai) than bothering with the issues you'll face with P2P. Moreover, you can concentrate your money into buying a better data link (either by renting/IRU a dedicated wavelength or even fiber pairs) than relying on a best-effort service.

Funnily, even bittorent downloads in practice seems to be concentrated to a few seedboxens - even for purely legal downloads like GIMP.

Postscript: technically P2P and CDN would be the best for reliability, but you're spending the time to do it (instead dealing with a single thing) and deal with the possible fallout (like Windows Updates before Microsoft decided to limit P2P to local networks due to bandwidth and privacy concerns). Speaking of bandwidth concerns, I promised myself to limit monetary concerns just to point out that's it's not good in actual deployment, but I forgot data caps (not just a problem in Africa but in the US too)! It'll be a disaster when P2P is not voluntary because you just wasted their precious money!

qwerty456127 · 4 years ago
> Try to watch/download a torrent file with small number of seeders and you'll get the experience you will be providing for the end user.

I always wondered why not combine the ways the media is served into a hybrid of P2P with the classic way by just using P2P (e.g. BitTorrent) and also seeding yourself (as the hoster). Just run a seedbox of the capacity you want/can and let the audience help you by handling a share of the load they can. Wouldn't this save you some bandwidth while keeping the same service quality?

matheusmoreira · 4 years ago
> let the audience help you by handling a share of the load

So why should we help these for-profit corporations? We're supposed to just donate our bandwidth to them? Are we getting a discount in exchange or are they just gonna pocket their bandwidth cost savings at our expense?

Seeding internet archive data is one thing. Letting corporations invisibly take advantage of resources we pay for is a completely different matter.

aasasd · 4 years ago
Archive.org seems to do something like that with its downloads via torrent: it appears as a web-seed for the torrents, and I usually snatch the files from those seeds faster than DHT seeds usually show up (which kinda undermines the idea, though).

Frankly, I never saw any other seeds on Archive's torrents—but one of those I snatched has a ratio of 3 now, so presumably it does work as intended.

e9 · 4 years ago
Roku can probably provide this service given they control hardware and can provide SDK for the apps to do it.
elsjaako · 4 years ago
This is what peertube does. That's why you can watch videos normally despite most not being popular.
jdrc · 4 years ago
That sounds like a perfect match for internet content which is often viral and temporary peaks. Ideally in a P2P world creators would care to have at least 1 node running which would serve the long tail.
iib · 4 years ago
I wonder if an asymmetric scheme would work best, even though it may be unbalanced? For example, for any one popular video you watch, while you watch it, you also have to download and seed a small part of an unpopular video, even if you don't see it?

Maybe bandwidth and storage caps would work better. You seed X MB, even if you don't need the full X yourself.

pmayrgundter · 4 years ago
İ also have experience building a CDN, tho not for video, and this sound right.

I wonder if this might be improved by an appointment system, to stream popular titles at pre-planned times so that people could benefit from others watching at the same time.

It would be like watching a broadcast television program during prime time.

I guess this has been tried before but also never heard of it

freefaler · 4 years ago
The clients that do plan in advance are more sensitive about the quality of service rather than price. There are QoS SLA contractually required and if they can't be met clients will leave.
eternityforest · 4 years ago
P2P with fallback to centralized would solve a lot of it.

Unfortunately P2P projects always try to be replacements for centralized rather than supplements, and almost none of them aside from BitTorrent are what I'd call great.

dehrmann · 4 years ago
The other part is that CDNs result in less total network utilization. They limit most bulky traffic to the last miles.
freefaler · 4 years ago
It's not always so straightforward. You decrease your traffic costs, but you'll incur capex (servers, switches, connectivity) or operation costs (renting servers). Additionally outside EU network contracts can be a hassle to sign because of market consolidation or outright oligopoly. So you'd need to commit capital to get bandwidth. It's an interesting optimisation problem and sometimes you can route your traffic differently based on SNMP data, price per Mbps and route performance. The higher you pay, the better connectivity you get. But off-peak times you can route traffic via cheaper provider.
blahgeek · 4 years ago
I don't know about other countries, but in China, almost all major video hosting companies (including both PGC and UGC contents) are using P2P technologies to save bandwidth.

Live streaming p2p is the easy one. Live streaming content is very skewed: top 1% streams can cover majority of the bandwidth, the top room can easily have tens of thousands of users, so p2p helps a lot.

Video-on-demand is harder. But now there's also many "seed boxes" out in the market: it's basically a custom home router with a big disk, users buy it and put it in their home as a regular wireless router, but in the background it would automatically connect to server, cache videos and serve the videos to other peers. The user may get some bonus from it (mostly digital points). Essentially, these companies are buying users' home internet as CDN edge nodes.

But in either way, P2P is used to save some bandwidth (cost), but the performance would almost always be worse. There would always be traditional CDN as a fallback.

Some possible reasons why it's more popular in china: 1. there's lots of people here; 2. users don't care about their privacy much

(I worked on this area in one of the largest video hosting company in china)

zinekeller · 4 years ago
I'll add 3. connections tend to be symmetric, and even guaranteed for fixed connections (well technically mobile internet are asymmetric by design, but it doesn't matter much because uplink is still reasonably fast) and 4. government-mandated peerings, before 2017 it'll be simply impossible because Telecom and Unicom (not sure about Mobile) don't have good peering back then with each other.
PaulHoule · 4 years ago
Where I live in the US we have ADSL and to make peer-to-peer viable at all I have to put an upload cap on, otherwise the uplink gets saturated and downloads start having problems because the ACK packets get delayed or lost.

It drives me absolutely up the wall that some torrent networks insist that you have a 1-1 ratio because it is close to impossible for me and there is no point because often the network has so many seeds…. The fact that I can complete a transfer uploading just 5% of what I download proves my upload isn’t needed.

drewvolpe · 4 years ago
That's fascinating. How popular are seed boxes? And does the consumer get them from their internet provider directly?
blahgeek · 4 years ago
There are several companies doing this business specifically: users buy seek boxes from them, and they sell the bandwidth to us video streaming companies. They are just like CDN providers, but instead of buying bandwidth and server from the data center, they “buy” it from home users. I don’t think ISPs would like this, because it affects the business in their data center side.

IIRC the total usable bandwidth is about 10Tbps

warrenm · 4 years ago
> Why aren't we see many video hosting companies adopt peertube or P2P?

Case Study: Bit Torrent [0]

Bit torrent is a brilliant idea - allows everyone who has part(s) of a file to contribute to the pool of availability so that any given central server/mirror isn't overwhelmed

And it has its place (eg in file sharing)

But for streaming? Not so much

Say you're getting the "next chunk" (whatever 'chunksize' is in this context) from me, and I go offline (it's the end of my day, need to reboot for updates, any of myriad reasons). Where does the next bit of the video come from in a way that is seamless to the viewer?

That is the fundamental problem of shared/p2p streaming protocols - everytime the host of the current/next blob o' data goes offline, you need to waste time finding a replacement

Even if the replacement can be found "quickly", how do you ensure they don't go offline in the middle of streaming? How do you ensure "enough" copies of every chunk are distributed that it, effectively, 'doesn't matter' how many go offline [at once], it will still stream?

--------------

[0] https://en.wikipedia.org/wiki/BitTorrent

Sohcahtoa82 · 4 years ago
You get around that with long buffers. Like...several minutes long, rather than several seconds.

You still have a good chance of buffer underruns when you start a video, but that would likely get mitigated by the fact that the data for the start of the video would likely end up highly duplicated on the network.

The bigger issue is seeking. Jumping to the middle of a video could take many seconds before it played. The UX would be awful.

warrenm · 4 years ago
>You get around that with long buffers. Like...several minutes long, rather than several seconds.

That sounds horrible - now you've got to have something buffering [potentially] 100s of MBs (depending on quality) over lousy residential upload speeds?

kenniskrag · 4 years ago
> everytime the host of the current/next blob o' data goes offline, you need to waste time finding a replacement

Ther is an algorithm in place where the rarest is downloaded. Also a priority list of older peers is implemented. Some protocol also request the chunk from muliple peers (kademlia).

paskozdilar · 4 years ago
>Even if the replacement can be found "quickly", how do you ensure they don't go offline in the middle of streaming? How do you ensure "enough" copies of every chunk are distributed that it, effectively, 'doesn't matter' how many go offline [at once], it will still stream?

You don't need to _ensure_ all that stuff, in the strictest sense of the word. Network hiccups happen, nodes go down. As long as the frequency of interruptions scales inversely with the number of nodes in the network, once you're on a big enough network everything will work smoothly.

bick_nyers · 4 years ago
Wouldn't the "next chunk" problem be mitigated greatly by just maintaining larger buffer sizes? I know most client devices (such as Smart TVs and HDMI Dongles) don't have the memory capacity for it, but a 10 second long buffer would effectively mask a lot of those issues.

That being said, most residential internet packages have low upload speeds, which is an issue

giantrobot · 4 years ago
A 1Mbps stream (high quality 480p or potato quality 720p) would need a 10Mbps burst to fill a 10 second buffer or wait some time before playback. It would need somewhere north of 1Mbps to keep the buffer full accounting for dropped packets, jitter, and seeds dropping out. If the stream drops below 1Mbps for longer than ten seconds then playback stops.

It all makes for a shitty user experience.

tsujp · 4 years ago
If I recall correctly the famous 1% rule (not that one) but rather[1] was the topic of a recent article (which I cannot now find but am hoping will randomly come into my memory at 2 AM and/or someone will correct me on) where the author gave some pretty convincing evidence that this kind of behaviour (the tendancy for _most_ people on the internet, 90%, to only consume content) also extends to open source ecosystems. I believe Matrix was given as one example.

Essentially no one wants to host and run the infrastructure for these things, only the "diehards" do. This is akin to how only the "diehards" produce content on Wikipedia, or YouTube. Remember, this is relative to the volume of people who only consume that content.

It's the same in P2P e.g. Bittorrent. The count of seeders (those who have the content and are uploading it to peers who need to download it) is almost always lower than leechers (those who are only downloading).

This is exacerbated with Peertube because video files are big, require a lot of bandwidth, and someone has to pay for that. If there is a set of diehards who love running Matrix or IRC (think other smaller networks not just Libera) or what have you then how many of that already small set can meaningfully afford to run a Peertube instance capable of serving a momentus volume of video traffic?

I wish it wasn't so but it is.

[1]: https://en.wikipedia.org/wiki/1%25_rule_(Internet_culture)

Tenoke · 4 years ago
>It's the same in P2P e.g. Bittorrent. The count of seeders (those who have the content and are uploading it to peers who need to download it) is almost always lower than leechers (those who are only downloading).

That's (at least partially) solved with incentives. Plenty of private torrent sites require you to keep a good ratio of download/upload to use them. As a kid I was definitely not a die-hard but still seeded to have access.

kmeisthax · 4 years ago
Yes, but that model only really works for piracy sites, and only ones with closed access[0], where you can maintain an internal ledger of who seeded what. On an actual streaming service where you have to pay for your TV shows or movies, the cost of video distribution will be dwarfed by the cost to license or produce that video. If you actually tried this on a premium video site you'd be charging $15/mo and then giving most people back a few pennies for their bandwidth.

Free-of-charge video might seem like a better fit, but that market was totally cornered by Google and YouTube; who basically have as much bandwidth as they could ever want. Yes, you can get around that with PeerTube, but there are so many other entirely unrelated advantages YouTube has over everyone else that this doesn't really confer much of a benefit.

[0] Interestingly, the FTP topsite scene would stick their nose up at even private BitTorrent trackers as not closed enough.

tsujp · 4 years ago
Yeah I remember those. One I used to use had a donation scheme where if you couldn't seed or didn't want to you could donate 10 Euros and 100 GiB of seed traffic would be applied to your account (just a number in a database not actual seeding) to keep that ratio up.

These people were _the_ tracker for the content in question so they were seeding a lot of content themselves hence the donations were a proxy.

jasode · 4 years ago
>For me it looks like the hosting provider gets a huge savings in terms of bandwidth.

The constraint on wide adoption is literally the quantity of "peers" in both Peertube and peer-2-peer.

The underlying human incentives are not there for most people to host a peer node. This limitation applies to all types of digital domains including videos, or files (IPFS or bittorrent), or crypto (Bitcoin/Ethereum peer node).

Let's follow the trail of incentives for one Peertube node:

- see list of Peertube instances: https://joinpeertube.org/instances#instances-list

- I pick the 2nd one on the list: https://the.jokertv.eu/about/instance

- in that instance's "About" page, it says: How we will pay for keeping our instance running -- personal funds. If you feel like donating, put up your own instance instead and host some creators you find interesting.

Companies can't build a Youtube/TikTok competitor based on examples like that. Same forces of economic incentives that limits quantity of IPFS peers which means businesses can't use it to replace Cloudflare CDN or AWS S3 buckets.

It should be understandable why most people aren't willing to spend personal funds on hosting home nodes so businesses can freeload off of p2p and save money on bandwidth.

elsjaako · 4 years ago
That list of instances is randomly ordered, it isn't like biggest first or anything.

I don't really see how the fact that there is someone who hosts videos using personal funds proves that it isn't possible for a professional company to host videos using the same company. It proves Peertube does work at small scale, and says "not much" about how it would handle large scale with a real budget.

toxik · 4 years ago
The human incentives are certainly there, it’s the copyright and IP industry that does not want this to come to pass. There was a time when you could find obscure things on torrents with seeders, but that was shut down by the IP rent seeking industry.
acdha · 4 years ago
I think you’re overestimating how strong that inventive is. There’s a lot that I’d change about the current copyright model but it’s not just rent-seeking keeping P2P from mainstream use: most people do not have high uplink capacity and congestion is a real issue. Companies like Netflix have built a consistent user experience where video plays quickly and smoothly, and the price is low enough that most people aren’t jumping for alternatives — I would bet that if you surveyed the average person the biggest copyright reform they’d want is blocking exclusivity so they could use a single service for everything.

From a business perspective, a big problem is trust and safety. Torrents were also notorious for having mislabeled content or malware, and users are not going to use a system which serves arbitrary content from their IPs, so you have challenges keeping the experience good while building a large enough peer network to provide a competitive experience.

lotsofpulp · 4 years ago
And also lack of upload capacity for 90% of people due to the lack of symmetric fiber availability.
CJefferson · 4 years ago
Having previously looked at peertube, the problem is it's hard to make money out of it -- because of the p2p technology, it's hard to have control of your videos, put adverts in, track users, etc.

While peertube saves one big pile of money (the streaming costs), it makes it significantly harder to make any money. Videos are expensive to make, so generally creators want a wide audience, and ways to monetise that audience.

zaik · 4 years ago
If creators moved to PeerTube they probably could still make money with sponsored content and donations like they already do on YouTube.

Of course, because of network effects and YouTube ad revenue there is little incentive to do so except for idealists and people who get banned on YouTube.

kempbellt · 4 years ago
If you can't track your users/views/etc, it's hard to sell sponsors on the idea that your video is worth sponsoring.
kazinator · 4 years ago
Peertube is the opposite of centralized video hosting; it's a federation of nodes.

A video hosting company needs to hold all the videos hostage so they can make money somehow, like via ads. Plus the ability to control content: the videos themselves (e.g. be able to censor or delete them), and other content like user accounts, comments and whatnot.

This is like asking, why doesn't Reddit just operate as a Usenet gateway with a Web UI?

tjansen · 4 years ago
> Peertube is the opposite of centralized video hosting; it's a federation of nodes.

Like a CDN? :)

BeefWellington · 4 years ago
While you're correct that in this way a CDN is similar to peer-federated systems, this is also a low-effort trite response that misses about 70% of the differences between the two.

The main difference being, of course, CDNs provide a delivery and storage service which you pay for, meaning the content remains yours, the circumstances under which it is viewed is under your control, and you're simply paying for them to deliver it for you. With peer-based systems, those two key points are no longer true.

kazinator · 4 years ago
The short answer is that geographical resource balancing isn't federation, even though both involve multiple hosts.
KnobbleMcKnees · 4 years ago
Geogprahical distribution is irrelevant if there is only a single gateway to the content.

Then the benefits of distribution are largely only realized by the owner.

jstummbillig · 4 years ago
> Why aren't we see many video hosting companies adopt peertube or P2P?

For the same reason we don't see many video hosting companies, period: YouTube. It eats the entire space for reasons that are entirely unrelated to the pros and cons of p2p. For hypothetical competitors that means solving their biggest issue – gaining users over YouTube – can also not be done by going p2p (but it might of course still be part of their hypothetical tech stack for other reasons).

toxik · 4 years ago
It’s strange that Youtube is so entrenched that even Russia could not muster its own alternative.
fulafel · 4 years ago
Yep, this is a stronger factor than delivery mechanism questions.