Readit News logoReadit News
PreInternet01 · 2 years ago
The cost of egress traffic is a very good reason for many organizations to not fully migrate to a cloud provider anytime soon. And since, unlike with storage costs, there doesn't seem to be an actual reason (other than: it makes migrating to competitors cost-prohibitive in a subset of cases), that seems kind of... weird?

Small example: an actual company I do some work for is in the business of delivering creative assets to distributors. This results in an egress of around 180TB per month, which is, on average just, around 500Mb/s.

So, this company currently operates 2 racks in commercial data centers, linked via 10Gb/s Ethernet-over-DWDM, with 2x512Mb/s and 1x1Gb/s Internet uplinks per DC. Each rack has 2 generic-OEM servers with ~64 AMD Zen cores, 1/2TB RAM, ~8TB NVMe and ~100TB SAS RAID6 storage per node.

Just the cost-savings over egress on AWS is enough to justify that setup, including the cost of an engineer to keep it all up and running (even though the effort required for that turns out to be minimal).

So, are cloud providers ignoring a significant market here, or is the markup on their current customers lucrative enough?

amluto · 2 years ago
> other than: it makes migrating to competitors cost-prohibitive in a subset of cases

My theory: it forces third party services into the same cloud.

Suppose you use AWS and you want to pay a third party SaaS provider for some service involving moderate-to-large amounts of data. Here’s one of many examples:

https://www.snowflake.com/en/data-cloud/pricing-options/

And look at this remarkable choice: you get to pick AWS, Azure, or GCP! Snowflake is paying a lot of money to host on those clouds, and they’re passing those costs on to customers.

Snowflake is big. They have lots of engineers. They are obviously cloud-agnostic: they already support three clouds. It would surely be much cheaper to operate a physical facility, and they could plausibly offer better performance (because NVMe is amazing), and they could split the cost savings with customers. But they don’t, and my theory is that egress from customers to Snowflake would negate any cost savings, and the variable nature of the costs would scare away customers.

So my theory is that the ways that customers avoid egress fees makes the major clouds a lot of money. IMO regulators should take a very careful look at this, but it’s an excellent business decision on the parts of the clouds.

snapplebobapple · 2 years ago
Its this plus locking customers to one cloud, because egress kills intercloud syncing on any moderately large data set. Any smart customer would have a duplicated setup across clouds if egress cost what it actually cost instead of 100x plus what it actually costs
quickthrower2 · 2 years ago
So an anticompetitive practice.
nyc_data_geek · 2 years ago
It's pretty simple. Excessive egress costs = vendor lock in, and yes, forcing third party services into the same cloud (the walled garden), and limiting customer choice.

Just another reason so many orgs are getting heartburn from going too deep too fast into the cloud.

rfoo · 2 years ago
While I agree with what you said in general, Snowflake is a poor example. Data warehouses like Snowflake really can use the Tbps+ aggregated bandwidth between S3 and EC2 in the same region. There is no way for this to work over the Internet.
ren_engineer · 2 years ago
Snowflake and most other cloud services offer on-prem for these use cases, so it really doesn't make sense for them to roll their own data centers because it would be pretty niche. Cloud works for startups because they are there already, on-prem for enterprise customers with their own hardware
ctrw · 2 years ago
It's not a theory it's literally why two projects I was on used snowflake in aws.
jupp0r · 2 years ago
It's an interesting point, but I doubt it's the lions share of egress that's going to other data centers vs to customers. Fan out is where it gets expensive.
Eridrus · 2 years ago
Snowflake's margins are like 90%+ on top of the compute they sell, and they pass on all these costs including egress directly to customers.
Spooky23 · 2 years ago
That’s definitely a factor. You can even pay AWS for the infrastructure used by SaaS apps to aggregate more volume.
thimp · 2 years ago
I saw a hilarious fuck up a few months ago. Company sets up an AWS hosted always on VPN solution. Connects 1000 staff through it. Celebrates how they saved $50k on the VPN solution. Gets $25k AWS bill for the just the first month of egress traffic. Turns out the data was leaving AWS egress three separate times.
constrain5795 · 2 years ago
AWS VPN base cost is $72/mo + $36/u/mo. So they were spending $80k/mo before?
ailurooo · 2 years ago
how were they spending that 50k previously? a bespoke saas thing or self hosting?
electroly · 2 years ago
If you still want to use some AWS services, you can get an AWS Direct Connect fiber cross connect from your data center rack to AWS, just like you do with your Internet connections. They operate Direct Connect in lots of third party carrier-neutral data centers. AWS egress over Direct Connect is $0.02/GB instead of $0.09/GB over the public Internet. You can serve customers through your unmetered Internet connections while accessing S3 (or whatever) via Direct Connect on the backend.
amluto · 2 years ago
I can pay overpriced cross-connect rates in giant name brand datacenters, with or without terminating one end at Direct Connect. (AFAICT the $1600/mo or so for 10Gbps doesn’t actually cover the cost of the cross-connect.)

But that extra $65k/mo to fully utilize the link is utterly and completely nuts. My mind boggles when someone calls that a good deal. I can buy and fully depreciate the equipment needed to utilize that in a couple of days. (Well, I can’t actually buy equipment on the AWS end, but I can _rent_ it, from AWS, for a lot less money than $64k/mo.)

And I don’t believe at all that it costs AWS anything close to this much to operate the thing. I can, after all, send 10Gbps between two EC2 instances (or between S3 and EC2) for a lot less money.

That $65k is simply a punitive charge IMO. AWS wants me to avoid paying it by not doing this or, in cases where I can’t avoid it (e.g. the thing is actually a link to my office or a factory or whatever) to collect as much money as they can without driving me off AWS entirely.

ricktdotorg · 2 years ago
i'm a customer of the GCP equivalent: partner interconnect. our DC is in an equinix facility, they wire up drops for us that layer 3 straight into GCP. unmetered 1Gbps for about 250 bucks a month per (paid to EQX not GCP). are AWS really charging you per Gb for data egress from AWS into your own DC over an AWS direct connect??
asmor · 2 years ago
It really depends on the quality of the peering you expect. It doesn't matter, until it does. Consumer ISPs sometimes do their utmost to not peer with open exchanges, and the entire thing gets even more complex when you go to places where bandwidth is more expensive (i.e. Oceania).

There's a reason the favorite chart to exemplify value Cloudflare reps like to show is Argo Smart Routing, and why it costs about $100 per TB just like AWS and GCP.

Fripplebubby · 2 years ago
I agree, and I would also put forward that most people don't understand what peering is or how it works. When people (usually developers who are not network engineers and have not worked at that level of the stack) talk about "egress", they mean delivering bits from your network (cloud or otherwise) to any other network on the internet. How can you put just one price on delivering a bit either to a host within the same datacenter or one on the opposite side of the planet? Physics still mean that one is more expensive than the other.

The existence of the world wide web has tricked us into thinking that sending traffic anywhere is/should be the same, but of course it is not. So while the price you (a cloud customer) pay for egress pricing is (often) indiscriminate on where that traffic is going, using common sense, we can understand that some traffic is more expensive than others, and the price we pay is a blended price with that aspect "baked in" or "priced in".

naiv · 2 years ago
We are using Argo for our xhr search traffic as it makes more sense than setting up different servers/vms in parts of the world. Each request is only 1kb max.

But I would not use it for static assets. For this we use Bunny edge storage to provide faster response times at very reasonable prices.

ozr · 2 years ago
Definitely lucrative enough. The use case you've described isn't particularly uncommon, but lots of companies just pay for the egress.

The problem is that there are now multiple generations of software engineers that do not know how bandwidth is priced. They've only used managed providers that charge per unit of ingress/egress, at some fractional dollar per GB.

api · 2 years ago
I’ve had people refuse to believe that bandwidth is actually very cheap and cloud markup is insane (hundreds or even thousands of times cost).

I show them bare metal providers and colo that bills by size of pipe rather than transfer. They refuse to believe it or assume there must be a catch. There usually isn’t, though sometimes the very cheapest skimp on things like rich peering and can be slightly slower or less reliable. But still cheapest is relative here. Expensive bare metal or colo bandwidth is still usually hundreds of times less than big three cloud egress.

It’s just nuts.

It’s a subset of a wider problem of multiple generations of developers being fully brainwashed by “cloud native” in lots of ways. What an amazing racket this all has been for providers…

toast0 · 2 years ago
My feeling is that egress is easily measured, so it's where costs that are hard to assess get moved to.

It doesn't feel great to be line item billed for stuff at 10x the rate of credible other offers.

I think there is also some geo-specific pricing that gets hidden in a global price; bandwidth can be a lot more expensive in some locations than others and if you are charged 5x for egress in south america, nobody will use the south america locations and that's not good for business.

raid2000 · 2 years ago
Right. Egress is an imperfect, but reasonable metric for overall utilization. If they started charging for CPU hertz above a certain threshold, that'd be a harder sell.
amluto · 2 years ago
I don’t believe this. Operating an internal cloud network is expensive, but it’s expensive because of internal traffic, and they don’t charge for that internal traffic. Egress is just like traffic to any other system, and AWS doesn’t charge for that.

Also:

> It doesn't feel great to be line item billed for stuff at 10x the rate of credible other offers.

It’s quite a bit worse than 10x

cocothem · 2 years ago
Are you a Google Cloud customer looking to exit Google Cloud? If so, you are eligible for free data transfer when you migrate all of your Google Cloud workloads and data from Google Cloud to another cloud provider or an on-premises data center.

https://cloud.google.com/exit-cloud

andix · 2 years ago
180 TB is still a small customer in the scale of big cloud providers, so they probably just don't care. If the customers are willing to pay the price they are taking their money, if they go to one of the smaller providers they are fine with it too.

Also it could be possible to set up some hybrid solution and offload the egress heavy assets serving to another provider and only run the "brains" inside AWS/etc.

mgaunard · 2 years ago
There is no scenario in which cloud is cheaper than bare metal to operate.

What's cheaper with cloud is that you don't need upfront costs, but here, you already have the whole infrastructure already there, up and running.

Why would you even consider getting rid of it and replacing it with AWS, makes no sense.

kiitos · 2 years ago
It's weird that AWS is just absurdly profitable, then, isn't it?

(You're wrong, there are many such scenarios)

hatsix · 2 years ago
Our 20tb/m costs us $414... if you're saying that you can amortize those servers, rent the space and pay for upkeep for less than $4k a month?

We have another service that has 20x the bandwidth, but it's a legacy GAE that has super cheap premium egress... But I'm told that AWS says that their pre-pay discount will be competitive in about a year, the rate that we're growing.

The negotiated egress prices are much lower, so long as you are buying future egress. If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal.

shiftpgdn · 2 years ago
I could buy a whole off lease fully deprcriated server and a year of colo with a gig link for under $4,000, yes. I would almost certainly see better performance than any cloud provider can give me too.
re-thc · 2 years ago
> If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal.

Shouldn't it be the other way around? Why are they giving you are discount if you're not likely to jump?

pinkgolem · 2 years ago
>If they're not worried about you jumping ship (you use a lot of their aws-native services), you can get a great deal.

Aka as long as you are not using the cloud as cloud the cloud can be cheap?

>rent the space and pay for upkeep for less than $4k a month?

I mean, you can just rent servers somewhere else.. or do a colocation rack. Also I would strongly suggest going with the first one

jonatron · 2 years ago
The AWS Enterprise Discount Program apparently requires $1M per year spend. 180TB is about $13k on AWS so presumably not enough to be interesting to them. Hopefully someone who works at AWS can share some info.
tky · 2 years ago
EDP can be great if you can meet their required year over year growth requirement and if your spend is high enough to make the discounts (which are variable and negotiated often at a product offering level, not blanket) offset the required top-shelf support contract. For smaller orgs even at the $1-2M/mo level, it can often be a risk not worth taking vs other cost savings mechanisms and locating high-egress use cases elsewhere.

Egress bandwidth pricing has been the sacred cow money firehose forever, despite transit costs continuing to shrink.

vidarh · 2 years ago
Yeah, I've built several setups where total cost of ownership was lower than egress cost alone would have been on AWS. Both w/physical hardware in colos and with rented managed servers at providers like OVH and Hetzner.
belter · 2 years ago
Context matters here. How critical is that workload? What the economic and reputational impact for the company, if one of the physical connections or some technical problem with the data center causes a downtime of hours or days?
pinkgolem · 2 years ago
looking at the uptime from aws and from most big outtage notices i have read in the last few years, there does not seem to be a benefit in regards to reliability when using cloud.

see reddit, see amazon/aws outages taking with them netflix/disney plus etc

honestly its a lot better to keep your architecture cloud agnostics and test restores regulary on a different provider/region

also: store your backups somewhere else, not on your primary cloud provider, not connected to your auth system, not overwritable from anywhere

intelVISA · 2 years ago
Between the vendor lock-in and egregious costs it's a sad state for so many in our profession to call this 'the future'.
kansi · 2 years ago
Are you able to share which provider do they use for their rack setup?
xyst · 2 years ago
just 1 engineer? lol. Let's hope it's not business critical for the servers to be up and running at all times if you only need 1 engineer.
pinkgolem · 2 years ago
you need an oncall team be it cloud or not, taking one fte position and dedicating it to managing 2 servers(documentation/updates/backup & restore procedures testing) seems rather.. high
rsync · 2 years ago
Not every cloud provider has egress charges…

Deleted Comment

Scubabear68 · 2 years ago
The article says “ Cloud providers charge for egress because it costs them money to send data out of their network. They have to pay for the infrastructure and bandwidth required to send data to users”.

The charge is not based on cost in the case of the big names. They charge an arm and a leg because they want to keep you and your data on their platform. When you move it you are breaking free.

Hence the high costs to deter this behavior.

ehhthing · 2 years ago
> They charge an arm and a leg because they want to keep you and your data on their platform. When you move it you are breaking free.

This isn't remotely true.

The bandwidth alliance exists, and a lot of cloud companies are on the list: https://www.cloudflare.com/en-gb/bandwidth-alliance/

The actual answer is much more complicated. For example, Google Cloud offers two different bandwidth tiers: premium and standard. The calculation on the OP assumes premium since that's the default option, but obviously it's much more expensive.

Google cloud's "premium" bandwidth is much akin to AWS Global Accelerator since it utilizes Google's own backbone network for as long as possible before exiting at the nearest peering point between Google and whatever ISP your end user is at. AWS Global Accelerator has some other options available, that make it fundamentally a different product, but the routing characteristics are much more similar to GCP Premium bandwidth than anything else AWS offers.

Scubabear68 · 2 years ago
That’s an impressive word salad, but sorry, no. Egress costs are high strictly to keep you there.

This is why egress is dirt cheap on other platforms outside the big 3 cloud providers.

This is also why ingress is free.

teraflop · 2 years ago
Google's "standard" bandwidth pricing is about 15%-45% cheaper than "premium", which is admittedly a significant discount, but it's still an order of magnitude more expensive than some of the other options on the list.
foofie · 2 years ago
> This isn't remotely true.

Nothing in your comment rejects or disproves the claim that egress costs are vendor lockdown.

Your link to the bandwidth alliance explicitly states that their justification for network costs is unloading infrastructure costs onto end users as data fees. That's their only and best justification. This is clearly a business decision that has no bearing in operational costs.

Some cloud providers charge nothing, others only start charging after hitting a high threshold from a single instance. Do they not operate infrastructure?

It's their business, it's their business model. Some restaurants charge you for a glass of tap water too. Let's not pretend they do it because of infrastructure costs.

acdha · 2 years ago
The bandwidth alliance exists to try to cut into AWS’ business. They could always have unilaterally cut rates closer to their cost but that margin was appealing, until they realized that they were never going to catch up with AWS without being cheaper.
toast0 · 2 years ago
> For example, Google Cloud offers two different bandwidth tiers: premium and standard. The calculation on the OP assumes premium since that's the default option, but obviously it's much more expensive.

Of course, non-premium tier is v4 only, and only available at some locations.

dilyevsky · 2 years ago
It also forces downstream vendors that ingest a lot of data (e.g datadog, snowflake) to maintain cloud presence because their customers will not to want to pay egress fees
rambambram · 2 years ago
Let's say you pay a lot for some cloud provider and you find out that your app is best and cheapest served by a good ol' regular webhosting provider. Please, in that case, make sure you leave your money and data at independent webhosts. There's a trend going on - at least in Europe - of consolidating smaller webhosts into big players, under the guidance of some big groups (private equity?). They quickly raise their prices by 100's of percents.

I experienced this a couple of times with different webhosts (in the Netherlands) over the last three to four years. Very recently my monthly bill at one of them went up from around 3,- to around 18,-.

Whenever I read about 'egress costs' I usually laugh about that because I don't even understand it; my data is my data, why pay to get it off your hard drive!? But for some time I'm also seeing dark clouds for my own use cases. I don't find it hard to believe anymore that webhost after webhost is consolidating, trying to play AWS, upping the prices a lot, and finally... also establishing egress costs!??

skrebbel · 2 years ago
Yeah wow huh. I had my wife’s food blog (wordpress) hosted at a Dutch provider called Neostrada. I think it was something like 50 euros per year at the start. They got bought, sent around emails with sleazy wording like “we’re upgrading your plan to fit our new pricing scheme” and then within ~2 years I ended up owing them 450 euros a year. I’ve never cancelled anything this quickly before.

A 9x price hike in just a few years has got to be the least scrupulous PE move I’ve ever witnessed in NL. I agree with your prediction that it’s just a matter of time before they tack on exorbitant egress fees.

Fripplebubby · 2 years ago
If you believe that egress prices are fake and pure profit, I encourage you (or someone else reading this with the same belief) to try running your own independent web hosting service ;)

I think what you will discover is that network bandwidth is a finite resource you have to pay for and ration out to your customers, and that the easiest way to finitize and ration something your customers expect (errantly) to be infinite is to charge for it, and charge exactly that price that will cap demand!

8organicbits · 2 years ago
One trick to watch out for: a cloud provider offering inexpensive egress can still make high-egress expensive by controlling the network speed of their instances. Several vendors require you to scale your whole instance if you saturate any of the components (like needing to upgrade from the 2 CPU to the 4 CPU instance to increase the network throughput, even though you don't need more CPU). On paper egress looks cheap and the instances look well priced for the hardware you get, but if you workload doesn't need balanced a balanced instance (1 CPU / 1 GB RAM / etc.) the pricing can get wonky.
foofie · 2 years ago
> Several vendors require you to scale your whole instance if you saturate any of the components (...)

Interesting. Which cloud providers do you have in mind?

PuffinBlue · 2 years ago
The GP might be referring to how most of the non-hyperscalers like Linode or DigitalOcean or Vultr tie total monthly egress to instance size. IIRC they do allow overage charges at $x per GB but it quickly becomes more expensive to do it that way.
Mortiffer · 2 years ago
Like scaleway
njitbew · 2 years ago
Oracle Cloud only charges a fraction of want Google, Microsoft, and Amazon charge. Any idea how Oracle is able to keep the cost so low? Or are the others just inflating the price so customers don’t move to the competitor? In that case Oracle deserves a shout out for not applying these vendor lock-in practices.
andersa · 2 years ago
The large providers are overcharging between 10 and 100x on egress. Cloudflare has a blog on it somewhere.
wongarsu · 2 years ago
Hetzner charges $1/TB. Oracle charges $7/TB, AWS $90/TB.

Oracle has probably really good margins on egress costs. With AWS/GCP/Azure the costs are absurd because for a lot of their customers it's not a big cost during operation, but makes moving data off cost prohibitive. It's simply a vendor lock-in mechanism for them.

dilyevsky · 2 years ago
The margin on egress is insane. Oracle got into cloud late in the game and burned a lot of goodwill downmarket so they have to sacrifice that to play catchup
pier25 · 2 years ago
Egress is an artificial cost. Cloud providers don't pay for it.
Culonavirus · 2 years ago
Well I'm sure there are some costs, but Google charging you an arm and a leg for traffic when they literally own multiple sea cables going around the world and a bazillion datacenters seems a bit sus...
suyash · 2 years ago
Other are just charging what they are because people are too ignorant to complain/object and will just pay.
vlovich123 · 2 years ago
Egress prices have insane levels of markup.
shell_game · 2 years ago
Oracle has other lock in practices lol… they don’t need to jack up egress fees.
hacker_newz · 2 years ago
Oracle does not deserve a shout out for charging market rate for their service.
mocamoca · 2 years ago
OVHCloud is free https://us.ovhcloud.com/public-cloud/faq/

As a data company we really benefit from Scaleway, Hetzner and Ovh.

However, Scale way has no plans to add an US infra

And I don't know if Hetzner has US présence?

reaperman · 2 years ago
Hetzner has a limited US presence for VM products but not baremetal/dedicated solutions.
raid2000 · 2 years ago
European providers benefit from lower cross-connect fees in datacenters and more internet exchanges for easy peering. It's not surprising they offer more bandwidth at the same cost.
toast0 · 2 years ago
The US has plenty of internet exchanges. And plenty of smaller hosters with cheap bandwidth. Usually located near an internet exchange.

It's not like AWS or GCP have locations that are terribly far from major exchanges either.

msravi · 2 years ago
Hetzner with its ARM offering of 2xvCPU, 4GB RAM, 40GB Storage, 20TB egress at EUR 3.79/mo is unbeatable value for a small-medium website.
severino · 2 years ago
The only downside is that Hetzner does not allow you to set a limit to control the egress traffic expenses, does it? So that you can be protected against the so called "denial-of-wallet" attacks.
msravi · 2 years ago
The "usage" tab allows you to set a euro value limit on each project for an email warning. I just use that. It isn't specific to egress, but total accumulated usage for the project.
geor9e · 2 years ago
I had no idea it was this crazy expensive. I can host a 1 TB folder of movies on my Google Fiber internet Synology NAS website, and let my friend download it for free. But if I hosted my website on Google Cloud, they bill me $111.60? How are these cloud services getting away with this pricing?
dehrmann · 2 years ago
> let my friend download it for free

Except you paid the cost for the fiber connection. You also get one more 9 of reliability from their DC.

hacker_newz · 2 years ago
Your Google Fiber IP is part of a residential network with a completely different service agreement and larger limitations than Google Cloud.
tootie · 2 years ago
Residential usually has some kind of hard data cap. Like if you transfer >10TB they'll just cut you off. You're also generally barred from running any kind of commercial application.
metadat · 2 years ago
The data cap thing is generally only with comcast and other tacky low-quality hall-of-mirrors bargain-bin types of cable ISPs that don't sell proper Internet connections (maybe Cocks Cable, too?).

For example, AT&T fiber has no bandwidth caps and offers symmetric 1-10 gigabit service in an ever expanding number of areas throughout the US. I pay $55/mo for 1gbps service, and have done more than 100TB/mo for many months and it's fine.

alberto180 · 2 years ago
Maybe in the US. Not here in Europe
redox99 · 2 years ago
They figure most people are hosting websites which are only a couple of MB, so they need to charge this much to be able to extract egress money from them.
jeffbee · 2 years ago
Your little NAS is not directly connected to hundreds of metro areas around the globe. Even on Google Fiber, which is an excellent network, you only have a few dozen points of exchange, all in America. As a Google Cloud customer your egress traffic will be on Google's network all the way to the very edges of the world, at hundreds of interconnect points and exchanges. Your traffic will be on private networks until it is with a few miles of your end user, in all likelihood. This is a comparison between apples and diamonds.
veeti · 2 years ago
Except they will charge you the same extortionate egress fees for outbound traffic from a single compute instance ("an apple"), no global content delivery network involved ("a diamond").