How many would say this normally? I.e., what's the baseline?
Meanwhile, how many are giving their teams the resources (i.e., people) necessary to do the analytics on the bill required to find & cut costs? Every cloud I've worked with has been terrible at billing in a way that could be reasonable investigated. At one point we literally pulled all the billing data into a SQLite database. (And that was a great idea, as it made queries against it much faster.)
Further, most higher ups seem to have a sense of "it won't be invented here", and so a lot of stuff ends up in vendor land; need more people then to pull those separate bills into one spot, and start attributing costs against the revenue they might help generate.
I've also worked a fair number of places where people were cagey about telling eng how much they're spending where. If you want your IT dept. to cut costs, they have to know where the money is going.
And then there's the yak shaving of it all: we've had a support ticket open w/ Azure for 9 days now asking "where did this $1k go?". Our Blob Storage "Cool" tier costs upticked, and I'm 99% sure the bill there is just wrong. Cool storage should be (price * amount stored) and the "how much are we storing" graph is not proportional to the cost graph. The only gotcha is an early delete penalty, and we are also pretty confident we're not hitting that. (We know what's being deleted when, and none of it should be subject to it.) But how does it take a over week to answer the question?
AWS in particular is handsomely engineered so you can't answer this sort of question, as well as making people spend more without need.
- Per resource cost is turned off by default. Turning it on costs money. How much? ¯\_(ツ)_/¯
- Services like S3 will bill you per API call. How do you know who is calling your APIs? Using Cloud trail, which is turned off by default. Turning it on for this sort of analysis might cost a lot $$$ for a reasonably used bucket.
- CloudWatch will bill you per custom metric and adds up quickly. Why do I have so many custom metrics? Good luck finding out.
- Many costs are tied with stuff that's really hard to control e.g. egress traffic.
- Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc...
- Many services are ha, multi az only. That makes AWS unnecessarily expensive for dev environments / non critical services.
The list goes on. I know a fair number of consulting companies making the money of their lives reducing AWS bills.
> - Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc...
And you can have business support only on one account... obviously you can go Enterprise and have the same support level on everything but pay a lot more $$$
Million dollar question - at which point it gets easier to learn how to host things on your own vs keeping up with this cloud mumbo-jumbo bull type of knowledge?
>- Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc...
As an anecdote on the knowing where money is going front:
One of the most effective ways I've found to drive change in how product management and execs treat engineering was getting average salary numbers by type of position for my team out of HR, and attributing costs toward every major feature request + meetings in a weekly report.
My manager at the time, the CEO of a VC funded startup, did not see the point. We were after all already paying for the engineers, and they were always fully loaded. Until he'd seen a couple of reports, and saw what it told us about prioritisation.
Our project and product managers got a lot less meeting-happy after seeing one regular meeting they were very fond of cost an estimated 10k/month due to the number of people attending, and they got a lot more thoughtful about feature request and how to prioritize after seeing the real costs of supposedly "trivial" requests that had previously been slipped past normal processes.
Same applies on the cloud front, yet a lot of what has driven the adoption of cloud is that it often helps engineering escape cost analysis and efforts to optimize costs.
And so you get organisations which often have no idea which decisions contribute what to those costs.
Hey, we've had our blob storage costs spike up +70% for two days (Oct 4th and Oct 5th), across almost all of our storage accounts, regions and hot/cold/archive tiers. The spike is a consistent ~70% for each of the affected accounts, and this includes accounts where there have been no writes for months (so why is there an increase), as well as accounts where we never delete (so while an increase would be surprising, the decrease on Oct 6th is impossible). Is this similar to what you're seeing ?
I was considering writing to Azure support about that...
My last billing issue with them took ~120 days to resolve, so I don't have much hope.
Incredible. You've nailed both the amount (the +70%)¹ and the days (yep, Oct 4th & 5th).
You are also right that it is across storage accounts, and access tiers: I've now looked at some of our other SAs, including ones in the Hot tier, and yep, they see the same bizarre uptick. Given that Hot tier is experiencing it, that rules out the possibility of "human error" w.r.t. to it not being a Cool-tier early deletion.
> Is this similar to what you're seeing ?
You've nailed it precisely; it sounds exactly like what we're seeing.
(What a small world HN is sometimes!)
¹the numbers on our end wiggle a bit, so I'm not sure about how precise I can estimate the uptick. But 70% is as close as any guess.
I manage one midsized company's set of cloud resources that I'm pretty sure could be reduced by 40% in cost or more if they were willing to take a 3 year advanced reserve. The problem is I can't get a handle on how much their infrastructure will or won't need to scale in the next three years, since they themselves don't know. There's always some new deal around the corner that would require scaling up. On top of that, I'm not sure what we have now will even be sufficient for the existing size of the business a year from now, as much depends on new features in the pipeline and how they end up being used. The result is they end up overspending by probably $3k a year, but the upshot is the flexibility. As far as getting completely off the cloud, I migrated them away from dedicated boxes years ago and I don't miss dealing with maintaining those. The extra time for that alone was worth at least $3k a year. Even with a smaller dedicated data center, there are still issues with getting tickets responded to promptly. And there's way less in the way of failover unless you build it yourself.
So there are three choices, essentially to argue over:
- run on demand instances (maybe mix in spot)
- run fargate instances (maybe mix in spot)
- reserve them.
Every company hates #3, but it's usually the best choice if you can't go spot - you can swap them within the family for size, use them for baseline, it's fine. You're incorporated right? So if you fold, it's just another debt on the pile, stop worrying. Go no upfront if you really want to avoid the cash outlay, or go spot if you can defend limited outages.
In my experience, everyone should be buying RIs or a Savings Plan, right as soon as they start using instances. At the very least it's an incentive to plan to move to cheaper tech after 3 years. But it also takes at least 3 years for businesses to get serious about trying to save on their spend. And you can change instances or types during that time. Most people don't end up changing anything, though, and wish they had been saving.
But how does it take a over week to answer the question?
In my experience it takes azure support over a week to answer any question. Unless the question happens to be which person without a substantial update is taking over the case. You’ll get lots and lots of those answers.
Azure forced me onto invoice billing against my will - and every month without fail they don't automatically match my payment and I have to endure increasingly threatening emails that my account is pending termination while their support takes days to reply to my tickets about it, and when they do reply the advice is usually just 'wait a little longer'.
I've moved critical services off Azure because the risk of wrongful termination is just too high.
If it's any consolation it's not much better on the AWS side. We are currently waiting for them to fix something which they recently broke but some dude is on holiday. I suspect he's the only dude who knows how it works. Even our account managers can't seem to brow beat anyone into fixing it.
Someone else's computer has been painful and very expensive for our organisation so far. Escalating costs aren't even the worst problem.
I've only had one experience with azure support, Back when i was first getting into cloud/playing around with servers, I had forgotten to shut down the VPC i rented there, and I ended up with a bill of ~100$, which is a decent chunk for a student.
Wrote to support, explained my situation, and they removed the bill without any hassle.
> we've had a support ticket open w/ Azure for 9 days now asking "where did this $1k go?". Our Blob Storage "Cool" tier costs upticked, and I'm 99% sure the bill there is just wrong.
Check your partial / interrupted multi-part uploads. If its like amazon the uncommitted blobs won't show up in storage graphs but you'll be getting charged for them just sitting there.
Not sure how you did it but I've worked for a company that enabled you to tag resources with operating hours and stuff got shut down outside of the those operating hours - pretty handy way to cut down on usage because most non-live environments don't need everything running 24x7.
> Every cloud I've worked with has been terrible at billing in a way that could be reasonable investigated.
Terrible for who? You or the cloud provider? I'm a bit of a worrier (and not a perfect coder), so infinite scalability with utility billing combined with a mistake on my part could result in an epic bill. All I'll say is that I won't practice cloud development on my own credit card.
But how does it take a over week to answer the question?
Sounds like the more basic problem is that we keep ending up working in companies where the "higher ups" are constantly shoving decisions down our throats.
Which, ultimately, is probably the root cause of most of the caginess, the yak shaving, the compunction to use cloud databases and cloud-everything, and the silly support tickets that stay open forever and ever and ever.
The silly support tickets that stay open forever and ever and ever is just a company being cheap.
If you don't fund, hire, and staff your support org with good people at multiple tiers, don't be surprised when you get crap. There just aren't enough good minds to service the incoming workload.
Oh, and for god's sake have a decent "to engineering" escalation path for the support folks.
Unfortunately, the support function is almost always a cost center and beholden to only, if anything, response time related SLAs. Instead of customer satisfaction, etc.
An engineer should be as aware of their $ costs as they are their CPU load, filesystem I/O, or memory usage. An engineer should know how many instances they scaled to, and when they're going to scale down.
In fact, the $ cost should be calculable (to a reasonable indicative amount by napkin math - precision not necessary) according to the factors the engineer knows.
Not being aware of the costs feels irresponsible on the part of the engineer. There was a time when we had finite capacity (a single server) and perhaps the engineer had to care less about the cost then (but perhaps they should've cared about utilisation by some measure even then)... but now we're in an age of your application being able to scale beyond reasonable bounds of your companies purse strings, a solid understanding of what your application costs is an essential and core skill. If this is not the game you wanted to play, stay on a VPS or VM.
Anyone possessing a solid understanding will also be looking to reduce or halt their spending... hence, I am surprised the number in the headline is not higher, even ignoring the macro-economic picture.
I have been in system administration for over 20 years... Cloud or OnPrem I have never once has a dev ask about the cost to deploy their new thing, software, etc
Never once has a dev asked about storage costs, of the differences in storing a file in a filesystem vs a blob in a database,
Never once ask about mem optimization could reduce infrastructure costs
This also means engineers need to be given the means to do so. From my experience, most engineers are not given the very basics to change things, observe important metrics and metric changes, and take the time to formulate a strategy.
In theory most places are running with slack. In practice they run under capacity, and engineers aren't going to worry about things out of their control.
The good thing about spending $20k a month on cloud is they will happily take your money, and make it easy to do so, whereas spending the same on a dev means hiring them first which is a bit of a hassle.
It’s a matter of priority. During periods of revenue or user growth, reducing infrastructure cost may be a premature optimization. Now may be a time to focus on operational and logistical inefficiencies
Whilst I high level agree, I also high level disagree.
Even during hyper growth you should be aware of your costs, because you should have an idea of your margin.
I'm not sure any engineer that has awareness of costs, CPU, memory, storage, network, etc... is not trying to reduce some of it - even during hyper growth.
The problem is that if your business is dependent on the cloud, that's not actually your choice: The providers can, and will, change the prices whenever they want. As the crunch comes and even the FAANGs start layoffs, watch how fast buying an in-house VMware stack seems like an absolute bargain, including paying staff to build and maintain it.
> The problem is that if your business is dependent on the cloud, that's not actually your choice: The providers can, and will, change the prices whenever they want
People often say this, but it's just not true. First, any company with significant spend has custom deals with custom pricing in exchange for a commit. Second, even if you don't have that, there's often reserved/"savings plans"/whatever where you commit in exchange for a lower price.
Third, the only time any of the big 3 cloud providers has increased pricing (it usually only goes down) is when GCP increased bandwidth and storage costs a few months ago. AWS, which is what the majority of the market uses, has only decreased prices, hundreds of times.
Regarding VMware, you know you have to pay for licenses and support in an ongoing manner and they are known to jack up prices, right? Furthermore, VMware's software is a dumpster fire that gets you around 5% of what you get from a serious public cloud. Maybe 30%-50% if you go in with NSX and vSAN and the vSuite where you lock in yourself to terrible software with the wrong abstractions, lack of APIs, at enormous inflexible costs. How is that any better?
If you have containerized loads, you're gonna need something more than just vanilla VMWare. Right now your choices are Tanzu ($$$), Rancher, RedHat Openshift on prem ($$$) or Ubuntu...
Probably your savings will come from beign able to get bigger machines.
After 4 years in the cloud I am still surprised to see how performant physical servers are.
I don't necessarily disagree. I've always felt that there was a lot to be said for not becoming dependent on the latest and greatest features of any specific cloud. If you're pushing the limits of some service you probably want to own it.
But my real gripe is that this survey is being presented as doom and gloom when you could equally summarize the results as "about 60% of IT leaders directed to stay the course or beef up cloud spending"
1. Broadcom, if history is any indication, is about to make VMware much less of a bargain, likely exceeding the cost of the cloud, if not matching it.
2. Often time when a company moves to a cloud model, they lay off the traditional administration staff in favor of "DevOps", many many devops engineers would not have the first clue how to manage the low level infrastructure of Storage SAN, FiberChannel, vLAN's etc... let alone managing vSphere or hyperv, etc..
3. Often times when a company moves to a cloud model they decommission their Onprem Hardware, Lead times right now are crazy still so any org wanted to move to OnPrem quickly is going to be hard pressed to do so.
I know we have had cloud for about a decade, but for a while I wondered if we would eventually see a sort of bait-and-switch: Cloud providers raising prices long after most corporations have lost the in-house expertise to run a data center.
Granted, I have no idea how the costs compare. In every company I have been at, the higher-ups tell us in tech we have to "learn about the business side of things", yet the business side never wants to share information.
The graphic doesn't even have a category for being directed to halt spend and less than half are being directed to reduce it in any way so I'm gonna say I still improved it ;)
"further 29% noting they’ve switched public cloud providers in the first half of 2022 due to high prices"
Wait, what? There's no way 29% of companies noped their way out of cloud provider lock-in. Not saying it doesn't happen, but 29% is a huge number.
Unless of course they're startups with no idea how they want their infrastructure run or legacy companies dipping their feet (?) in the cloud for the first time.
More and more companies (newer ones of course, those unburned by legacy) use public clouds as "managed Kubernetes and maybe a few other things like a database, load balancer, object storage". That way there is close to no lock-in, so the biggest challenge in switching clouds would be how to migrate the data and do the failover with minimal downtime.
It may be easier, but still take a lot of work to migrate. Perhaps k8s + object store + LB is easier for the developer, who has a few AWS APIs to replace, but is otherwise not affected. But talk to the person who has to rewrite 5k+ lines of terraform, figure out users, roles & policies in a new IAM, aunderstands the differences between EKS and GKE (what's the Google equivalent for IRSA?). None of it is rocket science, but it's time consuming. And while Google can migrate data from external databases (including AWS Aurora) it's still a big squeaky-bum moment.
I'd guess that does not mean that 29% moved all of their services to the competition. They might just buy a tiny bit from the competition compared to earlier just buying everything from their (expensive) main provider for convenience.
Our only lock-in was using the GCS api instead of S3. I spent a week porting it to S3 and now there's nothing binding us to GCP. There's a whole bunch of S3 compatible storage solutions, we went with Backblaze's B2 as a secondary, if it proves stable and performant we'll switch over and find another cheap provider for our backup.
We're a rather simple company though, less than a dozen services, we use Postgres and Redis for storage, everything deployed on Kubernetes.
A lot of companies run vanila VMs 24/7.
Switching providers is quite easy - just add some networking and a migration tool.
The key here is just the DNS or the AD.
The solution never tried: share the spoils of efficiency with your teams. Give them a percentage or even a few basis points of what they save back as charitable contributions to an organization of their choice.
>The solution never tried: share the spoils of efficiency with your teams.
Funny enough, for all the problems with studying business administration, they do teach this as part of the principal-agent problem. If the principal wants something done, and the agent has no incentive to do it, you need to give the agent some of the incentive so their interests are aligned with yours.
At the same time it's something you rarely see implemented by BA types, because they don't like to share.
Rather they than just "they don't like to share", the BA types don't do this because in one of their other classes they have been taught about the downsides of such incentives, like:
- Corrosive effects on team cohesion, when separate ICs compete to see who can find cost savings first.
- The "cobra effect", where engineers will deliberately introduce cost inefficiencies so that they can fix their own bugs later.
- (Related to the previous point) Prevention of cost overruns is even cheaper than fixing leaks after the fact, but is very hard to properly incentivize.
Any class on principal-agent problems will hopefully include a chapter warning against asymmetric incentives and how they will be ruthlessly exploited by the agents. Engineers are not some exception here, in fact because of their training they are often extra good at spotting loopholes in the rules.
Incentives are one of those things everyone knows we need, but nobody can design safely. A bit like security, we often come across systems that can be gamed due to unforeseen issues with the design.
Google "Hanoi rats" for an incentive blowup. Then there's myriad other incentive issues related to taxes and subsidies, all stuff that had the good intention of promoting good things and reducing bad things.
Sometimes its a meta-game: if a team sees a "share the spoils" scheme being trialled with another team, maybe they shouldn't fix their issues until there's a scheme in place for their team? What about the shape of the payoff? If you fix two issues in the same period, are you better off or worse off than splitting them over a boundary?
Even in sports, there's issues: there's an incentive payment to break the world record. Say you can pole vault 10cm higher than the previous record in your practices. But why break it once when you can break it 10 times? I read about this some time ago, should be findable online.
You’re making a new problem by creating that incentive. What happens when engineers are financially incentivised to cut corners? Do you need to tax their incentive pay for service disruptions? How complicated would the system have to get before the ICs realise that resource allocation is actually a governance responsibility, and they’ve actually just been tasked with doing the CEOs job?
As a SW Engineer I have to say that this would not really work without some healthy dose of abuse.
Mangers at one of my previous workplaces were very vocal with "everything is running with slacks" so whenever I gave them an estimate on how long a feature would take to implement or a bug to be fixed (bugfix estimates are BS to begin with but...) they took it and multiplied it by 0.75 at most. Then bitched about it when it was over the time they assigned. Guess what happened.
Exactly this would happen with a cost-cutting incentive as well - if managers told us "reduce cost by 30% to the end of the year" then the cost would creep up to begin with. Incompetent management making stupid rules and stupid incentives is what drives me nuts :) .
The devil really is in the detail: "you need to give the agent some of the incentive so their interests are aligned with yours". The vast majority of employers don't pay SW Engineers enough and don't incentivise them enough so they give a shit. This really has to be a company culture, not forced but lived especially by managers, then it would work with engineers as well.
So he just got $6000 once for saving $600k/yr? Let's say the company didn't overspent on that, especially if you factor in that after 5 years €3M has already been saved. I don't know the incentives for sales, but I can imagine that theirs look better than 0,2% (if you just limit the projected savings over 5 years).
There's a saying that I can't quite remember, so I'll just say it in my own words: reward people based on a metric, and that metric becomes their main concern. Worse still, they will learn to game the metric.
Building a new system that doesn't have a baseline cost? If the cost saving bonus is based on a reduction in existing cost then you don't get rewarded for a new implementation. The business don't have a baseline cost yet. So what do you do? Build it to run at a higher price, run it for a few months and then cut costs.
Also, is the reverse true? Deny bonus if costs rise. Because if it is, then the engineer has to prove that the increases are directly related to increased customer activity. But what if increases aren't relative to customer spend? Etc
You have the same bad incentive when finance comes and say "cut 10% this year and 8% next year". You can porobably already cut over 20% today - after the needed work - but you will do the bare minimum because otherwise you will have less budget and possibility to cut even further next year.
Why to charity? Give it to them directly. That being said, you'll create bad incentives: for a team to create badly optimized infra cost then to reduce it and cash in the difference.
Cloud computing made financial control the new engineering. Unless your service is running at an astronomical scale, nobody cares that your refactor saved 50 MB of RAM. The cloud vendors don't care if your workload needs 128 MB RAM, 256 MB RAM, 8 GB RAM. They'll run anything. Just pay for it.
The industry is in a weird spot right now where Finance doesn't understand Engineering well enough to provide proper direction on where Engineering spend is truly wasteful (versus necessarily large), and Engineering quite frankly doesn't care, as long as Finance keeps paying the bills. It's rare to find leadership that appreciates both the underlying financial concern as well as where best inside their tech stack the financial waste lies and how to tackle it and prevent it in the first place.
I always find it so weird that the first time in my whole career that I ever got authority to spend without literally any approval was when we got our cloud account. Every single other thing, even a $10 cab ride, had to be authorised. But when we signed up for AWS I just do whatever I do and finance pays the bill. I still do not understand how this happened.
But do they have to care? The right answer is to set an actual budget. Finance to engineering: you have $x to work with.
Then engineering cares, and the needed talks/concern about resource usage can happen. Part of engineering is optimization, cost is one metric to optimize.
You need to understand the engineering to set a sensible budget. What should the concrete budget be for a new 50-story skyscraper in New York City? You'd need to know quite a bit about the specific building and the construction market to answer that question. You couldn't just be some MBA with a spreadsheet.
Marketing can only go so far. Are more people finally starting to take their heads out of the cloud? Not only do cloud providers need to pay their own costs, they also need to make a profit on top of that; thus, if one thinks critically, the only time renting can be cheaper than owning is if it's short-term.
Their costs plus their profits is still cheaper than hiring your own people to rebuild what they've already done, except taking much longer and usually ending up with worse service.
Unless you're an infrastructure or network logistics company, it's a waste of resources to build your own. Cloud services are so commoditized already it's unlikely to be any sort of meaningful differentiator for your company. Especially if your field isn't even tech proper.
I hear this core competency stuff a lot and it’s just not so. I need organizations as small as teams of video editors run their own “datacenters” which may consist of as little as a Jellyfin server. A waste of resources? Is centralization into cloud datacenters or even CDNs all the way to the last mile a more efficient use of resources than plugging a 10gig connection into a NAS? Why haven’t all restaurants been destroyed by Taco Bell when Taco Bell has economies of scale and efficiency most restaurants couldn’t dream of?
Plenty of mid sized data centres make sense too. Something I see in common with many of these businesses is either the requirement to store a lot of stateful data or to have data locality and to not be concerned with things like being able to serve international customers or being a plodding business which will predict at a steady predictable rate. Some people run businesses that run headfirst into cloud companies entire pricing strategy because their business doesn’t work like a VC funded tech startup.
The “but somebody can figure all this stuff out for you” cuts both ways. Why can’t a skilled company make it easier to staff and operate an on premise data centre? Running a datacenter only gets less labour intensive for what you get every year. You know that you can hire people for pretty much any aspect of building or maintaining a datacenter that you can imagine? The datacenter is commodified so why is it special that the cloud is?
I just feel this analysis is a spurious oversimplification and what you’re far more likely to see is a sort of Pareto distribution of datacenters and the cloud is never going to swallow the whole market.
The thing is, there is an awful lot of YAGNI going on.
Engineering a web app in a way that autoscaling is effective is a lot of work. It is very well possible that renting a beefy dedicated server will do just fine and cost a magnitude or two less in renting and engineering both...
Also, people tend to say I lost my mind when I say complex and expensive high availability solutions are often not worth it: it's cheaper to be down. Because you will be down, no matter what you do, there are a multitude of services you will rely on -- and some downtime is completely expected and accepted in this industry. So adding a few minutes every few years when you need to manually fall over to a slave database versus using a complex HA database solution? A cost-benefit analysis is in order.
If you do need the cloud, yay but so many people no longer even consider whether they do.
> still cheaper than hiring your own people to rebuild what they've already done
That's plain false. Cloud companies are charging significant markups.
> except taking much longer and usually ending up with worse service ... it's a waste of resources to build your own ... Cloud services are so commoditized
All incorrect generalizations. Plenty of companies have capable engineers running their own services for cheaper. Do the math.
People do cook instead of eating out . Analogies are not problem-solvers, and it's a false dichotomy because owning is not the only alternative to the cloud
As far as I know, none of my farmers and grocers are building penis rockets and flying them to space, so they might be working with a bit lower margins than AWS.
The cloud is much more than that. The premium you pay for a cloud service is
- Ability to scale up and scale down in a matter of minutes.
- Reduction in variable capital costs required to meet peak demand.
- Ability to test and prototype products or services at scale without significant investments.
- Ability to have, on tap, products and services that in a traditional sense, would take a lot of time and manpower (A company can invest in equipment and manpower to spin a few services. But as you add more services, it costs more and more to build and maintain such systems. With cloud services, you can use, test and experiment before finalizing)
> Reduction in variable capital costs required to meet peak demand.
If I understand your point correctly it is the opposite - cloud not only lets you trade capex for opex, but also allows for opex variance. Yes, clouds do let you scale up and down (if the application is designed to do that) easily, however how many businesses do really have high load and geography variability?
That is the core point behind cloud vs on-prem: capex vs opex and opex variance.
> if one thinks critically, the only time renting can be cheaper than owning is if it's short-term.
If you skip economies of scale that is. AWS, Azure and GCP allow you to use stuff that has cost billions of man hours and matériel to develop and deploy. How much would it cost you to build a global network? A globally distributed DB? Custom ARM/Tensor chips which come out much cheaper than mainstream alternatives for some workloads?
If all you need is to host a few apps in a narrow geographic area, or do compute/graphics intensive stuff on an inflexible scale, it will probably cost less over a long period of time (3-5 years). But it's still much more inflexible, time consuming and hard. You need to hire the right people and they need the time to order, configure and deploy all that infrastructure.
No. Forget the economies of scale, because they don't help you, as a buyer. What helps you is leverage.
The economies of scale only get passed to the consumer if another provider is willing to pass the savings on, which only happens in commodity markets (like oil, wheat, etc.). Low-end server rentals are a competitive market. "Big 3" cloud is not: they use differentiated products and egress fees to lock you in to their oligopoly.
They don't have to pass on their economies of scale to you. They don't pass on their economies of scale to you. They charge you slightly less than your cost to leave them, and that's it.
There are plenty of cloud providers that offer cheap and feature-rich services if you look away from the hyperscalers, disclaimer: I'm part of a team building one. However most users aren't swayed by only price so it's a tough market to break into.
I was under the impression that outsourcing leveraged economies of scale and ought to be cheaper, if anything. Cloud-everything becoming commonplace does leave me worried that SaaS will drive out SaaP / self-hosted solutions, if it hasn’t already.
> thus, if one thinks critically, the only time renting can be cheaper than owning is if it's short-term.
There's no real alternative to cloud-based hosting if your application needs to be reachable from more than one geographical region or has wildly volatile usage patterns.
I run a small consulting practice and we have pivoted almost exclusively to helping customers reduce cloud spend. Across 100+ engagements, with cloud spend raging from $500k-$40M per year, I can confidently state the following:
>90% of companies are overspending on cloud resources. On average, 30-40% depending on provider (AWS is typically the worst).
>75% of IT/Engineering teams have overbuilt their environments to leverage the latest & greatest tech stacks / toolsets with no justifiable business need.
>50% will confidently state they have “optimized” their environment and in almost every case there is money being left on the table. We have only engaged with one customer that we couldn’t offer savings. They spend $30M+ per year on AWS and had a full time staff of 3 engineers solely focused on cost optimization.
>50% of IT / Engineering leaders became incredibly defensive when presented with savings analysis which almost always included reduction in technical complexity of stack. We have since pivoted and marketing/engaging almost exclusively to CEO/CFO as first contact.
>50% of companies had one of more cloud costing/monitoring tools in place that was not being leveraged, or worse, gamed to produce inaccurate reports.
I can confirm having had a similar experience, albeit at a smaller scale (fewer customers but similar annual spend.)
My favourite find was a database backup someone configured where it was keeping full uncompressed daily backups forever. Fixing just that cut their spend by $100K annually, and then we kept going.
The worst offenders are governments, especially in some countries that haven't had sufficiently brutal economic downturns recently to force anyone to do some belt tightening. Australian government is spectacularly wasteful, burning piles of money in the cloud for services that only malicious bots ever visit.
Like you said, these places love to pad out there resumes at the expense of the taxpayer. I regularly see multiple Kubernetes clusters deployed for one web app. Or architecture diagrams that look like a spiderweb of connecte d systems for something that should be running on a single VM.
Readit News
Meanwhile, how many are giving their teams the resources (i.e., people) necessary to do the analytics on the bill required to find & cut costs? Every cloud I've worked with has been terrible at billing in a way that could be reasonable investigated. At one point we literally pulled all the billing data into a SQLite database. (And that was a great idea, as it made queries against it much faster.)
Further, most higher ups seem to have a sense of "it won't be invented here", and so a lot of stuff ends up in vendor land; need more people then to pull those separate bills into one spot, and start attributing costs against the revenue they might help generate.
I've also worked a fair number of places where people were cagey about telling eng how much they're spending where. If you want your IT dept. to cut costs, they have to know where the money is going.
And then there's the yak shaving of it all: we've had a support ticket open w/ Azure for 9 days now asking "where did this $1k go?". Our Blob Storage "Cool" tier costs upticked, and I'm 99% sure the bill there is just wrong. Cool storage should be (price * amount stored) and the "how much are we storing" graph is not proportional to the cost graph. The only gotcha is an early delete penalty, and we are also pretty confident we're not hitting that. (We know what's being deleted when, and none of it should be subject to it.) But how does it take a over week to answer the question?
AWS in particular is handsomely engineered so you can't answer this sort of question, as well as making people spend more without need.
- Per resource cost is turned off by default. Turning it on costs money. How much? ¯\_(ツ)_/¯
- Services like S3 will bill you per API call. How do you know who is calling your APIs? Using Cloud trail, which is turned off by default. Turning it on for this sort of analysis might cost a lot $$$ for a reasonably used bucket.
- CloudWatch will bill you per custom metric and adds up quickly. Why do I have so many custom metrics? Good luck finding out.
- Many costs are tied with stuff that's really hard to control e.g. egress traffic.
- Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc...
- Many services are ha, multi az only. That makes AWS unnecessarily expensive for dev environments / non critical services.
The list goes on. I know a fair number of consulting companies making the money of their lives reducing AWS bills.
And you can have business support only on one account... obviously you can go Enterprise and have the same support level on everything but pay a lot more $$$
Not if you use shared VPCs.
One of the most effective ways I've found to drive change in how product management and execs treat engineering was getting average salary numbers by type of position for my team out of HR, and attributing costs toward every major feature request + meetings in a weekly report.
My manager at the time, the CEO of a VC funded startup, did not see the point. We were after all already paying for the engineers, and they were always fully loaded. Until he'd seen a couple of reports, and saw what it told us about prioritisation.
Our project and product managers got a lot less meeting-happy after seeing one regular meeting they were very fond of cost an estimated 10k/month due to the number of people attending, and they got a lot more thoughtful about feature request and how to prioritize after seeing the real costs of supposedly "trivial" requests that had previously been slipped past normal processes.
Same applies on the cloud front, yet a lot of what has driven the adoption of cloud is that it often helps engineering escape cost analysis and efforts to optimize costs.
And so you get organisations which often have no idea which decisions contribute what to those costs.
I was considering writing to Azure support about that...
My last billing issue with them took ~120 days to resolve, so I don't have much hope.
You are also right that it is across storage accounts, and access tiers: I've now looked at some of our other SAs, including ones in the Hot tier, and yep, they see the same bizarre uptick. Given that Hot tier is experiencing it, that rules out the possibility of "human error" w.r.t. to it not being a Cool-tier early deletion.
> Is this similar to what you're seeing ?
You've nailed it precisely; it sounds exactly like what we're seeing.
(What a small world HN is sometimes!)
¹the numbers on our end wiggle a bit, so I'm not sure about how precise I can estimate the uptick. But 70% is as close as any guess.
- run on demand instances (maybe mix in spot)
- run fargate instances (maybe mix in spot)
- reserve them.
Every company hates #3, but it's usually the best choice if you can't go spot - you can swap them within the family for size, use them for baseline, it's fine. You're incorporated right? So if you fold, it's just another debt on the pile, stop worrying. Go no upfront if you really want to avoid the cash outlay, or go spot if you can defend limited outages.
- Engineers may want to switch sizes or go serverless.
- Long term vendor lock-in.
- Miss out on price cuts.
There’s a reason the vendors push them so hard. (No spot market on Azure either)
In my experience it takes azure support over a week to answer any question. Unless the question happens to be which person without a substantial update is taking over the case. You’ll get lots and lots of those answers.
Azure forced me onto invoice billing against my will - and every month without fail they don't automatically match my payment and I have to endure increasingly threatening emails that my account is pending termination while their support takes days to reply to my tickets about it, and when they do reply the advice is usually just 'wait a little longer'.
I've moved critical services off Azure because the risk of wrongful termination is just too high.
Someone else's computer has been painful and very expensive for our organisation so far. Escalating costs aren't even the worst problem.
Wrote to support, explained my situation, and they removed the bill without any hassle.
Check your partial / interrupted multi-part uploads. If its like amazon the uncommitted blobs won't show up in storage graphs but you'll be getting charged for them just sitting there.
Surprisingly enough we have 3 people working on it right now.
Sadly they’ve found that it was extremely easy to save several thousands of dollars a day, and has been so for the past two years.
Terrible for who? You or the cloud provider? I'm a bit of a worrier (and not a perfect coder), so infinite scalability with utility billing combined with a mistake on my part could result in an epic bill. All I'll say is that I won't practice cloud development on my own credit card.
Sounds like the more basic problem is that we keep ending up working in companies where the "higher ups" are constantly shoving decisions down our throats.
Which, ultimately, is probably the root cause of most of the caginess, the yak shaving, the compunction to use cloud databases and cloud-everything, and the silly support tickets that stay open forever and ever and ever.
If you don't fund, hire, and staff your support org with good people at multiple tiers, don't be surprised when you get crap. There just aren't enough good minds to service the incoming workload.
Oh, and for god's sake have a decent "to engineering" escalation path for the support folks.
Unfortunately, the support function is almost always a cost center and beholden to only, if anything, response time related SLAs. Instead of customer satisfaction, etc.
I don't get time for that - I've got access to Cloudability but no one has ever showed me how to use it.
I don't believe a baseline exists. The cloud is a new thing, and the fashion on how to use it is still changing constantly.
You also can not use this number as a proxy for any other thing. It's just what it is, you can't look for deeper connections.
That feels like it should be 100%.
An engineer should be as aware of their $ costs as they are their CPU load, filesystem I/O, or memory usage. An engineer should know how many instances they scaled to, and when they're going to scale down.
In fact, the $ cost should be calculable (to a reasonable indicative amount by napkin math - precision not necessary) according to the factors the engineer knows.
Not being aware of the costs feels irresponsible on the part of the engineer. There was a time when we had finite capacity (a single server) and perhaps the engineer had to care less about the cost then (but perhaps they should've cared about utilisation by some measure even then)... but now we're in an age of your application being able to scale beyond reasonable bounds of your companies purse strings, a solid understanding of what your application costs is an essential and core skill. If this is not the game you wanted to play, stay on a VPS or VM.
Anyone possessing a solid understanding will also be looking to reduce or halt their spending... hence, I am surprised the number in the headline is not higher, even ignoring the macro-economic picture.
Never once has a dev asked about storage costs, of the differences in storing a file in a filesystem vs a blob in a database,
Never once ask about mem optimization could reduce infrastructure costs
never once
In theory most places are running with slack. In practice they run under capacity, and engineers aren't going to worry about things out of their control.
It’s a matter of priority. During periods of revenue or user growth, reducing infrastructure cost may be a premature optimization. Now may be a time to focus on operational and logistical inefficiencies
Even during hyper growth you should be aware of your costs, because you should have an idea of your margin.
I'm not sure any engineer that has awareness of costs, CPU, memory, storage, network, etc... is not trying to reduce some of it - even during hyper growth.
The most common response (39%) was the C suite wanted spending to remain the same.
People often say this, but it's just not true. First, any company with significant spend has custom deals with custom pricing in exchange for a commit. Second, even if you don't have that, there's often reserved/"savings plans"/whatever where you commit in exchange for a lower price.
Third, the only time any of the big 3 cloud providers has increased pricing (it usually only goes down) is when GCP increased bandwidth and storage costs a few months ago. AWS, which is what the majority of the market uses, has only decreased prices, hundreds of times.
Regarding VMware, you know you have to pay for licenses and support in an ongoing manner and they are known to jack up prices, right? Furthermore, VMware's software is a dumpster fire that gets you around 5% of what you get from a serious public cloud. Maybe 30%-50% if you go in with NSX and vSAN and the vSuite where you lock in yourself to terrible software with the wrong abstractions, lack of APIs, at enormous inflexible costs. How is that any better?
Probably your savings will come from beign able to get bigger machines. After 4 years in the cloud I am still surprised to see how performant physical servers are.
But my real gripe is that this survey is being presented as doom and gloom when you could equally summarize the results as "about 60% of IT leaders directed to stay the course or beef up cloud spending"
1. Broadcom, if history is any indication, is about to make VMware much less of a bargain, likely exceeding the cost of the cloud, if not matching it.
2. Often time when a company moves to a cloud model, they lay off the traditional administration staff in favor of "DevOps", many many devops engineers would not have the first clue how to manage the low level infrastructure of Storage SAN, FiberChannel, vLAN's etc... let alone managing vSphere or hyperv, etc..
3. Often times when a company moves to a cloud model they decommission their Onprem Hardware, Lead times right now are crazy still so any org wanted to move to OnPrem quickly is going to be hard pressed to do so.
Granted, I have no idea how the costs compare. In every company I have been at, the higher-ups tell us in tech we have to "learn about the business side of things", yet the business side never wants to share information.
The graphic doesn't even have a category for being directed to halt spend and less than half are being directed to reduce it in any way so I'm gonna say I still improved it ;)
Wait, what? There's no way 29% of companies noped their way out of cloud provider lock-in. Not saying it doesn't happen, but 29% is a huge number.
Unless of course they're startups with no idea how they want their infrastructure run or legacy companies dipping their feet (?) in the cloud for the first time.
We're a rather simple company though, less than a dozen services, we use Postgres and Redis for storage, everything deployed on Kubernetes.
Watch how fast the bill will go down
Funny enough, for all the problems with studying business administration, they do teach this as part of the principal-agent problem. If the principal wants something done, and the agent has no incentive to do it, you need to give the agent some of the incentive so their interests are aligned with yours.
At the same time it's something you rarely see implemented by BA types, because they don't like to share.
- Corrosive effects on team cohesion, when separate ICs compete to see who can find cost savings first.
- The "cobra effect", where engineers will deliberately introduce cost inefficiencies so that they can fix their own bugs later.
- (Related to the previous point) Prevention of cost overruns is even cheaper than fixing leaks after the fact, but is very hard to properly incentivize.
Any class on principal-agent problems will hopefully include a chapter warning against asymmetric incentives and how they will be ruthlessly exploited by the agents. Engineers are not some exception here, in fact because of their training they are often extra good at spotting loopholes in the rules.
Google "Hanoi rats" for an incentive blowup. Then there's myriad other incentive issues related to taxes and subsidies, all stuff that had the good intention of promoting good things and reducing bad things.
Sometimes its a meta-game: if a team sees a "share the spoils" scheme being trialled with another team, maybe they shouldn't fix their issues until there's a scheme in place for their team? What about the shape of the payoff? If you fix two issues in the same period, are you better off or worse off than splitting them over a boundary?
Even in sports, there's issues: there's an incentive payment to break the world record. Say you can pole vault 10cm higher than the previous record in your practices. But why break it once when you can break it 10 times? I read about this some time ago, should be findable online.
Mangers at one of my previous workplaces were very vocal with "everything is running with slacks" so whenever I gave them an estimate on how long a feature would take to implement or a bug to be fixed (bugfix estimates are BS to begin with but...) they took it and multiplied it by 0.75 at most. Then bitched about it when it was over the time they assigned. Guess what happened.
Exactly this would happen with a cost-cutting incentive as well - if managers told us "reduce cost by 30% to the end of the year" then the cost would creep up to begin with. Incompetent management making stupid rules and stupid incentives is what drives me nuts :) .
The devil really is in the detail: "you need to give the agent some of the incentive so their interests are aligned with yours". The vast majority of employers don't pay SW Engineers enough and don't incentivise them enough so they give a shit. This really has to be a company culture, not forced but lived especially by managers, then it would work with engineers as well.
One guy turned off staging at night and saved $600k/yr.
Building a new system that doesn't have a baseline cost? If the cost saving bonus is based on a reduction in existing cost then you don't get rewarded for a new implementation. The business don't have a baseline cost yet. So what do you do? Build it to run at a higher price, run it for a few months and then cut costs.
Also, is the reverse true? Deny bonus if costs rise. Because if it is, then the engineer has to prove that the increases are directly related to increased customer activity. But what if increases aren't relative to customer spend? Etc
Any other similar platforms out there?
The industry is in a weird spot right now where Finance doesn't understand Engineering well enough to provide proper direction on where Engineering spend is truly wasteful (versus necessarily large), and Engineering quite frankly doesn't care, as long as Finance keeps paying the bills. It's rare to find leadership that appreciates both the underlying financial concern as well as where best inside their tech stack the financial waste lies and how to tackle it and prevent it in the first place.
Then engineering cares, and the needed talks/concern about resource usage can happen. Part of engineering is optimization, cost is one metric to optimize.
Unless you're an infrastructure or network logistics company, it's a waste of resources to build your own. Cloud services are so commoditized already it's unlikely to be any sort of meaningful differentiator for your company. Especially if your field isn't even tech proper.
Plenty of mid sized data centres make sense too. Something I see in common with many of these businesses is either the requirement to store a lot of stateful data or to have data locality and to not be concerned with things like being able to serve international customers or being a plodding business which will predict at a steady predictable rate. Some people run businesses that run headfirst into cloud companies entire pricing strategy because their business doesn’t work like a VC funded tech startup.
The “but somebody can figure all this stuff out for you” cuts both ways. Why can’t a skilled company make it easier to staff and operate an on premise data centre? Running a datacenter only gets less labour intensive for what you get every year. You know that you can hire people for pretty much any aspect of building or maintaining a datacenter that you can imagine? The datacenter is commodified so why is it special that the cloud is?
I just feel this analysis is a spurious oversimplification and what you’re far more likely to see is a sort of Pareto distribution of datacenters and the cloud is never going to swallow the whole market.
Engineering a web app in a way that autoscaling is effective is a lot of work. It is very well possible that renting a beefy dedicated server will do just fine and cost a magnitude or two less in renting and engineering both...
Also, people tend to say I lost my mind when I say complex and expensive high availability solutions are often not worth it: it's cheaper to be down. Because you will be down, no matter what you do, there are a multitude of services you will rely on -- and some downtime is completely expected and accepted in this industry. So adding a few minutes every few years when you need to manually fall over to a slave database versus using a complex HA database solution? A cost-benefit analysis is in order.
If you do need the cloud, yay but so many people no longer even consider whether they do.
That's plain false. Cloud companies are charging significant markups.
> except taking much longer and usually ending up with worse service ... it's a waste of resources to build your own ... Cloud services are so commoditized
All incorrect generalizations. Plenty of companies have capable engineers running their own services for cheaper. Do the math.
If I understand your point correctly it is the opposite - cloud not only lets you trade capex for opex, but also allows for opex variance. Yes, clouds do let you scale up and down (if the application is designed to do that) easily, however how many businesses do really have high load and geography variability?
That is the core point behind cloud vs on-prem: capex vs opex and opex variance.
If you skip economies of scale that is. AWS, Azure and GCP allow you to use stuff that has cost billions of man hours and matériel to develop and deploy. How much would it cost you to build a global network? A globally distributed DB? Custom ARM/Tensor chips which come out much cheaper than mainstream alternatives for some workloads?
If all you need is to host a few apps in a narrow geographic area, or do compute/graphics intensive stuff on an inflexible scale, it will probably cost less over a long period of time (3-5 years). But it's still much more inflexible, time consuming and hard. You need to hire the right people and they need the time to order, configure and deploy all that infrastructure.
The economies of scale only get passed to the consumer if another provider is willing to pass the savings on, which only happens in commodity markets (like oil, wheat, etc.). Low-end server rentals are a competitive market. "Big 3" cloud is not: they use differentiated products and egress fees to lock you in to their oligopoly.
They don't have to pass on their economies of scale to you. They don't pass on their economies of scale to you. They charge you slightly less than your cost to leave them, and that's it.
There's no real alternative to cloud-based hosting if your application needs to be reachable from more than one geographical region or has wildly volatile usage patterns.
Dead Comment
My last company had all the AWS goodies, spending thousands per month...on a site that got around 50 hits a day.
>90% of companies are overspending on cloud resources. On average, 30-40% depending on provider (AWS is typically the worst).
>75% of IT/Engineering teams have overbuilt their environments to leverage the latest & greatest tech stacks / toolsets with no justifiable business need.
>50% will confidently state they have “optimized” their environment and in almost every case there is money being left on the table. We have only engaged with one customer that we couldn’t offer savings. They spend $30M+ per year on AWS and had a full time staff of 3 engineers solely focused on cost optimization.
>50% of IT / Engineering leaders became incredibly defensive when presented with savings analysis which almost always included reduction in technical complexity of stack. We have since pivoted and marketing/engaging almost exclusively to CEO/CFO as first contact.
>50% of companies had one of more cloud costing/monitoring tools in place that was not being leveraged, or worse, gamed to produce inaccurate reports.
My favourite find was a database backup someone configured where it was keeping full uncompressed daily backups forever. Fixing just that cut their spend by $100K annually, and then we kept going.
The worst offenders are governments, especially in some countries that haven't had sufficiently brutal economic downturns recently to force anyone to do some belt tightening. Australian government is spectacularly wasteful, burning piles of money in the cloud for services that only malicious bots ever visit.
Like you said, these places love to pad out there resumes at the expense of the taxpayer. I regularly see multiple Kubernetes clusters deployed for one web app. Or architecture diagrams that look like a spiderweb of connecte d systems for something that should be running on a single VM.