When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity. No problem that makes sense - it's an easy way to prove authentication as a bot can't spam a credit card (or else it would be financial fraud and most likely a felony).
Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion and if I ever had to use cloud computing I'd use anyone else for that very reason. The entire service with it's horrifically complicated click through dashboard (but you can get a certification! It's so complicated they invented a fake degree for it!) just to confuse the customer into losing money.
I still blame them for missing an opportunity to be good corporate citizens and fight bot spam by using credit cards as auth. But if I go to the grocery store I can use a credit card to swipe, insert, chip or palm read (this is now in fact a thing) to buy a cookie. As opposed to using financial technology for anything useful.
This is an example of why cloud hosting is so scary.
Yes, Amazon, and I assume Azure and Google's cloud and others, "usually" refund the money.
But I don't want to be forced into bankruptcy because my five visitor a week demo project suddenly becomes the target of a DDOS for no reason at all and the hosting company decides this isn't a "usually" so please send the wire transfer.
They refund those that know how to demand it, and that notice. If you have complex infra and not a lot of observability, you'll just assume the costs are legitimate. Imagine how much they're making off of those oops moments. Probably a bug chunk of their revenue reports.
When I am playing around in the cloud I am super paranoid about charges, so I end up locking the ACLs to only permit traffic to my home IP. It’s too bad that they don’t have a better built in way of making sandbox labs. When I was doing cloud training with A Cloud Guru, it would generate a whole global AWS instance that would only last for 30 minutes.
If you sign up for electrical service for your house, and your shithead neighbor taps your line to power his array of grow lamps and crypto mining rigs, the power company will happily charge you thousands of dollars, and you will need a police report and traverse many layers of customer service hell to get a refund. If you sign up for water service and a tree root cracks your pipe, the water company will happily charge you thousands of dollars for the leaked water, and will then proceed to mandate that you to fix the broken pipe at your own expense for a couple tens of thousands more; and yes, that may well bankrupt you, water company don't care. So why do you expect different treatment from a computing utility provider?
I think one of the reasons I appreciate AWS so much is that any time there has been snafu that led to a huge bill like this they've made it pretty painless to get a refund- just like you experienced.
If it is a "free tier", Amazon should halt the application when it exceeds quota. Moving the account to a paid tier and charging $100k is not the right thing to do.
Amazon is currently permissive which splits opposition, this won’t always be the case, they will tighten the screws eventually as they have done in the past in other areas. Amazon because it’s so broadly used undermines the utility of chargebacks, you can do it but it’ll be a real hassle to not be able to use Amazon for shopping. A lot of people will just eat the costs, is Amazon knows this they will force the situation more often because it’ll make them more money.
As the saying goes, when you owe the bank $100 you've got a problem, when you owe the bank $100k the bank has a problem...
On serverless, I can enter numbers in a calculator and guess that running my little toy demo app on AWS will cost between $1 and $100. Getting hit with a huge $1000 bill and a refusal to refund the charges (and revocation of my Prime account and a lifetime ban from AWS and cancellation of any other services I might otherwise run there) would be totally possible, but I have zero control over that. Expecting to go on social media begging for a refund is not a plan, it's evidence of a broken system - kinda like those "heartwarming" posts about poor people starting a GoFundMe so their child can afford cancer treatment. No, that's awful, can we just be sensible instead?
If a server would have cost me $20 at a VPS provider to keep a machine online 24/7 that was at 1% utilization most of the time and was terribly laggy or crashed when it went viral, that's what $20 buys you.
But, you say, analysis of acttual traffic says that serverless would only cost me $10 including scaling for the spike, in which case that's a fantastic deal. Half price! Or maybe it would be $100, 5x the price. I have no way of knowing in advance.
You are a bit naive. They are making a ton of money with this dark pattern. As others have said Free-to-100K is not in the most generous realm of expectations. Its also why they have been doing the refunds as long as AWS has been a thing. They know it will not hold up in court. Not a month goes by without some HN story about something like this post.
They do this and make it easy to get a refund because for every demo account that does it some bigger account accidentally gets billed 10K and they have to pay for it. They have skin in the game and cannot risk their account being down for any time period.
Once I've been kidnapped by a guy who also happen to run a security business. After a bit of discussion, I was about to convince some of his sbire to release me without paying the ransom. I'm so glad they did accept that, and I never fail to use and recommend the services of the security business now.
I've got a $25k bill right now because I had enabled data-plane audit logging on an sqs queue that about a year ago I had wired to receive a real-time feed of audit events. So for every net-new audit event there would be an infinite loop of write events to follow. My average daily bill is about $2 on that account and has been for nearly ten years. It suddenly ballooned to $3k/day and zero warning or intervention from AWS.
Since this seems to be getting some comments. Yes, it is in fact easy to shut down an instance if it goes over a spending limit. As in you monitor traffic tied directly to the billing system and you set up an if statement and if it goes over the limit you shut down the server and dump the service to a static drive.
It's the easiest thing in the world - they just don't want to because they figured that they could use their scale to screw over their customers. And now you have the same guys who screwed everyone over with cloud compute wanting you to pay for AI by using their monopoly position to charge you economic rents. Because now things like edge compute is easy because everyone overspent on hard drives because of crypto. And so you have jerks who just move on to the next thing to use their power to abuse the market rather than build credibility because the market incentivizes building bubbles and bad behavior.
Smart evil people who tell others "no you're just too dumb to 'get it' (oh by the way give me more money before this market collapses)" are the absolute bane of the industry.
It's weird that you have people in here defending the practice as if it's a difficult thing to do. Taxi cabs somehow manage not to charge you thousands of dollars for places you don't drive to but you can't set up an if statement on a server? So you're saying Amazon is run by people that are dumber than a taxi cab company?
Ok, well you might have a point. And this is how Waymo was started. I may or may not be kidding.
> I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion
They refunded you $100k with few questions asked, and you hate them for it?
I’ve made a few expensive mistakes on AWS that were entirely my fault, and AWS has always refunded me for them.
I imagine if Amazon did implement “shut every down when I exceed my budget” there’d be a bunch of horror stories like “I got DDOSed and AWS shutdown all my EC2s and destroyed the data I accidentally wrote to ephemeral storage.”
> They refunded you $100k with few questions asked, and you hate them for it?
They exposed him to 100K of liability without any way to avoid it (other than to avoid AWS entirely), and then happened to blink, in this case, with no guarantee that it would happen again. If you don't happen to have a few hundred thousand liquid, suddenly getting a bill for 100K might well be a life-ruiningly stressful event.
Given how complicated configuring AWS is, surely there could be some middle ground between stop all running services and delete every byte of data. The former is surely what the typical low spend account would desire.
In what world is that not the preferable solution? Want to know if your shit is actually robust just set your cap and ddos yourself as the first test of you architecture.
> horror stories like “I got DDOSed and AWS shutdown all my EC2s and destroyed the data I accidentally wrote to ephemeral storage.”
I mean, S3 also incurs ongoing charges, so if you're going to stop accruing charges you'd also be deleting your data that wasn't on ephemeral storage...
And potentially deleting all of your DNS zones (and recreating them will likely give you different nameservers so you'll need to wait for the registrar to update them once you're back)...
Something similar happened to me, but not at the outrageous scale. I wanted to try some AI example on Bedrock. So the tutorial said I needed to set up some OpenSearch option. Voila. A few days later I had a bill for $120. The scale is not as horrible, but the principle is the same.
I’ve never trusted AWS with personal work for exactly this reason. If I want to spend $20 on a personal project I should be able to put a cap on that directly, not wake up to a $100k bill and go through the stress of hoping it might be forgiven.
I use AWS out of expedience but I hate the no-hard-cap experience and this is my primary reason for shifting (WIP) to self hosting. Plus self hosting is cheaper for me anyway. In general I would like a legally forced liability limit on unbounded subscription services, perhaps a list maintained at the credit card level. If the supplier doesn’t like the limit they can stop supplying. The surprise $100K liabilities are pure insanity.
> When I was learning to program through a bootcamp I spun up an elastic beanstalk instance
Didn't the bootcamp told you to, at least, setup a budget alert?
I'm not trying to reduce AWS' responsibility here, but if a teaching program tells you to use AWS but doesn't teach you how to use it correctly, you should question both AWS and the program's methods.
> cloud computing I'd use anyone else for that very reason. The entire service with it's horrifically complicated click through dashboard just to confuse the customer into losing money.
I feel like this brand of sentiment is everywhere. Folks want things simple. We often figure out what we need to do to get by.
Over time we learn the reason for a handful of the options we initially defaulted through, find cause to use the options. Some intrepid explorers have enough broader context and interest to figure much more out but mostly we just set and forget, remembering only the sting of facing our own ignorance & begrudging the options.
This is why k8s and systemd are have such a loud anti-following.
It’s interesting because on the posted site there’s only 2 AWS posts on the main page and they’re rather mild compared to the other posts using google, vercel, cloudformation, etc.
> When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity. No problem that makes sense
It does on the surface, but what doesn't make sense is to register with a credit card and not read the terms very carefully: both for the cloud service and for the bank service.
In this aspect cash is so much better because you have only one contract to worry about...
> When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity.
Is it just me or is this just a cheap excuse to grab a payment method from unsuspecting free-tier users?
AWS services aren't designed for people just learning to program. Beanstalk and other services have billing limits you can set, but those aren't hard limits because they are measured async to keep performance up.
With that said, AWS is notoriously opaque in terms of "how much will I pay for this service" because they bill so many variable facets of things, and I've never really trusted the free tier myself unless I made sure it wasn't serving the public.
Not that Amazon needs any defending, but for anyone running a bootcamp: this is a good reason to start with services like Heroku. They make this type of mistake much harder to make. They're very beginner friendly compared to raw AWS.
It's easy yes, but better than nothing. The verification requirements are a balance between desired conversion rate, probability of loss (how many bad guys want to exploit your system without paying) and the actual costs of said loss (in this case it's all bullshit "bandwidth" charges, so no actual loss to AWS).
allways set cost alarm and max spending. AWS has great tools to controll costs.
You could have blocked this with good config but I understand its confusing and not super apparent. IMHO there should be a pop up or sth asking " you want to stop the instance the moment it costs anything?"
its so easy to get billed a ridicules amount if money
> Amazon then charged me one hundred thousand dollars as the server was hit by bot spam.
That would make you one of the most successful websites on the internet, or the target of a DDoS -- which was it? I assume you're not saying that "bots" would randomly hit a single, brand-new "hello world" site enough to generate that kind of bill.
Many of the people who have this problem on toy websites end up offering what amounts to free storage or something similar. They are then surprised when "bots" come to "DDoS" them. These bills are as much a product economics problem as a technical one.
Did you do any training before launching the elastic beanstalk instance, or you just though a F-16 should be pretty easy to fly, at least according to most pilots?
An F-16 doesn't have a prominently-featured "getting started" tutorial, which has a bunch of step-by-step instructions getting a complete novice 40.000ft into the air at mach 2.
AWS also provides training and education on how to use their services. If launching a "hello world" Elastic Beanstalk instance is so dangerous, why doesn't the tutorial require you to first provide proof that you are an AWS Certified Cloud Practitioner?
> The entire service with it's horrifically complicated click through dashboard (but you can get a certification! It's so complicated they invented a fake degree for it!) just to confuse the customer into losing money.
By that logic, any technology that you can get certified in is too complicated?
Most systems are now distributed and presenting a holistic view of how it was designed to work can be useful to prevent simple mistakes.
Traffic requires a certification (license) too. Must be a fake degree as well because they made it too complicated
> By that logic, any technology that you can get certified in is too complicated?
That is a common view in UX, yes. It's a bit of an extreme view, but it's a useful gut reaction
> Traffic requires a certification (license) too. Must be a fake degree as well because they made it too complicated
In the US roads are designed so that you need as close to no knowledge as possible. You need to know some basic rules like the side of the road you drive on or that red means stop, but there is literal text on common road signs so people don't have to learn road signs. And the driving license is a bit of a joke, especially compared to other Western countries
There is something to be said about interfaces that are more useful for power users and achieve that by being less intuitive for the uninitiated. But especially in enterprise software the more prevalent effect is that spending less time and money on UX directly translates into generating more revenue from training, courses, paid support and certification programs
The history of making things complicated often involves "unintended" use by malicious actors.
But infact, it is intended side effects. Things like Jaywalking or "no spitting" laws let police officers harass more people _At their whim_. And they're fullying designed that way but left as "unintended" for the broader public scrutiny.
So, just like, learn that "logic" is not some magic thing you can sprinkle on everything and find some super moral or ethic reality. You have to actually integrate the impact through multiple levels of interaction to see the real problem with "it's just logic bro" response you got here.
The problem with the AWS certificate is that the entity issuing the certificate and the entity honoring the certificate have opposing priorities. When a company wants to use AWS, preferably they'd want to avoid needlessly expensive solutions and vendor lock-in, while Amazon wants to teach people how to choose needlessly expensive solutions with vendor lock-in.
> By that logic, any technology that you can get certified in is too complicated?
In IT, I am inclined to agree with that. In real engineering, it's sometimes necessary, especially dangerous technology and technology that people trust with their life
Businesses are only taxed on actual revenue earned.
What you decide to charge—whether $100, $50, or even giving it away for free—is purely a business decision, not a tax one.
—
This is different from a nonprofit donation scenario though. For example, if your service normally costs $X but you choose to provide it for free (or at a discount) as a donation to a non-profit, you can typically write off the difference.
I took a workshop class and was told to setup a track saw. The course didn't bother explaining how to utilize it properly or protect yourself. I ended up losing a finger. I truly hate Stanley Tools with a passion and if I ever need to use another track saw, I'll use someone else.
This analogy would make sense if the saw lacked a basic and obvious safety feature (billing limits) because Stanley profited immensely from cutting your finger off.
Protect yourself how? Most cloud providers don't support any way to immediately abort spending if things get out of hand, and when running a public-facing service there are always variables you can't control.
Even if you rig up your own spending watchdog which polls the clouds billing APIs, you're still at the mercy of however long it takes for the cloud to reconcile your spending, which often takes hours or even days.
I thought this would be about the horrors of hosting/developing/debugging on “Serverless” but it’s about pricing over-runs. I scrolled aimlessly through the site ignoring most posts (bandwidth usage bills aren’t super interesting) but I did see this one:
> I thought this would be about the horrors of hosting/developing/debugging on “Serverless” but it’s about pricing over-runs.
Agreed about that. I was hired onto a team that inherited a large AWS Lambda backend and the opacity of the underlying platform (which is the value proposition of serverless!) has made it very painful when the going gets tough and you find bugs in your system down close to that layer (in our case, intermittent socket hangups trying to connect to the secrets extension). And since your local testing rig looks almost nothing like the deployed environment...
I have some toy stuff at home running on Google Cloud Functions and it works fine (and scale-to-zero is pretty handy for hiding in the free tier). But I struggle to imagine a scenario in a professional setting where I wouldn't prefer to just put an HTTP server/queue consumer in a container on ECS.
I've had similar experiences with Azures services. Black boxes impossible to troubleshoot. Very unexpected behavior people aren't necessarily aware of when they initially spin these things up. For anything important I just accept the pain of deploying to kubernetes. Developers actually wind up preferring it in most cases with flux and devsoace.
Every time I've done a cost benefit analysis of AWS Lambda vs running a tiny machine 24/7 to handle things, the math has come out in favor of just paying to keep a machine on all the time and spinning up more instances as load increase.
There are some workloads that are suitable for lambda but they are very rare compared to the # of people who just shove REST APIs on lambda "in case they need to scale."
Is that what people do is test/develop primarily with local mocks of the services? I assumed it was more like you deploy mini copies of the app to individual instances namespaced to developer or feature branch, so everyone is working on something that actually fairly closely approximates prod just without the loading characteristics and btw you have to be online so no working on an airplane.
I raised that exact same issue to AWS in ~2015 and even though we had an Enterprise support plan, AWS response was basically: well, you problem.
We then ended up deleting the S3 bucket entirely, as that appeared to be the only way to get rid of the charges, only for AWS to come back to use a few weeks later telling us there are charges for an S3 bucket we previously owned. After explaining to them (again) that this way our only option to get rid of the charges, we never heard back.
Seems an interesting oversight. I can just imagine the roundtable, uhh guys who do we charge for 403? Who can we charge? But what if people hit random buckets as an attack? Great!
How to destroy your competition. Love it. Also why i dislike AWS. Zero interest to protect their SMB customers from surprise bills. Azure isn't much better but at least they got a few more protections in place.
Same, I was hoping for tales of woe and cloud lock-in, of being forced to use Lambda and Dynamo for something that could easily run on a $20/month VPS with sqlite.
The webflow one at the top has an interesting detail about them not allowing you to offload images to a cheaper service. Which you can probably work around by using a different domain.
> I reported my findings to the maintainers of the vulnerable open-source tool. They quickly fixed the default configuration, although they can’t fix the existing deployments.
Anyone wanna guess which open source tool this was? I'm curious to know why they never detected this themselves. I'd like to avoid this software if possible as the developers seem very incompetent.
> Imagine you create an empty, private AWS S3 bucket in a region of your preference. [...] As it turns out, one of the popular open-source tools had a default configuration to store their backups in S3. And, as a placeholder for a bucket name, they used… the same name that I used for my bucket.
What are the odds?
(Not a rhetorical question. I don't know how the choice of names works.)
The assignment of blame for misconfigured cloud infra or DOS attacks is so interesting to me. There don't seem to be many principles at play, it's all fluid and contingent.
Customers demand frictionless tools for automatically spinning up a bunch of real-world hardware. If you put this in the hands of inexperienced people, they will mess up and end up with huge bills, and you take a reputational hit for demanding thousands of dollars from the little guy. If you decide to vet potential customers ahead of time to make sure they're not so incompetent, then you get a reputation as a gatekeeper with no respect for the little guy who's just trying to hustle and build.
I always enjoy playing at the boundaries in these thought experiments. If I run up a surprise $10k bill, how do we determine what I "really should owe" in some cosmic sense? Does it matter if I misconfigured something? What if my code was really bad, and I could have accomplished the same things with 10% of the spend?
Does it matter who the provider is, or should that not matter to the customer in terms of making things right? For example, do you get to demand payment on my $10k surprise bill because you are a small team selling me a PDF generation API, even if you would ask AWS to waive your own $10k mistake?
Then you’re the person who took down their small business when they were doing well.
At AWS I’d consistently have customers who’d architected horrendously who wanted us to cover their 7/8 figure “losses” when something worked entirely as advertised.
Small businesses often don’t know what they want, other than not being responsible for their mistakes.
Yes and no. 100% accurate billing is not available in realtime, so it's entirely possible that you have reached and exceeded your cap by the time it has been detected.
Having said that, within AWS there are the concepts of "budget" and "budget action" whereby you can modify an IAM role to deny costly actions. When I was doing AWS consulting, I had a customer who was concerned about Bedrock costs, and it was trivial to set this up with Terraform. The biggest PITA is that it takes like 48-72 hours for all the prerequisites to be available (cost data, cost allocation tags, and an actual budget each can take 24 hours)
The real serverless horror isn't the occasional mistake that leads to a single huge bill, it's the monthly creep. It's so easy to spin up a resource and leave it running. It's just a few bucks, right?
I worked for a small venture-funded "cloud-first" company and our AWS bill was a sawtooth waveform. Every month the bill would creep up by a thousand bucks or so, until it hit $20k at which point the COO would notice and then it would be all hands on deck until we got the bill under $10k or so. Rinse and repeat but over a few years I'm sure we wasted more money than many of the examples on serverlesshorrors.com, just a few $k at a time instead of one lump.
this is really the AWS business model - you can call it the "planet fitness" model if you prefer. Really easy to sign up and spend money, hard to conveniently stop paying the money.
Sounds like your organization isn’t learning from these periods of high bill. What lead to the bill creeping up, and what mechanisms could be put in place to prevent them in the first place?
At only 20k a month, the work put into reducing the bill back down probably costs more in man hours than the saving, time which would presumably be better spent building profitable features that more than make up for the incremental cloud cost. Assuming of course the low hanging fruit of things like oversized instances, unconstrained cloudwatch logs and unterminated volumes have all been taken care of.
> what mechanisms could be put in place to prevent them in the first place?
Those mechanisms would lead to a large reduction in their "engineering" staff and the loss of potential future bragging rights in how modern and "cloud-native" their infrastructure is, so nobody wants to implement them.
With that model, your cost doesn't change, though. When/if you find you need more resources, you can (if you haven't been doing so) audit existing applications to clear out cruft before you purchase more hardware.
> I had cloudflare in front of my stuff. Hacker found an uncached object and hit it 100M+ times. I stopped that and then they found my origin bucket and hit that directly.
Pardon my ignorance, but isn’t that something that can happen to anyone? Uncached objects are not something as serious as leaving port 22 open with a weak password (or is it?). Also, aren’t S3 resources (like images) public so that anyone can hit them any times they want?
This story is giving "I leave OWASP top 10 vulns in my code because hacker mindset".
It's not that hard to configure access controls, they're probably cutting corners on other areas as well. I wouldn't trust anything this person is responsible for.
It's about rate limiting, not access controls. Without implementing limits your spend can go above what your budget is. Without cloud you hit natural rate limits of the hardware you are using to host.
No, s3 objects should always be private and then have a cloudfront proxy in front of them at the least. You should always have people hitting a cache for things like images.
I don't understand why it should be called "serverless" when using cloud infrastructure. Fundamentally you're still creating software following a client-server model, and expecting a server to run somewhere so that your users' clients work.
To me, "serverless" is when the end user downloads the software, and thereafter does not require an Internet connection to use it. Or at the very least, if the software uses an Internet connection, it's not to send data to a specific place, under the developer's control, for the purpose of making the software system function as advertised.
A "Server" is typically a single machine that has a specific OS and runs layers of various software that allows your business logic to be accessed by other computers (by your users). For a "Server" you typically have to choose an OS to run, install all the support software (server monitoring, etc), update the software, and if the server fails you have to fix it or rebuild it.
With "Serverless", your code is in a "function as a service" model where all you have to worry about is the business logic (your code). You don't have to set up the server, you don't have to install the server OS, or any basic server software that is needed to support the business logic code (http server, etc). You don't have to update the server or the underlying server software. You don't have to perform any maintenance to keep the server running smoothly. You never (typically) have to worry about your server going down. All you have to do is upload your business logic function "somewhere" and then your code runs when called. Essentially you do not have to deal with any of the hassle that comes with setting up and maintaining your own "server", all you have to do is write the code that is your business logic.
That's why it's called "Serverless" because you don't have to deal with any of the hassle that comes with running an actual "server".
> Essentially you do not have to deal with any of the hassle that comes with setting up and maintaining your own "server", all you have to do is write the code that is your business logic.
Also known as "shared hosting". It's been done since the 90's (your folder full of PHP files is an NFS mount on multiple Apache servers), just that the techbros managed to rebrand it and make it trendy.
I understand the underlying reasoning. I just don't like the terminology. Hence, "I don't understand... should be", rather than "... is". I think it's wrong that people end up using words like that. Like, almost on a moral level.
More generally, I don't like that a term ending with "-less" marks an increase in system complexity.
Serverless is easier to say than "load controlled ephemeral server management." Which is the real point. As my load increases the number of allocated resources, like servers, increases, and as it decreases so do the allocations and costs.
This is great if you are willing to completely change your client-server code to work efficiently in this environment. It is a strain over a standard design and you should only be using it when you truly need what "serverless" provides.
I keep telling customers: "The cloud will scale to the size of your wallet."
They don't understand what I mean by that. That's okay, they'll learn!
Anyway, this kind of thing comes up regularly on Hacker News, so let's just short-circuit some of the conversations:
"You can set a budget!" -- that's just a warning.
"You should watch the billing data more closely!" -- it is delayed up to 48 hours or even longer on most cloud services. It is especially slow on the ones that tend to be hit the hardest during a DDoS, like CDN services.
"You can set up a lambda/function/trigger to stop your services" -- sure, for each individual service, separately, because the "stop" APIs are different, if they exist at all. Did I mention the 48 hour delay?
"You can get a refund!" -- sometimes, with no hard and fast rules about when this applies except for out of the goodness of some anonymous support person's heart.
"Lots of business services can have unlimited bills" -- not like this where buying what you thought was "an icecream cone" can turn into a firehouse of gelato costing $1,000 per minute because your kid cried and said he wanted more.
"It would be impossible for <cloud company> to put guardrails like that on their services!" -- they do exactly that, but only when it's their money at risk. When they could have unlimited expenses with no upside, then suddenly, magically, they find a way. E.g.: See the Azure Visual Studio Subscriber accounts, which have actual hard limits.
"Why would you want your cloud provider to stop your business? What if you suddenly go viral! That's the last thing you'd want!" -- who said anything about a business? What if it's just training? What if your website is just marketing with a no "profit per view" in any direct sense?
Bit of a nit pick but this is a pet peeve of mine.
Creating a new word for a more specific category is never Orwellian. The project in 1984 was to create a language which was less expressive. They were destroying words describing fine distinctions and replacing them with words that elided those distinctions. Creating a new word to highlight a distinction is the opposite.
There's definitely criticisms to be made of the term serverless and how it obscures the role of servers, but Orwellian is not the correct category. Maybe we could say such services run on servelets to describe how they're "lighter" in some sense but still servers.
Yea, I agree after more thought. I think the key is what you said; the term is useful for dividing within a specific domain. People outside that domain see the word and think "those guys are calling this Category-A thing "not-category-A", that makes no sense! Inside the Category A world, there is much more nuance.
They’re clearly referring to “doublethink” which was absolutely part of Newspeak in 1984…
Quote from the book:
“The Ministry of Peace concerns itself with war, the Ministry of Truth with lies, the Ministry of Love with torture and the Ministry of Plenty with starvation. These contradictions are not accidental, nor do they result from ordinary hypocrisy: they are deliberate exercises in doublethink.”
Serverless being in fact server-based seems like a pretty clear example of this, and so calling it an Orwellian term seems perfectly reasonable.
This is where is becomes confusing to me: Here are a few types of software/infrastructure. Embedded devices. Operating systems. PC software. Mobile device software. Web frontends. GPU kernels. These all truly don't use servers. When I hear "serverless", I would think it is something like that. Yet, they're talking about web servers. So it feels like a deception, or something poorly named.
If you are in the niche of IT, servers, HTTP operations etc, I can see why the name would make sense, because in that domain, you are always working with servers, so the name describes an abstraction where their technical details are hidden.
Readit News
Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion and if I ever had to use cloud computing I'd use anyone else for that very reason. The entire service with it's horrifically complicated click through dashboard (but you can get a certification! It's so complicated they invented a fake degree for it!) just to confuse the customer into losing money.
I still blame them for missing an opportunity to be good corporate citizens and fight bot spam by using credit cards as auth. But if I go to the grocery store I can use a credit card to swipe, insert, chip or palm read (this is now in fact a thing) to buy a cookie. As opposed to using financial technology for anything useful.
Yes, Amazon, and I assume Azure and Google's cloud and others, "usually" refund the money.
But I don't want to be forced into bankruptcy because my five visitor a week demo project suddenly becomes the target of a DDOS for no reason at all and the hosting company decides this isn't a "usually" so please send the wire transfer.
We can't implement a basic cost limiter policy.
I think we all know why.
https://docs.aws.amazon.com/cost-management/latest/userguide...
I think one of the reasons I appreciate AWS so much is that any time there has been snafu that led to a huge bill like this they've made it pretty painless to get a refund- just like you experienced.
At minimum they should provide hard billing caps.
On serverless, I can enter numbers in a calculator and guess that running my little toy demo app on AWS will cost between $1 and $100. Getting hit with a huge $1000 bill and a refusal to refund the charges (and revocation of my Prime account and a lifetime ban from AWS and cancellation of any other services I might otherwise run there) would be totally possible, but I have zero control over that. Expecting to go on social media begging for a refund is not a plan, it's evidence of a broken system - kinda like those "heartwarming" posts about poor people starting a GoFundMe so their child can afford cancer treatment. No, that's awful, can we just be sensible instead?
If a server would have cost me $20 at a VPS provider to keep a machine online 24/7 that was at 1% utilization most of the time and was terribly laggy or crashed when it went viral, that's what $20 buys you.
But, you say, analysis of acttual traffic says that serverless would only cost me $10 including scaling for the spike, in which case that's a fantastic deal. Half price! Or maybe it would be $100, 5x the price. I have no way of knowing in advance.
It's just not worth the risk.
They do this and make it easy to get a refund because for every demo account that does it some bigger account accidentally gets billed 10K and they have to pay for it. They have skin in the game and cannot risk their account being down for any time period.
It's the easiest thing in the world - they just don't want to because they figured that they could use their scale to screw over their customers. And now you have the same guys who screwed everyone over with cloud compute wanting you to pay for AI by using their monopoly position to charge you economic rents. Because now things like edge compute is easy because everyone overspent on hard drives because of crypto. And so you have jerks who just move on to the next thing to use their power to abuse the market rather than build credibility because the market incentivizes building bubbles and bad behavior.
Smart evil people who tell others "no you're just too dumb to 'get it' (oh by the way give me more money before this market collapses)" are the absolute bane of the industry.
It's weird that you have people in here defending the practice as if it's a difficult thing to do. Taxi cabs somehow manage not to charge you thousands of dollars for places you don't drive to but you can't set up an if statement on a server? So you're saying Amazon is run by people that are dumber than a taxi cab company?
Ok, well you might have a point. And this is how Waymo was started. I may or may not be kidding.
They refunded you $100k with few questions asked, and you hate them for it?
I’ve made a few expensive mistakes on AWS that were entirely my fault, and AWS has always refunded me for them.
I imagine if Amazon did implement “shut every down when I exceed my budget” there’d be a bunch of horror stories like “I got DDOSed and AWS shutdown all my EC2s and destroyed the data I accidentally wrote to ephemeral storage.”
They exposed him to 100K of liability without any way to avoid it (other than to avoid AWS entirely), and then happened to blink, in this case, with no guarantee that it would happen again. If you don't happen to have a few hundred thousand liquid, suddenly getting a bill for 100K might well be a life-ruiningly stressful event.
I mean, S3 also incurs ongoing charges, so if you're going to stop accruing charges you'd also be deleting your data that wasn't on ephemeral storage...
And potentially deleting all of your DNS zones (and recreating them will likely give you different nameservers so you'll need to wait for the registrar to update them once you're back)...
And...
Didn't the bootcamp told you to, at least, setup a budget alert?
I'm not trying to reduce AWS' responsibility here, but if a teaching program tells you to use AWS but doesn't teach you how to use it correctly, you should question both AWS and the program's methods.
Deleted Comment
I feel like this brand of sentiment is everywhere. Folks want things simple. We often figure out what we need to do to get by.
Over time we learn the reason for a handful of the options we initially defaulted through, find cause to use the options. Some intrepid explorers have enough broader context and interest to figure much more out but mostly we just set and forget, remembering only the sting of facing our own ignorance & begrudging the options.
This is why k8s and systemd are have such a loud anti-following.
It does on the surface, but what doesn't make sense is to register with a credit card and not read the terms very carefully: both for the cloud service and for the bank service.
In this aspect cash is so much better because you have only one contract to worry about...
Is it just me or is this just a cheap excuse to grab a payment method from unsuspecting free-tier users?
With that said, AWS is notoriously opaque in terms of "how much will I pay for this service" because they bill so many variable facets of things, and I've never really trusted the free tier myself unless I made sure it wasn't serving the public.
Dead Comment
Given the relative accessibility of stolen credit card info, isn't the CC-as-ID requirement easy for a criminal to bypass?
its so easy to get billed a ridicules amount if money
Dead Comment
I call bullshit
That would make you one of the most successful websites on the internet, or the target of a DDoS -- which was it? I assume you're not saying that "bots" would randomly hit a single, brand-new "hello world" site enough to generate that kind of bill.
AWS also provides training and education on how to use their services. If launching a "hello world" Elastic Beanstalk instance is so dangerous, why doesn't the tutorial require you to first provide proof that you are an AWS Certified Cloud Practitioner?
> as in how am I going to pay it?
Really?
Amazon charged your card for $100,000 and your bank allowed it?
You're filthy rich by most people's standard, and you were able to pay it.
Amazon was operating in such a good faith that they ate the computational cost you spent. And you hate them for this to this day...
Deleted Comment
Dead Comment
Dead Comment
By that logic, any technology that you can get certified in is too complicated?
Most systems are now distributed and presenting a holistic view of how it was designed to work can be useful to prevent simple mistakes.
Traffic requires a certification (license) too. Must be a fake degree as well because they made it too complicated
That is a common view in UX, yes. It's a bit of an extreme view, but it's a useful gut reaction
> Traffic requires a certification (license) too. Must be a fake degree as well because they made it too complicated
In the US roads are designed so that you need as close to no knowledge as possible. You need to know some basic rules like the side of the road you drive on or that red means stop, but there is literal text on common road signs so people don't have to learn road signs. And the driving license is a bit of a joke, especially compared to other Western countries
There is something to be said about interfaces that are more useful for power users and achieve that by being less intuitive for the uninitiated. But especially in enterprise software the more prevalent effect is that spending less time and money on UX directly translates into generating more revenue from training, courses, paid support and certification programs
But infact, it is intended side effects. Things like Jaywalking or "no spitting" laws let police officers harass more people _At their whim_. And they're fullying designed that way but left as "unintended" for the broader public scrutiny.
So, just like, learn that "logic" is not some magic thing you can sprinkle on everything and find some super moral or ethic reality. You have to actually integrate the impact through multiple levels of interaction to see the real problem with "it's just logic bro" response you got here.
It is a fake degree.
In IT, I am inclined to agree with that. In real engineering, it's sometimes necessary, especially dangerous technology and technology that people trust with their life
They get a nice tax write-off.
It's couch-cushion change for them, but it adds up. They have whole armies of beancounters, dreaming this stuff up.
It's also the game behind those "coupons" you get, for outrageously-priced meds that aren't covered by insurance.
If they charge $1,000 for the medication, but give you a "special discount" for $90%, they get to write off $900.
Businesses are only taxed on actual revenue earned.
What you decide to charge—whether $100, $50, or even giving it away for free—is purely a business decision, not a tax one.
—
This is different from a nonprofit donation scenario though. For example, if your service normally costs $X but you choose to provide it for free (or at a discount) as a donation to a non-profit, you can typically write off the difference.
Even if you rig up your own spending watchdog which polls the clouds billing APIs, you're still at the mercy of however long it takes for the cloud to reconcile your spending, which often takes hours or even days.
You forgot to mention Stanley Tools paid for the hospital bill.
https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-c...
About how you make unauth’d API calls to an s3 bucket you don’t own to run up the costs. That was a new one for me.
Agreed about that. I was hired onto a team that inherited a large AWS Lambda backend and the opacity of the underlying platform (which is the value proposition of serverless!) has made it very painful when the going gets tough and you find bugs in your system down close to that layer (in our case, intermittent socket hangups trying to connect to the secrets extension). And since your local testing rig looks almost nothing like the deployed environment...
I have some toy stuff at home running on Google Cloud Functions and it works fine (and scale-to-zero is pretty handy for hiding in the free tier). But I struggle to imagine a scenario in a professional setting where I wouldn't prefer to just put an HTTP server/queue consumer in a container on ECS.
There are some workloads that are suitable for lambda but they are very rare compared to the # of people who just shove REST APIs on lambda "in case they need to scale."
Dead Comment
We then ended up deleting the S3 bucket entirely, as that appeared to be the only way to get rid of the charges, only for AWS to come back to use a few weeks later telling us there are charges for an S3 bucket we previously owned. After explaining to them (again) that this way our only option to get rid of the charges, we never heard back.
Anyone wanna guess which open source tool this was? I'm curious to know why they never detected this themselves. I'd like to avoid this software if possible as the developers seem very incompetent.
What are the odds?
(Not a rhetorical question. I don't know how the choice of names works.)
Deleted Comment
Customers demand frictionless tools for automatically spinning up a bunch of real-world hardware. If you put this in the hands of inexperienced people, they will mess up and end up with huge bills, and you take a reputational hit for demanding thousands of dollars from the little guy. If you decide to vet potential customers ahead of time to make sure they're not so incompetent, then you get a reputation as a gatekeeper with no respect for the little guy who's just trying to hustle and build.
I always enjoy playing at the boundaries in these thought experiments. If I run up a surprise $10k bill, how do we determine what I "really should owe" in some cosmic sense? Does it matter if I misconfigured something? What if my code was really bad, and I could have accomplished the same things with 10% of the spend?
Does it matter who the provider is, or should that not matter to the customer in terms of making things right? For example, do you get to demand payment on my $10k surprise bill because you are a small team selling me a PDF generation API, even if you would ask AWS to waive your own $10k mistake?
At AWS I’d consistently have customers who’d architected horrendously who wanted us to cover their 7/8 figure “losses” when something worked entirely as advertised.
Small businesses often don’t know what they want, other than not being responsible for their mistakes.
Having said that, within AWS there are the concepts of "budget" and "budget action" whereby you can modify an IAM role to deny costly actions. When I was doing AWS consulting, I had a customer who was concerned about Bedrock costs, and it was trivial to set this up with Terraform. The biggest PITA is that it takes like 48-72 hours for all the prerequisites to be available (cost data, cost allocation tags, and an actual budget each can take 24 hours)
Imagine the horror stories on Hacker News that would generate.
I worked for a small venture-funded "cloud-first" company and our AWS bill was a sawtooth waveform. Every month the bill would creep up by a thousand bucks or so, until it hit $20k at which point the COO would notice and then it would be all hands on deck until we got the bill under $10k or so. Rinse and repeat but over a few years I'm sure we wasted more money than many of the examples on serverlesshorrors.com, just a few $k at a time instead of one lump.
Those mechanisms would lead to a large reduction in their "engineering" staff and the loss of potential future bragging rights in how modern and "cloud-native" their infrastructure is, so nobody wants to implement them.
Sure they're probably VMs but their cost isn't 0 either
Pardon my ignorance, but isn’t that something that can happen to anyone? Uncached objects are not something as serious as leaving port 22 open with a weak password (or is it?). Also, aren’t S3 resources (like images) public so that anyone can hit them any times they want?
I'm glad I use a Hetzner VPS. I pay about EUR 5 monthly, and never have to worry about unexpected bills.
It's not that hard to configure access controls, they're probably cutting corners on other areas as well. I wouldn't trust anything this person is responsible for.
with AWS, you wake up to a 6 figures bill.
To me, "serverless" is when the end user downloads the software, and thereafter does not require an Internet connection to use it. Or at the very least, if the software uses an Internet connection, it's not to send data to a specific place, under the developer's control, for the purpose of making the software system function as advertised.
With "Serverless", your code is in a "function as a service" model where all you have to worry about is the business logic (your code). You don't have to set up the server, you don't have to install the server OS, or any basic server software that is needed to support the business logic code (http server, etc). You don't have to update the server or the underlying server software. You don't have to perform any maintenance to keep the server running smoothly. You never (typically) have to worry about your server going down. All you have to do is upload your business logic function "somewhere" and then your code runs when called. Essentially you do not have to deal with any of the hassle that comes with setting up and maintaining your own "server", all you have to do is write the code that is your business logic.
That's why it's called "Serverless" because you don't have to deal with any of the hassle that comes with running an actual "server".
Also known as "shared hosting". It's been done since the 90's (your folder full of PHP files is an NFS mount on multiple Apache servers), just that the techbros managed to rebrand it and make it trendy.
More generally, I don't like that a term ending with "-less" marks an increase in system complexity.
This is great if you are willing to completely change your client-server code to work efficiently in this environment. It is a strain over a standard design and you should only be using it when you truly need what "serverless" provides.
They don't understand what I mean by that. That's okay, they'll learn!
Anyway, this kind of thing comes up regularly on Hacker News, so let's just short-circuit some of the conversations:
"You can set a budget!" -- that's just a warning.
"You should watch the billing data more closely!" -- it is delayed up to 48 hours or even longer on most cloud services. It is especially slow on the ones that tend to be hit the hardest during a DDoS, like CDN services.
"You can set up a lambda/function/trigger to stop your services" -- sure, for each individual service, separately, because the "stop" APIs are different, if they exist at all. Did I mention the 48 hour delay?
"You can get a refund!" -- sometimes, with no hard and fast rules about when this applies except for out of the goodness of some anonymous support person's heart.
"Lots of business services can have unlimited bills" -- not like this where buying what you thought was "an icecream cone" can turn into a firehouse of gelato costing $1,000 per minute because your kid cried and said he wanted more.
"It would be impossible for <cloud company> to put guardrails like that on their services!" -- they do exactly that, but only when it's their money at risk. When they could have unlimited expenses with no upside, then suddenly, magically, they find a way. E.g.: See the Azure Visual Studio Subscriber accounts, which have actual hard limits.
"Why would you want your cloud provider to stop your business? What if you suddenly go viral! That's the last thing you'd want!" -- who said anything about a business? What if it's just training? What if your website is just marketing with a no "profit per view" in any direct sense?
Creating a new word for a more specific category is never Orwellian. The project in 1984 was to create a language which was less expressive. They were destroying words describing fine distinctions and replacing them with words that elided those distinctions. Creating a new word to highlight a distinction is the opposite.
There's definitely criticisms to be made of the term serverless and how it obscures the role of servers, but Orwellian is not the correct category. Maybe we could say such services run on servelets to describe how they're "lighter" in some sense but still servers.
Quote from the book:
“The Ministry of Peace concerns itself with war, the Ministry of Truth with lies, the Ministry of Love with torture and the Ministry of Plenty with starvation. These contradictions are not accidental, nor do they result from ordinary hypocrisy: they are deliberate exercises in doublethink.”
Serverless being in fact server-based seems like a pretty clear example of this, and so calling it an Orwellian term seems perfectly reasonable.
"Here's some code, make sure it runs once an hour, I don't care where."
There becomes a point where being mad that the specific flavor of PaaS termed serverless achtually has severs is just finding a thing to be mad at.
It doesn't just "have" servers; they aren't a hidden implementation detail. Connecting to a website is an instrumental part of using the software.
If you are in the niche of IT, servers, HTTP operations etc, I can see why the name would make sense, because in that domain, you are always working with servers, so the name describes an abstraction where their technical details are hidden.