Readit News logoReadit News
Posted by u/DeborahEmeni_ 5 months ago
Why do small teams keep getting wrecked by AWS bills?
Saw someone on Reddit lose $86k from a compromised AWS account. I've heard way too many stories like this — misconfigured IAM, tokens in repos, no billing alerts...

If you're on a small team, how are you actually protecting yourself from stuff like this? Is there a sane setup that works without needing a full-time AWS security person?

PaulHoule · 5 months ago
Gotta look at your monthly bill every day.

I've been wondering about the question of "where did the web go?" and how even technically savvy people have given up on blogging for behavioral sinks like Medium.

Part of the story is that $50 a month dedis have given way to the "free" plan on AWS or a system that costs $10 a month to run if you're not successful but has no upper bound on the bills if you are successful. So if you make a blog you are praying every night that you don't make it to the front of Hacker News and that you don't build up a large following because boy those egress charges will add up. People are furious now that they are getting eaten alive by the egress costs run up by AI bots but 10 years ago I was thinking "Boy Bing crawls my site twice as hard as Google and sends 5% of the traffic and Chinese webcrawlers crawl my site 5x harder than Google and send me no detectable traffic."

DeborahEmeni_ · 4 months ago
yeah makes sense. the idea that success = risk now is kind of wild. did you end up doing anything to limit that egress, or just ride it out?
scarface_74 · 5 months ago
They look at code on stackoverflow and the web that initializes the SDK resources that have you explicitly put the access key and secret key in code.

For instance, the correct way to initialize the s3 client in Python is

     s3 = boto3.client('s3')
The SDK will automatically get the credentials that are configured locally within your environment or the IAM role attached to your Lambda, EC2 instance, Docker (ECS, EKS) container runner etc.

Your access keys never need to be part of your repository.

DeborahEmeni_ · 4 months ago
yep this is the one. feels like a lot of people just copy-paste from stackoverflow without realizing they're hardcoding keys. have you seen this cause issues in prod or mostly in hobby stuff?
QuinnyPig · 5 months ago
One of the best things you can do is enable AWS’s free Cost Anomaly Detector. I like getting pinged when something substantial changes.
DeborahEmeni_ · 4 months ago
yeah cost anomaly detector is underrated. do you usually set custom thresholds or just let it use the defaults?
QuinnyPig · 4 months ago
Start with defaults; you can turn it back if it’s too noisy.
ArinaS · 5 months ago
DeborahEmeni_ · 4 months ago
yep that’s the one. rough situation but the replies were interesting
chistev · 5 months ago
I like that you linked old reddit. Using the normal domain makes the site utterly unusable.
DeborahEmeni_ · 4 months ago
agreed. old reddit actually works. no extra stuff and doesn’t get in the way
_rm · 5 months ago
Because AWS doesn't have out-of-the-box costs killswitches.
DeborahEmeni_ · 4 months ago
yeah that’s the part that gets me too. feels wild that there’s no built-in way to cap spend at the account level. even a “pause all” button would go a long way
neuroelectron · 5 months ago
Best way to defeat competition is to make sure they never get out of the gate
DeborahEmeni_ · 4 months ago
yeah exactly. feels like the pricing model alone keeps a lot of folks from even starting. one wrong config and you’re done before you ship anything.
8b16380d · 5 months ago
We’re not since cost controls are secondary to building features.
DeborahEmeni_ · 4 months ago
makes sense tbh. i’ve seen teams take that approach and only think about cost once it’s already a problem. ever had that come back to cause issues later?

Deleted Comment