MrAwesome (u/MrAwesome)

MrAwesome commented on Mystery Blips mosquitocapital.substack.... · Posted by u/bo0tzz

Google didn't have SRO. The closest thing was Traffic Team's oncall; Which was stressful, for sure, and hired some dedicated alcoholics, but it wasn't anything like Facebooks (I knew people on both teams during my time at each). Google had a constant stream of alerts that someone was aware of and handling. Facebook had a constant stream of fire that someone was beating back the flames of.

SRO is madness. SRO is admitting that you're enough underwater and there's enough unexpected critical events to dedicate a team to burn out on it. There was a lot of spend in overcapacity and failover automation testing that kept it that way, and explicit mandates by a C-Suite and VP to keep keeping it that way. More importantly, though, it was baked into the architecture.

Everyone's talking this week about how Twitter will go up in flames in the next few days; I don't think so. I think they're closer to Google in that respect. There's a lot of automation that will keep things alive and a lot of redundancy that will handle the immediate problems of having a lot of the team leave. I actually think the company can continue to run indefinitely if they just follow the operations manuals and replace failed drives and machines. But I think there will be a major outage when they try to launch a new feature and fail.

MrAwesome · 3 years ago

Author here - a few months after the time of this story, we reoriented around a single purpose for the team, "Automate ourselves out of existence". It took a few months to remove all of the dependencies, tag all the alarms, train up all the team-specific oncalls, but we managed to do it, and I had the great honor of taking the very last SRO oncall on March 31, 2014 :)

MrAwesome commented on Mystery Blips mosquitocapital.substack.... · Posted by u/bo0tzz

retrocryptid · 3 years ago

Lol. During "high velocity events" at Amazon, we would have a war room with all the engineers and a couple old hands directing responses.

Some of our metrics came in 5 minutes delayed which wasn't a problem for normal days. These metrics moved slow enough that when you got an alarm, there was still plenty of time to take corrective action.

But for HVEs this was an issue. During black Friday or prime day, sometime some metrics spiked so fast you had no time to respond (usually from people hitting page reload a few minutes before a sale kicked off.)

To get an idea for what was going on, I would go in twitter and search for things like "amazon failure" or "amazon 502."

We often got problem reports via Twitter before they showed up on our dashboards.

MrAwesome · 3 years ago

In my time in SRO at FB, we actually had an alarm that would fire whenever enough #facebookisdown posts were made on Twitter!

MrAwesome commented on Mystery Blips mosquitocapital.substack.... · Posted by u/bo0tzz

IAmGraydon · 3 years ago

It's funny - the very first thing I thought while reading this was "check the news". That's probably because I've been involved in markets and when you see this sort of behavior unfold in market price and volume, the first thing you do is check the news.

MrAwesome · 3 years ago

Yeah! By the end of my time on the team, checking the news when I saw a rapid change in traffic was second nature. And later on in my career becoming interested in markets, even more so. Forex in particular - markets with trillions in daily liquidity don't take huge leaps unless something huge happened.

MrAwesome commented on Mystery Blips mosquitocapital.substack.... · Posted by u/bo0tzz

nerdponx · 3 years ago

One interesting aspect of the story is that the author was a junior at the time (almost brand new to the job, if I read it correctly), and it was one of the more experienced operators who realized it was exogenous.

in addition to being a good story about the site reliability, it's also a great lesson in the value of collaboration, mentorship, and having senior people around whom you can ask for help!

MrAwesome · 3 years ago

Author here, and that's exactly right! I think it was in my second or third month on the job.

And I really agree with your second point too! Without that in-person transfer of tribal knowledge, it would have taken me ages to learn the ropes well enough to take such a high-pressure high-stakes oncall.

MrAwesome commented on Why does everyone think Twitter is doomed? twitter.com/mosquitocapit... · Posted by u/ShoutAtTheDevil

rekabis · 3 years ago

When the foundation to almost every company’s long-term success is the people it has hired, eviscerating those ranks of people indiscriminately is a really great way to have loads of institutional knowledge flee and the job become much, much harder for anyone who remains.

At this point, my only immediately rational conclusion is that Elon is acting out some sort of a revenge fantasy, trying to create as much harm to Twitter as possible without any care that the company might actually collapse. Or maybe even to that end, explicitly, just with plausible deniability.

These tweets makes some great arguments that give legs to my supposition - anyone with any minimal amount of business sense would know of these arguments, and yet Elon is ignoring them wholesale to the detriment of Twitter. What remains is the determination of whether it is ignorance or intentional malice.

MrAwesome · 3 years ago

OP of that thread here. Wanted to say that the first paragraph of your reply is the perfect encapsulation of what I wanted to get across in that post. It's not about the disaster scenarios, it's about the fact that these scenarios every web-scale company faces every day become a hell of a lot scarier when you just lost >50% of the company without an orderly transition

MrAwesome commented on What is the web revival? thoughts.melonking.net/gu... · Posted by u/spansoa

scyclow · 4 years ago

The Worm is a really fun one. The idea is that you transfer the worm NFT from wallet to wallet (it's "trying to visit every wallet on the Ethereum blockchain"), and it leaves a non-transferrable token wherever it's been. To my knowledge, it's been transferred about 750 times without anyone selling it. https://theworm.wtf/

Another cool one is minting amulets. You write a poem, and if the hash of that poem has an amulet in it (four or more consecutive 8s), then you can mint it as a token for your collection. But you can't mint the same hashed poem twice https://text.bargains/collection/

I also met someone once who gave me (and everyone he meets) an NFT from an RFID chip implanted in his hand. That was pretty whacky.

MrAwesome · 4 years ago

Pardon my ignorance here, but do you need to pay gas fees or buy tokens to transfer/mint?

MrAwesome commented on What is the web revival? thoughts.melonking.net/gu... · Posted by u/spansoa

voldacar · 4 years ago

Monero isn't commercialized. It's just a currency that people actually use for transactions, not a product.

MrAwesome · 4 years ago

A currency is generally intended mainly for commercial purposes, isn't it? I think the word commercialized isn't quite right for what I mean then.

So more specifically, I mean: "not for the purpose of creating or transferring monetary value, and not requiring the expenditure of money to create/send the amount of data necessary to host a simple plaintext/html page"

MrAwesome commented on What is the web revival? thoughts.melonking.net/gu... · Posted by u/spansoa

scyclow · 4 years ago

It makes me very happy to know that all of these websites exist.

It also makes me sad to see unnecessary hostility towards crypto. I get it: most NFTs are hyper commercialized trash, and they're in direct opposition to the values outlined in this article. But there's also so much weird, non-commercialized crypto stuff out there that shares a lot of the ethos of early web tinkerers.

MrAwesome · 4 years ago

I am not aware of any non-commercialized crypto anything (from my understanding, this is sorta by definition, but maybe I'm missing something). Do you have any examples?

MrAwesome commented on Investing in lighting did great things for my mental and physical health bramadams.dev/projects/in... · Posted by u/_bramses

abletonlive · 4 years ago

I'm not convinced about blackout curtains being healthy. Why is it that when you go camping you feel so refreshed and good when the sun comes up and wakes you up?

Did our far ancestors use blackout curtains? How did they deal with the sun waking them up everyday? My guess is that it's the optimal way to wake up actually.

MrAwesome · 4 years ago

Yeah, as other commenters have said - the curtains are for bright city lights I can't turn off.

Personally, I have a simple electrical timer set to turn on a bright sunlight bulb at 6:30am, and I open the curtains during the day.

You're right on the money about camping and how it affects the body, in my experience. Light obviously isn't the only factor in camping feeling good - but I've also been reading lately that light exposure (and possibly direct sun exposure?) during the day is quite good for you.

MrAwesome commented on Investing in lighting did great things for my mental and physical health bramadams.dev/projects/in... · Posted by u/_bramses

MrAwesome · 4 years ago

It seems like there has been a lot of buzz lately about the importance of reducing light exposure at night and the impact it has on your health. I can certainly speak to it anecdotally - getting a nice pair of blackout curtains and putting electrical tape over the electronics LEDs in my bedroom seemed to do wonders for my sleep.

A quick search for sources led me to this article, which has a nice collection of links to studies for those interested in learning more: https://www.sleepfoundation.org/bedroom-environment/light-an...