Readit News logoReadit News
solfox · 2 years ago
Seeing such a large web property go down like this is fascinating.

It's like a power plant grid failure, except for attention instead of energy.

When meta is down, a hoard of internet users desperately seek somewhere else to place their attention... But the system is designed with the expectation that meta will take all that traffic... And boom! everything starts falling over. Wild.

Vicinity9635 · 2 years ago
Classic failure casdcade.
ykonstant · 2 years ago
And rather than an indication that it is not working, I was just told to login again; since the attempts were unsuccessful, I was led to reset my password. Now I have no clue if the password has been reset properly or an old one is used, or login with google is used, or if I will continue to be logged out after they fix stuff.

If your service is down, please say "my service is down".

mrweasel · 2 years ago
One weekend I was on-call, we had a system we ran for the government, but authentication was handled by another government owned service, hosted and managed by another company.

A user called in early Saturday evening, saying that the system was down. After a bit of debugging, wondering where our alerting had failed, I concluded that the system was perfectly fine, but the authentication service had been returning 500 errors for around an hour. When I called the user back he made a comment that rather changed how I think about monitoring and systems. He says: Well, from my point of view it really doesn't matter which part isn't working, it's just down.

muttled · 2 years ago
I've learned the same lesson, and why things like ping checks aren't enough. Even just a WGET of the front page isn't enough. You need the monitor to login and replicate at least part of the user experience if you have a complex setup. It's very embarrassing when someone tells a user "shows it's up from my side." and they're relying on a naïve monitor.
geerlingguy · 2 years ago
I lament the fact error messages are rarely, if ever, displayed anymore. Just a generic message in its place, usually not even indicating whether it's a 'you' problem or a 'them' problem :(
UniverseHacker · 2 years ago
It amazes me how most smartphone apps- which operate in a wireless environment of likely flaky connections, don't seem to be tested or designed at all to handle loss of connections. They tend to just become non-responsive or do erroneous things like play the first second of an audio file and then auto-pause, without reporting any errors.
diggan · 2 years ago
Yeah, but at least it's a bit understandable that its changed in that direction, and usually you can get a developer-friendly error message if you look at the HTTP responses in the devtools.

I remember one time a company I worked at received a support message where the title was (paraphrased) "DONT HURT MY CHILDREN" from some person who saw an error message saying something about "couldn't dispose of child" or something similar, when the frontend broke. "Child/children" being kind of common in programming, I'm sure others faced similar scenarios.

I'm not sure if they were genuinely scared or just decided to have some fun with us while reporting the issue, but after that we made the error messages even more generic, so nothing could be misunderstood.

I guess it is this quest to not alienate the lowest common denominator that is the reason behind the stupidification of error messages. On one hand, people won't get scared, on the other hand, people get less context about why the error happened in the first place.

Wonder how many of the average users actually care?

mewpmewp2 · 2 years ago
There are so many microservices everywhere with different teams responsible, that if just one team doesn't forward errors properly it is messed up.
MIlBianco · 2 years ago
I thought I got hacked and tried to retrieve my password.

Got a weird email with the same code every time from facebookmail.com

sircastor · 2 years ago
My first thought was that I'd been hacked as well...

I have gotten 4-6 "here's your facebook reset" type emails over the last month. All unrequested. I've always assumed they're casual attacks (mostly interested in seeing if my password can be stuffed into another, more valuable account)

ceejayoz · 2 years ago
I really hate that companies do this. PayPal does it sometimes, too.
mrzaggy · 2 years ago
Password works on THREE other machines, fails completely on ONE.

I was logged in when the outage happened on machine A and phone B - immediately tried to reset password, which took me into the hellish abyss many of us are experiencing now...

This evening - about 18hrs about the event, I fire up machine C - it logs STRAIGHT INTO Messenger. OK... This is something, too scared to open a browser in case that triggers the session disconnect...

So I fire up laptop D - STRAIGHT INTO MESSENGER... OK, open browser, STRAIGHT INTO FB. Log into Google password manager, VISUALLY CHECK password, and it is my last known good password (in use at time of outage).

Fire up iPad E - Straight into Messenger!!! All these machines are on the same network!

Back to Machine A - clear cookies and try to log in, no joy; different browser and try to log in, no joy; try to reset password on this different browser (I might add, I did get a new Change password Token number off this attempt), but no joy; clear cookies and restart, then attempt to log in, no joy!

WHAT THE F?!?!?!?

I initially thought it may have been a 24hr block due to password change attempts? But now not so sure... I've also tried logging in on via Machine A in a VM from a different O/S to see if it may have something to do with it - but again no joy - this environment had NOT been logged in to FB before...

Thoughts????

mrzaggy · 2 years ago
MAC Address being flagged?

But on the flip side, I was able to create a back up profile in a different VM on this machine.....

teeray · 2 years ago
I'm just waiting for all my relatives to breathlessly call me saying "I THINK MY FACEBOOK WAS HACKED".
basch · 2 years ago
Already got a call from someone who regularly gets caught in unprompted xfinity password reset loops saying “it’s happening again.”

Inexcusable to catch people in a reset loop during a login outage, and not confirm which password is the current one.

ffsm8 · 2 years ago
Same... This was literally the first time I wanted to use my Facebook account in years, so I changed my password twice before I realized that there must be an issue on their end.

I just wanted to use their oauth login to buy a jonsbo n3 case from AliExpress. Sadge

CivBase · 2 years ago
Logging everyone out across all platforms and telling them to just try logging back in sounds like an excellent way to DDOS yourself.

This should be a fun postmortem.

bbarn · 2 years ago
Yeah, and they've dug themselves a hole with having near permanent login persistence. I can't remember the last time on a computer I use frequently that I've had to log in.
jeltz · 2 years ago
Yeah, same thing happened to me so will likely need to reset my password again once Facebook is up again.
the42thdoctor · 2 years ago
That was my cue to check hacker news
mrzaggy · 2 years ago
Yup - same here...

I fear my initial attempts to 'reset my password' and getting that same security(at)facebookmail.com email with the SAME reset code, have not helped me - if only I'd been asleep, I probably could have slept through it.

At least Insty came back up - though to be honest I cant remember that password either and now terrified to try and recover or change that now too!

dimask · 2 years ago
Me too, after the first couple of "is it down" websites I tried failed to load.

HN never fails us when a big website is down.

I was also almost ready to reset my password.

getlawgdon · 2 years ago
I have exactly this issue. Been frantically jumping around this morning dealing with perceived security issues and have no idea where my many FB resets led to.
bertan14 · 2 years ago
MY SERVICE IS DOWN.... I did exactly as you.. I have no clue now

Dead Comment

jonnycomputer · 2 years ago
It logged me out and told me that my credentials were incorrect; I thought my credentials had been stolen, so I'm kinda personally glad that it seems to be happening to a lot of other people too. I know that's a bit selfish, but :shrug:
mfrommil · 2 years ago
A much better UX would be clear error messaging informing users that the service is down and there is no problem with their individual account.

This would prevent people from panicking they've been hacked and/or unnecessarily resetting their password.

mns · 2 years ago
There are quite some harsh comments here below. You can't plan for every possible failure point, who knows what part of a system/infra out of everything that they have went down and triggered this behaviour. Some things you just can't catch/predict. Especially in huge systems like theirs. I would expect people here to understand things like these and not just call people names for something like this, we all know things seem simple/clear from the outside, but the job of debugging and fixing something like this take quite some effort.
Kalium · 2 years ago
You are absolutely correct. That would be a much better experience.

That said, getting there strikes me as pretty challenging. Automatically detecting a down state is difficult and any detection is inevitably both error-prone and only works for things people have thought of to check for. The more complex the systems in question, the greater the odds of things going haywire. At Meta's scale, that is likely to be nearly a daily event.

The obvious way to avoid those issues is a manual process. Problem there tends to be that the same service disruptions also tend to disrupt manual processes.

So you're right, but also I strongly suspect it's a much more difficult problem than it sounds like on the surface.

boring_twenties · 2 years ago
Well you can't expect to hire engineers with half a brain for the pitiful compensation Meta offers, can you?
eurekin · 2 years ago
There was for a brief moment. I got that once
jonnycomputer · 2 years ago
It would be a better UX, but, depending on the outage, that might be a really hard behavior to guarantee.
barbazoo · 2 years ago
Not the worst thing that a bunch of Facebook users are resetting their passwords.
KMag · 2 years ago
Yea, the wife came to me in a bit of a panic that her Facebook account got hacked. I tried logging in to FB to check if I had been unfriended, and I also got errors indicating my password was incorrect. My FB password is 96 bits from /dev/urandom in a GPG-based password manager I wrote for myself a couple decades ago. So, no my password wasn't wrong, and I'm not a big enough target for someone to put enough effort into figuring out how to snarf up my password data and crack my GPG passphrase.

Anyway, when FB thought my password was wrong I calmed way down. I thought maybe FB corrupted their password DB or something, so I just tried to reset my password, got into an odd workflow loop, and then quacked "downdetector facebook".

sigil · 2 years ago
> My FB password is 96 bits from /dev/urandom in a GPG-based password manager I wrote for myself a couple decades ago.

We have the same approach to password management!

NikolaNovak · 2 years ago
Yes. My spidey sense went off and I told my work I'll be off for an hour while I redo all my passwords... might still do that but glad to know it's not necessarily me getting hacked.
pratnala · 2 years ago
don't bother. fb's forgot password flow is broken too.
smcl · 2 years ago
I called out some comment for being racist a little earlier (yeah I know, just report and move on...) and figured they'd managed to pwn my account somehow. Good to know it's not just me.
fishnchips · 2 years ago
Strictly speaking, just because there's an outage does not mean you're not pwn'ed.

Dead Comment

kortex · 2 years ago
Same same. I went through the password reset flow (I was overdue anyways), it never sent anything to my SMS, so I did it again with email, reset the password and went to log in with the new password, "Incorrect password" error. Old password, also incorrect.

Didn't help that I had just posted a lukewarm spicy take on how linguistic prescriptivism is BS.

All the while the website felt like it was unstable, hard to describe, but it felt like it was bouncing around between URLs too much and reloading a lot.

Definitely feels like a botched update on their end.

E: Instagram is misbehaving as well, banner loads but big "Something is wrong" error on the feed.

E: now youtube has "Something went wrong" - WTF. I can't believe I'm saying this, but thank goodness for reddit and X[itter]???

E: interesting, seeing a big spike across multiple platforms on downdetector, including AWS: https://downdetector.com/status/aws-amazon-web-services/ I'm not able to log in right now, but that could be PEBCAK, I have too many saved IDs and I don't want to fail2ban myself

pratnala · 2 years ago
The password reset flow was broken too. And I got logged out on every device. My friend who works there said they can't login either.
samaritano · 2 years ago
downdetector reports has gone down but to me is still bugged out, been catching a livestream on youtube all along though, meta stocks are back up from the dip so I take it some regions are restored to normality
cratermoon · 2 years ago
Discord is also having issues.
101008 · 2 years ago
I panicked the same. Even when I tried to recover my password, it said my email address wasn't associated to any account. I thought I lost it forever
marcosdumay · 2 years ago
That's the natural endgame of the "user-facing services must not stop, if something they depend upon stops, they must only degrade" philosophy.
Twirrim · 2 years ago
Likewise, started password reset process that won't complete, asked my wife to double check my account wasn't compromised and posting cryptocurency crap or somesuch.
alkonaut · 2 years ago
Yeah your product should NEVER confuse an auth service being down with a failed auth. This is really terrible by FB.
suyash · 2 years ago
me too, also Instagram, could be that Facebook got hacked ?
suyash · 2 years ago
Pro tip: in chase meta actually got hacked, it would be good idea to not use that password on any other websites, change them immediately.
MyFirstSass · 2 years ago
Same, thought all of my friends were getting spammed and i'll look like a "boomer" who got phished.

Then i remembered i have a very long and secure password, then immediately panicked about someone having access to my Gmail.

The sense of security is more brittle than i thought.

sandspar · 2 years ago
On a psychological note, I think the threat detection part of our brain doesn't always notify our conscious thought that it's actively monitoring for threats. I've often noticed that when I'm carefully handling a hot frying pan then my ringing phone is more likely to startle me than usual.
srvmshr · 2 years ago
I have an active Instagram account. Today, coincidentally was the first time I thought of promoting my best few posts as an ad to improve reach to fellow photographers. Bad luck! The app now seems to be back online but my ads which were supposed to run for 48-72 hours, show <Disabled>. And there is a new "Pay Now" link, even though it was paid for and seems ad-spend is already showing it used some of that payment.

As an individual, this is pretty confusing. I don't have much to lose. I am glad I spent 10-15 bucks on favorite 2-3 posts only. I can imagine many others to be more affected. What is normally to be expected for SME users? Does Meta resume the ads automatically? Do they make good for the lost time since the clock seems to be ticking -- although no one saw any impression.

Edit: I submitted a ticket to Instagram Help and they responded by asking for a screencast video. The first time I sent, the video bounced. I have re-sent this by trimming the video.

Out of curiosity I want to know firsthand how Meta handles the small customers.

MandieD · 2 years ago
Special bonus: today is Super Tuesday, which is when a large number of US states have their primary elections.
apetresc · 2 years ago
Neither party's primary is in the slightest way contested, so what do you think the "bonus" is here?
ak_111 · 2 years ago
If we get huge surge of "Uncommitted" votes to Biden as we did in Michigan, you can argue this can cause significant pressure on him and might even lead to his ousting...
bbarn · 2 years ago
I've seen this posted everywhere like it's some grand conspiracy.

What is the impact of it being Super Tuesday? Are people worried they can't vote without social media?

onlyrealcuzzo · 2 years ago
Social Media is very successful in Get Out the Vote campaigns for younger people - who typically vote for one party disproportionately.
lxgr · 2 years ago
Facebook and other social media platforms have played somewhat of a role in the 2016 election, for example. Meta has been under a lot of pressure for that, compare e.g. this recent Instagram change: https://about.instagram.com/blog/announcements/continuing-ou...
InitialLastName · 2 years ago
I can't get into the US-local conspiracy-mongering, but I'm not sure how you've missed social media becoming a hot-spot for election-day information/misinformation?

Shutting down social media has gone to the top of the list for regimes either "attempting to fabricate positive election results" or "attempting to combat the spread of misinformation about elections".

More sympathetically, for better or for worse (definitely the latter) there will be people trying to look up election information ("what are my local polling hours", "who is on my ballot") on social media websites, who will now not be able to be guided to the correct information.

kortex · 2 years ago
Messenger is often used to coordinate logistics among friends. I would not be surprised if a Meta disruption lead to at least momentary confusion if someone was planning on carpooling. Not to mention it messes with "get out the vote" posting.

I'm not very conspiracy-minded but this does smell a little weird.

At the very least, "there's a big event in the country of one of our biggest userbases, maybe hold off on risky deploys until tomorrow"

mynameishere · 2 years ago
The primaries are pretty well decided, so I suppose the conspiracy would be "trial run".
ak_111 · 2 years ago
Actually I can see this morphing into a Russia interference conspiracy.

If the main social media used by liberals goes down, while the one used by Trump (forgot what it's called) stays up, surely that is an advantage for him?

nso · 2 years ago
I'm sure people will be able to vote even if their vacation pictures are unavailable
Ancapistani · 2 years ago
Not relevant to the primary election in any way, but this has been more than annoying to me already.

We have a small livestock operation, and won an online auction late last night for a pig about four hours away. Facebook was the only listed means of contacting the person, and we were planning on driving to pick it up this morning.

Now I get to re-arrange my day today to deal with that, and will probably have to take a PTO day from work to drive there later in the week.

Real businesses are in fact impacted by Facebook being down - including those not based around Facebook and that you might never expect.

Brian_K_White · 2 years ago
Did they imply anything as ridiculous as needing facebook to vote? The implication is merely that everyone would otherwise have been talking about voting and the results, ie, simply higher than normal traffic. (I'm not sure it would be that much but that was the reasonable interpretation of the comment.)
jeltz · 2 years ago
I agree with the first part but the second is taking it too far. Plenty of people use Facebook messenger for communication about semi-important things. Not seeing how that would affect the primaries but it is not jsut for vacation pictures.
1970-01-01 · 2 years ago
Which one this time - BGP or DNS?

Either way, global GDP should be up :)

kordlessagain · 2 years ago
My sense is that it's BGP.

Deleted Comment

ototot · 2 years ago
Given the range. Doesn't look like DNS
marcosdumay · 2 years ago
So... No way it could be DNS?
gene31337 · 2 years ago
I'm betting memory.
dadver · 2 years ago
Sweden, logged out from all devices, Google authentication to reset password results in error. Authorization codes sent via SMS to reset passwords seems non-responsive. Authorization codes to reset password sent by e-mail works, but setting passwords results in set password page two times in a row and after second try "An unexpected error occurred. Please try logging in again."
jph · 2 years ago
Facebook is throwing up the sign in page, as if your sign in has failed or someone's logged you out. Instagram and Messenger also seem to be affected.
jzw8833 · 2 years ago
Yes, just checked Instagram and it says “Couldn’t refresh feed” and is generally not loading. Messenger has also logged me out.
subtra3t · 2 years ago
For some reason I couldn't browse the home page but I could still access my messages (website).
TheOtherHobbes · 2 years ago
And Threads.
coffedean · 2 years ago
And the Quest headsets.