I too am in the wilderness and Starlink is my only broadband, but I was smart enough to keep my 1.5Mbps WISP connection (radio mounted 170' up one of my redwoods) so I'm still in good shape for basic communication, even some streaming and videoconferencing, though those aren't great this morning.
Also, after my Starlink terminal rebooted a few times on its own over the last hour or so, things are looking good again.
Digression ... you must be using "wilderness" in it's colloquial form, because technically, if he's in a cabin he can't be in wilderness. Federally designated wilderness areas don't allow permanent human habitation.
Furthermore, the planet we are on cannot possibly be in space. Space is mostly empty, has no breathable atmosphere, there are no trees or grocery stores there, etc.; Earth is quite the opposite.
It’s like when our vendor’s tech support number tells you you’re 300th in queue (or busy cuz their lines are saturated) instead of the usual 2nd or 3rd.
I broadly agree with what you're saying, but, that's not the issue here.
They don't even have a dedicated status/outage page, afaik.
The website being down is a more classic problem. The outage probably increased traffic to their website by 1000x if not more and the infrastructure for the website simply couldn't cope.
Good lesson on keeping your status infrastructure simple and on something which is highly scalable.
Having a CDN where the main page of their site was 99% cached globally would have probably mitigated this issue.
How would you host a status page on Starlink? Are there web servers that are only connected to a satellite connection in house or something? Or is this just speculation that's how their marketing website works.
Starlink has servers for the website and subscriber sites. Importantly, they have servers for controlling the satellite network. They also have multiple gateways for satellites to connect to the network.
A problem in their network would take down the sites, and maybe the control plane for satellites.
Starlink still has infrastructure outside the satellites to run normal company stuff. If they screwed up something in their core routing system, or DNS, etc it could affect everything.
https://radar.cloudflare.com/as14593?dateRange=1d shows a sharp drop in traffic, but no BGP changes before the drop. We can see BGP changes after the drop, so maybe some datacenter/control plane thing.
Curious as to what could cause a global outage like that. The system consists of birds in the air and individual ground stations. There's no big choke point that I'm aware of that could cause the whole shebang to go dark. Am I missing something here?
BGP is the canonical way to take your entire global network offline within seconds. Glancing at BGP looking glasses, Starlink's prefixes seem to still be announced, but there could still be an accidental blackhole or routing loop within their AS, or something broken in one of their transit providers.
No idea if that's what's going on, but routing protocols are one of a few effectively global control planes that can go wrong very quickly like this.
If it were BGP/routing, you would think we'd be able to still get a signal and the modem would think it's healthy (although maybe not if the issue prevented us from obtaining our public IP), we just wouldn't be able to route to any dst. In the current case we don't have a signal (orange light on the modem)
A network of satellites gives you entirely new and exciting ways of taking your network offline, such as bricking them with a firmware update and no way to actually get up there and fix it.
it's pretty amazing the amount of damage a BGP oopsie can do. Also, you can fit pretty much all the BGP admins for the entire Internet in one large room.
I rebooted my terminal and I can’t tell for sure if it sees any satellites. It looks like it doesn't.
It says it didn’t, and it says the “which way is down” thing hasn’t converged. Occasionally, the signal to noise ratio light in the app goes gray which means < 3.
It also rebooted itself.
Before the first reboot, 30% of pings went through. It’s almost like the azimuth or some other timely but cached data was corrupted.
I'm hardly an expert, but it seems like usually stuff like this happens to major backbones because of a bad configuration being rolled out too fast. Maybe the pushed a bad DNS/BGP/<insert networking acronym here> update to every satellite and/or ground station near simultaneously.
The fleet of satellites is still managed centrally, even if the physical infrastructure is distributed globally to be able to see satellites in different hemispheres, at some point they are very likely managing these from a single control plane
Cyberattack. Given the usage in active wars, it's a big juicy target. There's precedent too; Russia hacked Viasat at the start of the Ukraine war. They actually bricked user terminals in that attack.
They still have tons of ground-based infrastructure that has all the regular failure modes. I suspect the satellites also route IP traffic statically, and they only do so to a fixed set of ground gateways all operated by Starlink (compared to e.g. Iridium, which makes in-space switching decisions and can at least continue ongoing phone calls between two devices without actually needing a reachable ground gateway).
National telcos manage to take themselves out countrywide from time to time too, so I'd assume the same usual suspects apply: Routing problems, botched global configuration changes etc.
If anything, due to centralization, ubiquitous and effectively free connectivity, and centralization/automation of configuration changes, the blast radius of any given error or malicious action has probably been steadily increasing over the years.
Same things have been said of eg the massive Facebook years ago. It was DNS, which affected service discovery across the whole org. The massive Roblox outage was also caused by service discovery failure (Consul). If your machines don't know which other machines to talk to, shit goes bad fast.
Almost always. Most people look at global networks like black magic and don’t realize this.
For most large businesses, 90%+ of major network events are caused by internally-driven network config changes.
Depending on how far along the business is towards automation, a proportion of the 90% can be attributed to a human going “off script” - I.e.: making a change that had not been reviewed.
I think the network putting out warning labels to prevent loops is probably worse off than the place that just forgot to leave some form of loop prevention enabled.
I was browsing Reddit and X and a lot of people saying that their base stations couldn’t even find a satellite. I find that pretty odd and can’t make sense out of it if it was simply a network problem.
You would expect to be able to at least connect to a satellite unless the satellite has no Internet connection so it does not make itself available to be connected to you.
Anyone have any deeper knowledge about this for the curious?
I'd bet on a bad update or configuration change. (likely one that prevents the affected systems from reaching the internet and being automatically rolled back)
I mean, why do systems go down at all? a lot of big outages are simple misconfiguration or cascading failure from what seemed like small changes. It's rarely due to the physical constraints of the world
I'm posting this from my Starlink connection, been working fine here. I'm close to the arctic circle so maybe the polar orbiting satellites are not affected
What a great writing prompt! Would love to read read the rest of this as a short story. Any takers?
Edit: Wow, that’s the fastest any of my HN posts have ever been buried. Still, as an old guy, I have to say it’s pretty awesome to be living in a future where polar satellites can keep someone in touch with the rest of civilization.
So, if (just spitballing here) Russia were to have wanted to take out Starlink in order to handicap Ukrainian operations, and they didn't want to do something as visible as actually taking out the satellites, what would be the most likely way for them to have done it? And how would we be able to check if that's what happened?
No idea if this is plausible at all, just raising the question for anyone who knows more about this system than I do.
I would still bet against the Russians taking down all of Starlink. If they had a zero day that could do that, they'd save it for just before a big war.
But Russian low-intensity hybrid warfare, including infiltration and sabotage, is very real. Particularly in Europe and especially so in eastern Europe. There's a lot of nasty things going on in border areas that isn't widely reported.
Things get patched over time, Nato has their own comms systems whilst Ukraine relies on this third party, and the US just accepted the EU funded weapons sale so their Trump advantage seems to have disappeared with his patience in P's promises.
Thus using the weakness to launch a coordinated summer offensive now for a decisive breakthrough before US weapons supplies reaches full speed again seems like a reasonably good choice.
Conquering Ukraine is the key for restoring the USSR, Belarus is already under the yoke and Khz,etc won't dare resist after Ukr falls. P sees a victory here as moving the linchpin that makes him immortal in historybooks.
> what would be the most likely way for them to have done it?
Ensure Starlink hires an agent. Establish skills / access permissions. Push an update which bricks both the update system and the running system, so the satellites become useless.
It's essentially impossible for anyone to disable the Starlink network by "taking out satellites" through physical means. There aren't enough anti satellite missiles in the world to make a dent, it would cost trillions to manufacture enough, and SpaceX can launch satellites faster than anyone could manufacture interceptors anyway. And Kessler syndrome is impossible at the altitudes Starlink uses.
Cyberattack is the only plausible way to take down Starlink.
The anti-satellite devices could be deployed in the same manner as the Starlink satellites. And they wouldn't need communications equipment, so they could be lighter and cheaper to launch. And you really wouldn't need to take them all out, just enough to make communication unreliable.
Starlink launches reduce costs by launching a bunch of satellites with similar orbits on the same vehicle, replacing one or a few satellites is going to cost a lot more per satellite. So just disrupting the network is a lot cheaper than fixing it.
Thought the cheapest is still probably paying an existing employee to break some stuff.
If they had that capability, they could presumably do it any time, and then Starlink would recover at some point.
So it would make sense to combine this with some other offensive action where hampering communication would help them. Otherwise they are just poking at something just to do it... unless this were a more serious attack with long-term consequences.
I'd wonder about someone targeting Musk seeing as Starlink is one of his ventures that seems to be running well up to now while his other companies have issues. Having typed that out it seems like a spy movie plot, but that seems to be the world we're living in recently.
Yeah, Russia maanged to knock Viasat offline with hacking during the initial invasion. The original donated Starlinks were supposed to replace Viasat and SpaceX was surprised that they ended up being used on the front lines. Russia has certainly been trying to knock them out ever since.
If my memory is accurate, that didn't knock Viasat offline, it only made most of Viasat's customers unable to connect, but the network was just fine, only user routers were broken.
Starlink has been supporting Russia over Ukraine (congressional investigation into this). So I would bet against Russia, but you never know. Could be rouge. Anonymous has promised in the past to mess with Musk... could be that. Well funded and some crazy good hackers.
In terms of actual actions, Starlink has done a lot more for Ukraine than almost anyone else, since early on. Musk says stuff from time to time in order to try to keep Putin from noticing this, and pushing him out of a window or something. But his actions are that he let Ukraine use Starlink for over a year for free, and they still use it widely today.
Yea it's funny because I was JUST thinking about how important StarLink has been for us here in rural Oregon. When it goes out, it's even more stark — but it hasn't happened in a very long time here.
Interestingly, this webpage doesn't load due to "no healthy upstream" and there's no status.starlink.com . That's.. quite the outage! Interested to hear what went wrong.
Readit News
[1] https://www.rfc-editor.org/rfc/rfc1149 [2] https://archive.google/pigeonrank/
Also, after my Starlink terminal rebooted a few times on its own over the last hour or so, things are looking good again.
Deleted Comment
Deleted Comment
You just hang up cuz you know it’s everyone.
They don't even have a dedicated status/outage page, afaik.
The website being down is a more classic problem. The outage probably increased traffic to their website by 1000x if not more and the infrastructure for the website simply couldn't cope.
Good lesson on keeping your status infrastructure simple and on something which is highly scalable.
Having a CDN where the main page of their site was 99% cached globally would have probably mitigated this issue.
A problem in their network would take down the sites, and maybe the control plane for satellites.
Deleted Comment
No idea if that's what's going on, but routing protocols are one of a few effectively global control planes that can go wrong very quickly like this.
It says it didn’t, and it says the “which way is down” thing hasn’t converged. Occasionally, the signal to noise ratio light in the app goes gray which means < 3.
It also rebooted itself.
Before the first reboot, 30% of pings went through. It’s almost like the azimuth or some other timely but cached data was corrupted.
It's not like you can have a tech go out and reboot it or reinstall windows.
This is hybrid warfare, not a coincidence.
Russia needs to learn some manners, the hard way.
National telcos manage to take themselves out countrywide from time to time too, so I'd assume the same usual suspects apply: Routing problems, botched global configuration changes etc.
If anything, due to centralization, ubiquitous and effectively free connectivity, and centralization/automation of configuration changes, the blast radius of any given error or malicious action has probably been steadily increasing over the years.
For most large businesses, 90%+ of major network events are caused by internally-driven network config changes.
Depending on how far along the business is towards automation, a proportion of the 90% can be attributed to a human going “off script” - I.e.: making a change that had not been reviewed.
Additional points for a user to cause the loop by plugging an rj-45 where they weren't supposed to, despite no warning or label telling them not to!
I love when devs talk about networking.
You would expect to be able to at least connect to a satellite unless the satellite has no Internet connection so it does not make itself available to be connected to you.
Anyone have any deeper knowledge about this for the curious?
Deleted Comment
[1] https://www.pcmag.com/news/spacexs-starlink-hit-by-major-out...
Dead Comment
Edit: Wow, that’s the fastest any of my HN posts have ever been buried. Still, as an old guy, I have to say it’s pretty awesome to be living in a future where polar satellites can keep someone in touch with the rest of civilization.
Deleted Comment
No idea if this is plausible at all, just raising the question for anyone who knows more about this system than I do.
But Russian low-intensity hybrid warfare, including infiltration and sabotage, is very real. Particularly in Europe and especially so in eastern Europe. There's a lot of nasty things going on in border areas that isn't widely reported.
Thus using the weakness to launch a coordinated summer offensive now for a decisive breakthrough before US weapons supplies reaches full speed again seems like a reasonably good choice.
Conquering Ukraine is the key for restoring the USSR, Belarus is already under the yoke and Khz,etc won't dare resist after Ukr falls. P sees a victory here as moving the linchpin that makes him immortal in historybooks.
Ensure Starlink hires an agent. Establish skills / access permissions. Push an update which bricks both the update system and the running system, so the satellites become useless.
Potentially you could burn the constellation with reentry.
Cyberattack is the only plausible way to take down Starlink.
Starlink launches reduce costs by launching a bunch of satellites with similar orbits on the same vehicle, replacing one or a few satellites is going to cost a lot more per satellite. So just disrupting the network is a lot cheaper than fixing it.
Thought the cheapest is still probably paying an existing employee to break some stuff.
Few big nukes are enough to take out all satellites.
User terminals rely on GNSS for timing and position and I wouldn't be surprised if the satellites did too.
So it would make sense to combine this with some other offensive action where hampering communication would help them. Otherwise they are just poking at something just to do it... unless this were a more serious attack with long-term consequences.
Either exploit existing employees or take years to seed an agent into the company.
State level persistent attacks are well funded and have long timelines.
If not and this was a cyber attack the this an attack on the USA.
Deleted Comment
Deleted Comment