Really interesting to me that none of the commentators I've seen in the press have even hinted that maybe an OS that requires frequent security patches shouldn't be used for infrastructure in the first place. For just one example, I've seen photos of BSODs on airport monitors that show flight lists -- why aren't those built on Linux or even OpenBSD?
Security is not a feature that can be layered on. It has to be built in. We now have an entire industry dedicated to trying to layer security onto Windows -- but it still doesn't work.
The vendor who makes the software has always written for Windows (or in reality, wrote for either DOS or OS/2 then transitioned to NT4). History, momentum, familiarity, cost, and ease of support all are factors (among others, I'm sure).
Security is a process, not a product.
And yes, distros require frequent updates, though more to your point, you can limit the scope of installed software. I'm sure airport displays don't need MPEG2, VP1 and so on codecs, for instance.
It's also important to remember that there is a lot of 'garageware' out there with these specialized systems. Want SAML/OIDC support? We only support LDAP over cleartext, or Active Directory at best. Want the latest and greatest version of Apache Tomcat? Sorry, the vendor doesn't know how to troubleshoot either, so they only "support" a three year old vulnerable version.
Ran into that more than a few times.
Given the hypothesis of what caused the BSOD with Crowdstrike (NUL pointer), using a safe language would have been appropriate -- it's fairly easy in this case to lay the blame with CS.
Microsoft supplies the shotgun. It's the vendors responsibility to point it away from themselves.
> I'm sure airport displays don't need MPEG2, VP1 and so on codecs, for instance.
They don't, until the day the airport managers are approached by an advertising company waving the wads of cash the airport could be 'earning' if only they let "AdCo" display, in the top 1/4 of each screen, a video advertising loop. At which point, those displays need the codecs for "AdCo's" video ads.
The vendor who makes the software has always written for Windows (or in reality, wrote for either DOS or OS/2 then transitioned to NT4). History, momentum, familiarity, cost, and ease of support all are factors (among others, I'm sure)...
That's starting the argument with "weight loss is about overall diet process, not individual choices" and then hopping to "ice cream for dinner is good 'cause it's convenient and I like it".
The statement "Security is a process, not a product." means you avoid shitty choices everywhere, not you make whatever choices are convenient, try to patch the holes with a ... product ... and also add an extra process to deal with the failures of that product.
> an OS that requires frequent security patches
> Security is not a feature that can be layered on. It has to be built in
This is a common misunderstanding, an OS that receives frequent security updates is a very good thing. That means attention is being paid to issues being raised, and risks are being mitigated. Security is not a 'checkbox' it's more of a neverending process because the environment is always in a state of flux.
So to flip it, if an OS is not receiving updates, or not being updated frequently, that's not great.
What you want is updates that don't destabilize an OS, and behind that is a huge history and layers of decisions at each 'shop' that runs these machines.
Security is meant to be in layers and needs to be built in.
> but it still doesn't work.
It does work because the 'scene' has been silent for so long, but what we as humans notice is the incident where it didn't.
This sort of thinking is one of the main problems with the industry, in my opinion.
We've got a bunch of computers that mostly don't make mistakes at the hardware layer. On top of that, we can write any programs we want. Even though the halting problem exists, and is true for arbitrary programs, we know how to prove all sorts of useful security properties over restricted sets of of programs.
Any software security pitch that starts with "when the software starts acting outside of its spec, we have the system ..." is nonsense. In practice, "acting outside its spec" is functionally equivalent to "suffers a security breach".
Ideally, you'd use an operating system that has frequent updates that expand functionality, that is regularly audited for security problems, and that only rarely needs to ship a security patch. OpenBSD comes to mind.
If software has frequent security updates over a long period of time, that implies that the authors of the system will continue to repeat the mistakes that led to the vulnerabilities in the first place.
Remote update is a nice way of saying remote code execution. It is really really hard to ensure that only the entity that you want to update your system, can update your system, when facing a state-funded adversary. Sometimes that state adversary might even work in concert with your OS vendor.
Frequent security updates are a good thing, frequent security auto-updates are not, at least when it comes to situations like this. Technology that runs 24 hour services such as airports and train stations should not be updated automatically just like that, because all software updates have high potential to break or even brick something. Automation is convenient and does saves money which would have to be paid for additional labor to do manual updates, but in cases like this, it should be understood that it's better not to break the airport and roll-out update manually in stages.
Airport staff need to be able to support them. Not HN types.
Most people know how to use a windows computer.
Most IT desktop support knows how to use and manage windows. Even building facilities folks can help support them.
Microsoft makes it easy to manage a fleet of computers. They also provide first party (along with thousands of 3rd parties) training and certifications for it.
Most people don't know how to tell what's going wrong with a windows computer
A windows computer that relies on cloud services, as an increasing and often nonsensical subset of the functionality on one does, can often only be fixed by Microsoft directly
Microsoft intervenes directly and spends billions of dollars annually on anticompetitive tactics to ensure that other options are not considered by businesses
And with this monopoly, it has shielded itself from having to compete on even crucial dimensions like reliability, maintainability, or security
I know of a very small airport where what is displayed over the HDMI part is essentially Firefox at fullscreen with powersaving disabled so the screen does not blank. Some of them are Intel NUC, some of them are Raspberry Pi with HSM in a box. These devices basically "boot to Firefox" with relevant credentials read off internal TPM/HSM.
Those among airport staff who do not know how to use a computer at all can get them working by just plugging them in.
> Most people know how to use a windows computer.
They know enough to open a browser.
> Most IT desktop support knows how to use and manage windows.
They know how to cope with Windows, at best.
> Finding someone who knows a BSD is not easy.
BSD is everywhere and in far more places than Windows, like almost every car sold after 2014. But you never ever see BSD because it's already-working with nothing for the end customer to do.
For many CTO/CISO it is more important to have a good target to shift responsibility when things go awry than to have a reliable/secure system. A Big Brand is a good target, an open-source project like OpenBSD is not. I doubt any CTO will be fired for choosing Widnows+CrowdStrike (instead of Linux/BSD) despite many million losses.
"Nobody ever gets fired for buying IBM" is as true as ever at least in the corporate world.
> I doubt any CTO will be fired for choosing Widnows+CrowdStrike (instead of Linux/BSD)
I was personally involved in a meeting where my firm's leadership advised a client who did fire their CTO and a bunch of other people for what was ultimately putting what they thought were smart career moves over their actual responsibilities.
Unfortunately, as you did just point out, the CEO, other execs, and board are often just as incompetent as the CTO/CISO who have such shit-brained mindset.
Or don't use an OS at all. We need to think about minimizing the use of software in critical infrastructure. If that means less efficiency because you have to be near something to maintain it then so be it. That would be good for jobs anyway.
Even unikernel applications have an OS compiled into the application. It's necessary to initialize the hardware it's running on, including the CPU and GPU and storage.
I suppose you could build it as a UEFI module that relies on the UEFI firmware to initialize the hardware but then you get a text only interface. But then the UEFI is the OS.
But this outage was not an OS problem. It was an application bug that used invalid pointers. If it was a unikernel it still would have crashed.
To pick on your airport example a bit… all of the times I’ve gotten to enjoy a busted in-seat entertainment system, I’ve found myself staring at a stuck Linux boot process. This goes well beyond the OS.
Those sorts of things just need to boot to a web browser in full screen with some watchdog software in the background, launching from a read only disk (or network image). Get a problem, just unplug it and plug it back in. Make it POE based so you can easily do it automatically, stick them on a couple of distros (maybe even half on bsd, half on linux, half using chrome, half on firefox)
A web browser is an unbelievably complex piece of software. So complex that there are now only two. And also so complex that there are weekly updates because there's so many security holes.
I'm sure we've all heard the phrase "We're a Windows shop" in some variation.
I understand the reasons for it, and why large, billion dollar companies try to create some sort of efficiency by centralising on one "vendor", but, then this happens.
I don't know how to fix the problem of following "Industry Trends" when every layer above me in the organisation is telling me not to spend the time (money) to investigate alternative software choices which don't fit into their nice box.
I read the T&C of this CrowdStroke garbage and they have the usual blurb about not using it in critical industry. Maybe we just charge & arrest the people that put it there and this checkbox-software mess stops real quick.
from the reporting so far, no one has died as a result of the Crowdstrike botch. For my money, that sounds like it's not being used in 'critical industry'.
/unset
There were several 911 service outages included in the news yesterday, so I would definitely say agree those fall into the category. I haven't seen how many hospitals were deeply affected; I know there were several reports of facilities that were deferring any elective procedures.
>Really interesting to me that none of the commentators I've seen in the press have even hinted that maybe an OS that requires frequent security patches shouldn't be used for infrastructure in the first place.
Nobody's commenting on that because it's the wrong thing to focus on.
1) This fuckup was on CrowdStrike's Falcon tool (basically a rootkit) bricking Windows due to a bad kernel driver they pushed out without proper hygiene, not on Windows's security patches being bad.
2) Linux also needs to get patches all the time to be secure (remember XZ?) It's not just magically secure by default because of the chubby penguin but is only as secure as it's most vulnerable component, and XZ proved it has a lot of components. I'd be scared if a long period goes by and I see no security patches being pushed to my OS. Modern software is complex and vulnerabilities are everywhere. No OS is ever bug-free and fully bullet proof in order to believe it can be secure without regular patches. Other than TempleOS of course.
The lesson is whichever OS you use, don't surrender your security to a single third party vendor who you now have to trust with the keys of your kingdom as that now becomes your single point of failure. Or if you do be sure you can sue them for the damages.
1) While CrowdStrike can be run on Linux it is less of a risk to use Linux without it than Windows. I don't think most Linux/BSD boxes would benefit from it. It could be useful for a Linux with remotely accessible software of questionable quality (or a desktop working with untrusted files) but this should not be the case for any critical system.
2) There is a difference between auto-updates (common in Windows world) and updates triggered manually only when it is necessary (and after testing in non-prod environment). Also while Linux is far from being bug-free, remotely exploitable vulnerabilities are rare.
every year multiple times per year there's reports of Microsoft Windows systems having either mass downtime or exploitation.... it's kind of amazing that critical systems would rely on something that causes so much frustration on a regular basis.... I've been running systems under Linux and Unix for decades and never had any down time... so I don't know I mean it's nice to know that Linux is pretty solid and always has been the worst that's ever happened has been like a process that might go down during an upgrade, but never the whole system.
Linux is vulnerable too (but not as vulnerable as windows of course) it’s just not targeted by hackers because it’s market share is so small. That wouldn’t be the case if, say, half of all users ran Linux.
I've never managed linux IT departments--how well are the management tools compared to what Microsoft offers such as tooling for managing thousands of computers across hundreds of offices.
Layering is absolutely possible, but more at the network layer than the individual computer layer.
Minimal software and OS running on linux as a layer between any windows/whatever and internet connectivity. Minimize and control the exact information that gets to the less hardened and trustworthy/complicated computers
I'm sorry but even Linux requires frequent security updates due it's large ecosystem of dependencies. It's more or less required by every cyber security standard to update them just like windows.
On the other hand OpenBSD doesn't require very frequent patching assuming a default install which comes with batteries included. For a web server there's just one relevant patch since April for 7.5: https://www.openbsd.org/errata75.html
I agree that all dependencies should be treated as attack surface. For that reason, systems for which dependencies can be more tightly controlled are inherently more secure than ones for which they can't. The monolithic and opaque nature of windows and other proprietary software makes them harder to minimize risk about in this way
> why aren't those built on Linux or even OpenBSD?
Because in the non-Silicon-Valley world of software, if you pick Linux and it has issues, fingers will get pointed at you. If you pick Windows and it has issues, fingers will get pointed at Microsoft.
This sort of emergent behavior is a feature, not a bug.
Operating systems that don't require frequent security patches aren't profitable.
Anyway, this is the step of late-phase capitalism that comes after enshittification. Ghost in the Shell 2045 calls it "sustainable war". I'd link to an article, but they're all full of spoilers in the first paragraph.
It probably suffices to say that the series refers to it as capitalism in its most elegant form: It is an economic device that can continue to function without any external inputs, and it has some sort of self-regulatory property that means the collateral damage it causes is just below the threshold where society collapses.
In the case of Cloud Strike, the body count is low enough, and plausible deniability is low enough that the government can get away with not jailing anyone.
Instead, the event will increase the money spent on security theater, and probably lead to a new regulatory framework that leads to yet-another layer of mandatory buggy security crapware (which Cloud Strike apparently is).
In turn, that'll lower the margins of anyone that uses computers in the US by something like 0.1%, and that wealth will be transferred into the industry segment responsible for the debacle in the first place. Ideally, the next layer of garbage will have a bigger blast radius, allowing the computer security complex to siphon additional margins.
I don't think CS type endpoint protection is appropriate for a lot of cases where it's used. However:
Consider the reasons people need this endlessly updated layer of garbage, as you put it. The constant evolution of 0-days and ransomware.
I'm a developer, and also a sysadmin. Do you think I love keeping servers up to the latest versions of every package where a security notice shows up, and then patching whatever that breaks in my code? I get paid for it, but I hate it. However, the need to do that is not a result of "late-stage capitalism" or "enshittification" providing me with convenient cover to charge customers for useless updates. It's a necessary response to constantly evolving security threats that percolate through kernels, languages, package managers, until they hit my software and I either update or risk running vulnerable code on my customers' servers.
> I've seen photos of BSODs on airport monitors that show flight lists
The kiosk display terminal is not something I care about that much.
> We now have an entire industry dedicated to trying to layer security onto Windows
Too bad we have no such layering in our networks, our internet connections, or in our authentication systems.
Thinking about it another way there's actually no specific system in place to ensure your pilot does not show up drunk. We don't give them breathalyzers before the flight. We absolutely could do this even without significant disruption to current operations.
We have no need to actually do this because we've layered so many other systems on top of your pilot that they all serve as redundant checks on their state of mind and current capabilities to safely conduct the flight. These checks are broader and tend to identify a wider range of issues anyways.
This type of thinking is entirely missing at the computer network and human usability layer.
Was there ever such a time? If so then tell me when it was.
"The latest chaos wasn’t caused by an adversary, but it provided a road map of American vulnerabilities at a critical moment."
I've no doubt that road maps of American vulnerabilities are currently being planned, roadmaped and stockpiled for future use by those who aren't on the best terms with the US.
In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
Remember, this happened big-time when Microsoft introduced Windows especially Windows 95. To capture the market Microsoft made everything as easy as possible for nontechnical users—just click on something and it'd happen, things would happen with ease. And all this happened without due consideration to security.
When viruses, vulnerabilities, breaches got out of hand restrictions were introduced which meant users had less freedom to do what they'd gotten used to doing. What Microsoft did was to get the world used to slack operating procedures and efforts reign this in has met with user resistance ever since.
We're now stuck with a major problem that was easily foreseeable even before Microsoft launched Windows 95. Fixing it will be extremely difficult.
> In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
"Show me the incentives, and I'll show you the outcomes." - Charlie Munger.
We do not incentivize companies to operate secure, redundant, reliable computer systems. We incentivize companies to make the number at the bottom of the spreadsheet beat the expectations some analyst in Lower Manhattan set 90 days prior. And since companies handle the majority of societal work in the United States, that's how most critical systems are designed.
Now, there's a chance that this will play out in court, and that Crowdstrike will have to be bought out to make up for the damages their customers suffered starting on July 19th. However, that will take years, and the outcome could very well be that the plaintiffs will receive symbolic or even no damages. By then, the market will have hedged, captured regulatory authorities, cut its losses, and just altogether moved on. The assets will be purchased in a firesale by people who see this as "creative destruction" and won't care that peoples' lives were put at risk because of this.
> We do not incentivize companies to operate secure, redundant, reliable computer systems.
Except in the gambling industry. As part of a long-standing tradition, companies in the gambling industry are usually contractually required to take financial responsibility for errors.
GTECH's annual report, before they were acquired by an Italian company, says "We paid or incurred liquidated damages with respect to our contracts in an amount equal to 0.61%, 0.18%, 0.50%, 0.47% and 0.14% of our annual revenues in fiscal 2006, 2005, 2004, 2003 and 2002, respectively."[1]
So, forcing a transaction process service to take full responsibility for errors cost, at worst, 0.61% of revenue. This is sufficient to force gambling companies to use unusually good security technologies.
The Nevada Gambling Commission has technical rules.[2]
* "On-line slot systems may only communicate with equipment or programs external to the system
through a secure interface. This interface will specifically not allow any external connection to directly access the alterable data of the system." Which means no privileged "security" systems such as Crowdstrike.
* "Gaming device application
access to the system based game must be logged automatically on the system component of the
game and on a computer or other logging device that resides outside the secure area and is not
accessible to the individual(s) accessing the secure area." Which means the really important info must not only be logged, the logs have to be kept where the people who run the systems can't get at them. There are more logging requirements. Most things require two logs, one used for normal operation and a remote backup with tamper resistance and secure hashes.
* "Conditions for changing active software on a conventional gaming device or client
station that is part of a system supported or system based game:
(a) Be in the idle mode with no errors or tilts, no play and no credits on the machine for at least
two (2) minutes;
(b) Not be participating in an in-house or inter-casino linked payoff schedule..." There's more, but the general idea is that to change anything, you have to take the component being changed down to the idle, fully backed up state. Only then can changes be applied. All of which are logged.
The gaming industry has faced hostile actors for decades. They have reasonably strong defenses. Yet they're still very profitable.
Baking in resiliency is expensive. Its not obvious to me that it would be better to deal with that than to deal with issues like this once in a blue moon. Why not let the markets decide? If this ends up costing a bunch of money it will be fixed, if it doesnt it wasnt that big of a deal.
This is an area where studying Ukraine's experience will be very useful (and probably has already been useful)
There were years of cyberattacks against pretty much every peice of critical infrastructure they have. Things went down, there were disruptions, but they adapted. Sometimes by falling back to low-tech solutions, sometimes by developing new systems with robustness into new systems and purging the old (much easier to politically justify when the problem is tangible and immediate).
I seem to recall that one of the first things we did when tensions started ramping up was sending teams of cyber security experts from the NSA to help them lock down and root out infiltrations.
How nice of the NSA to help them after their exploit was leaked (vulnerability known for many years before that) and weaponized by Russia to attack Ukraine.
> This is an area where studying Ukraine's experience will be very useful
Are they unique in any way? Or is it just yet _another_ case of Windows software being deployed in critical roles and basic 0day vulnerabilities and exploits being applied against it?
If so.. the lesson has been known for decades.
> sending teams of cyber security experts from the NSA
It's nice to know our security agencies have time for games of whack a mole.
My first thought in all this was wondering if there's a business opportunity for a consulting firm or startup that designs and manages offline paper backup systems that can quickly and seamlessly integrate back with digital systems once they come back online.
The "cyber agencies" focus on offence, because that's easy to score points with and appear to be doing something, whereas defence is a very boring job of securing a zillion outdated endpoints. Or trying to get profitable megacorps to do something less vulnerable and less profitable.
Offense is also easy in that there is a ton of software out there, and you just need to find one vulnerability. There is a "win" condition" Defense is impossible as there is a ton of software and you need to protect all of it every time, there is only a "lose" condition.
>Was there ever such a time? If so then tell me when it was.
The 90s and into the early 2000s at least. You would get laughed out the room and then fucking fired if you hooked anything critical up to the internet.
"You would get laughed out the room and then fucking fired if you hooked anything critical up to the internet."
Perhaps this happened where you were, and lucky you it seems you were in a good environment.
But back then I was in IT management and I had precious little power to stop it especially given other senior managers were the culprits. The operation had another function and not IT as its primary role. Moreover, I saw very simular problems in other organizations that I was familiar with.
Also, during that period I was with another outfit whose principal function was surveillance—not of people but of info and physical stuff and I can assure you that whilst the system worked well try as we might it wasn't watertight.
I agree. Constant internet access and the assumption that other people should be able to push new code to your machine and have it run without you even being aware of it has killed all hope of resiliency.
I miss the days when any application that dared to phone home even just to check for updates was considered spyware. Today there is are huge numbers of people who have access to install and run whatever new code they want on our systems whenever they feel like it. If it's not the AV software, it's the browser, or the video card, or the mouse driver, or windows itself. It's totally unmanageable.
> Was there ever such a time? If so then tell me when it was.
It was a goal for a long time, and I'd say we use to be more resilient pre-cloud SaaS auto-update everything. When every software solution installation is on private networks, with fundamentally different architectures (both machine and topology), along with a wide selection of even very poor quality software, was a lot more resilient than what we have today.
Today a single outage in a single service (say AWS) can grind a large number of companies to a halt. A bad update like this one immediately impacts everyone all at once and has a domino effect. That didn't use to happen.
We've been concentrating our collective architecture into a few best practice tools but that all become single points of failure for not only digital attacks, but misconfigurations, mismanagement, company failures, exhausted underpaid engineers, optimizations, etc.
> Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
This isn't necessarily true, and I'd argue quite the opposite direction has been happening in the security industry over the past decade or so. People realized that hard security would only cause users to find simple predictable bypasses that would overall _weaken_ the security posture. You just have to look at the evolution of NIST recommendations around passwords to see this happening.
Must change a password every 90 days that can't be the same as your last 10 passwords and complex password requirements? Well users are going to use the minimum size in predictable patterns and just increment a number at the end. Those old password hashes you have to keep around to check if the user is reusing the password? Those are a liability that, when broken, tell the attacker which pattern each user is using. Not the case anymore and there is a lot more usable security rolled that is entirely transparent to end users or almost entirely transparent.
Think about how prevalent and bad captchas used to be on the website and how easy they were to circumvent. Cloudflare's and Google's captcha solution are pretty transparent and has much greater efficacy than the old ones.
Did Microsoft's general and on-going laxness contribute to bad security practices? Absolutely, but that is one ecosystem that had weird other by the nature of how inherently unstable that environment was and is not and hasn't except for maybe a brief peak ever been a core foundation of the internet infrastructure, just enterprise infrastructure unfortunately. They definitely never got the memo about usable or transparent security. I hope they're at least trying behind the scenes now.
> Was there ever such a time? If so then tell me when it was
It seems very plausible that "digital resilience" that this has been buzz phrase repeated often enough in meetings of security-adjacent corporate bureaucrats that some number of people convinced themselves it was a real thing.
And the same divorced-from-specifics approach allows these decision makers to paper over any and all choices that inherently weakened security 'cause the triage needed to partially protect the resulting structurally insecure system can be presented with similar glowing buzz phrases.
In a twisted way, Crowdstrike just gave western civilization a disaster recovery and resilience forced test. an actual attack won't be rolled back within an hour.
In case you don't know, Crowdstrike is hardly the only company with large scale access to this many companies,governments and resources. It takes one rogue employee to deploy a disk wiper that destroys every computer (including linux and macos) and affected systems won't recover at all. it would be months before critical systems are back online, the global economy would come to a halt worse than how it did with COVID in such a scenario.
It isn't "why didn't Crowdstrike do better" (although they should have), it is more, why isn't technology in critical systems more resilient to one vendor screwing up or getting hacked?
For example, let's say it wasn't just a boot loop but a disk wiper erased every boot disk, is there any reason pxe booting a recovery image or a backup image configured already on servers, atms, kiosks, point of sale systems,etc...? even if UEFI and bios were erased, it is technically not impossible to have an auto-recovery mechanism implemented right?
If you have never been in an incident response (IT and security incidents) root cause analysis, I don't blame you for not thinking deeper about the root cause, but that is the type of root cause analysis that has been missing despite over a decade of rampant ransomware, disk wipers, and supply chain risks.
Finding someone to blame and be angry at is easy and doesn't solve the root cause. Making hard technical decisions and not wasting this opportunity (never waste a good crisis) to push for resilient technology investments actually solves the root cause behind this and other repeating problems.
if the firmware is totally nuked, you'd need backup firmware. at some point, all of this crap can be made non-recoverable, but that isn't the real problem to solve.
imma take your comment one step further and say that the emphasis on security is coming at the expense of discussions on resilience. and security matters a lot less, especially financially, than resilience.
This has been an open secret for decades. Just a handful of major OS and browser vendors, constantly shipping patches to their systems and most software having such vast software supply chains that it's effectively impossible to audit anything, let alone truly certify anything as safe, and "security" software just expands the attack surface.
If you are a non-US company you have to be insane to use this CrowdStrike service. The FBI can legally use a secret warrant[1] and force CrowdStrike to inject a DLL into your infrastructure!
Are you sure that is correct? I was under the impression that US government could order companies to turn over data, but that they could not compel them to actually do work. This was the center of the dispute between the government and Apple after the San Bernadino shooting: Apple was within their legal rights to refuse to provide assistance. https://en.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption_d...
The lengths that the NSA and CIA would go to to implant backdoors (interdicting shipments of laptops/phones and doing the work themselves) further suggests that they cannot compel this sort of action.
That case was theatre / kayfabe. The FBI was using an emotive case for turning public opinion against encryption and set some legal precedent. The goal wasn't really to unlock the phone which could and was be done by other means.
If they have a path to covertly compel action as a state secret under National Security / anti-terror laws we will only hear about it from whistleblowers. It won't be something the target can disclose let alone test in court.
FWIW I also don't believe in Apple's nobility as resisting on user's behalf. They happily bow to the state and remove apps for e.g. organising protests, monitoring deaths in US wars, csam scanning etc. IMHO their interest in encryption is to prevent jail-breaking and protect their app-store cash cow.
> interdicting shipments of laptops/phones and doing the work themselves
I don't think that proves anything about their powers. Given the option, I'm sure they would prefer to install things themselves without third-party knowledge or consent.
We have evidence of complicit action e.g. black rooms like Room 641A. I think the nature of "consent" and "obligation" gets pretty grey when it comes to the security agencies. They don't get results using court orders. I'm sure they have assets employed as staff in security sensitive positions.
Just told my family yesterday that if we are ever in a real war expect everything to stop working within 8 hours. We will go back to cash and paperwork but it will be painful and slow.
We're already there. The fact that we didn't see civilization collapse is evidence that there is a ton of infrastructure not running Windows and Crowdstrike.
This wasn't nearly as bad as it could have been. What if the crash wasn't just a crash but resulted in data corruption? And what if it took longer to stop the rollout and deploy a fixed version? How long would it have taken to recover from this kind of incident? If affected machines didn't fix themselves after several reboots but needed to be actively reimaged?
On top of that, I am still struggling to understand how the people in charge of running orgs that run highly critical systems were OK with the idea that a 3rd party software provider could push at anytime patches to the software they provide.
Sorry for being harsh with my following statement, but I believe that the companies affected by Crowdstrike share some responsibility on what happened yesterday.
You're making the mistake of assuming that the people running those companies care about anything other than their job security, and buying in solutions is the best way to have a ready-made scapegoat when things go wrong. The mantra "no-one ever got sacked for buying IBM" still holds, you can just substitute "Oracle", or "Microsoft", or now - apparently - "Crowdstrike".
- pushing patches is objectively a good idea, rapid response to threats and all.
- Whats bad is instant global 0->1 rollout, instead of more gradual, blue/green/canary however you call it. With gradual rollout policy this whole thing could have been caught at their first couple guinea pig customers, and not the whole world
I think I agree with you.
On the other hand, I can also imagine that if autoupdates weren't the case, then 90% of installations would be a terribly outdated and probably vulnerable version. It's hard to imagine a common sense middle ground.
One could make the argument that automatically patched software is, in aggregate, more secure/less problematic than chronically under-patched software that requires manual, human attention.
Not necessarily. CrowdStrike isn't even the #1 player in this space, but this still happened because of network effects. The number of platforms you'd need for this much safety is impractically high.
"Network effects"? You mean like, "I'd be fine, but I depend on a service from a Windows machine, so I'm still screwed" ?
> The number of platforms you'd need for this much safety is impractically high
I don't see why this becomes an impossible problem. If all the essential services are not provided by a single software infrastructure, then we have the required diversity, right?
You do realize that CrowdStrike also runs on Linux and that there have been a variety of instances of bad CrowdStrike updates breaking Linux machines, right?
Readit News
Security is not a feature that can be layered on. It has to be built in. We now have an entire industry dedicated to trying to layer security onto Windows -- but it still doesn't work.
The vendor who makes the software has always written for Windows (or in reality, wrote for either DOS or OS/2 then transitioned to NT4). History, momentum, familiarity, cost, and ease of support all are factors (among others, I'm sure).
Security is a process, not a product.
And yes, distros require frequent updates, though more to your point, you can limit the scope of installed software. I'm sure airport displays don't need MPEG2, VP1 and so on codecs, for instance.
It's also important to remember that there is a lot of 'garageware' out there with these specialized systems. Want SAML/OIDC support? We only support LDAP over cleartext, or Active Directory at best. Want the latest and greatest version of Apache Tomcat? Sorry, the vendor doesn't know how to troubleshoot either, so they only "support" a three year old vulnerable version.
Ran into that more than a few times.
Given the hypothesis of what caused the BSOD with Crowdstrike (NUL pointer), using a safe language would have been appropriate -- it's fairly easy in this case to lay the blame with CS.
Microsoft supplies the shotgun. It's the vendors responsibility to point it away from themselves.
They don't, until the day the airport managers are approached by an advertising company waving the wads of cash the airport could be 'earning' if only they let "AdCo" display, in the top 1/4 of each screen, a video advertising loop. At which point, those displays need the codecs for "AdCo's" video ads.
Security is a process, not a product...
The vendor who makes the software has always written for Windows (or in reality, wrote for either DOS or OS/2 then transitioned to NT4). History, momentum, familiarity, cost, and ease of support all are factors (among others, I'm sure)...
That's starting the argument with "weight loss is about overall diet process, not individual choices" and then hopping to "ice cream for dinner is good 'cause it's convenient and I like it".
The statement "Security is a process, not a product." means you avoid shitty choices everywhere, not you make whatever choices are convenient, try to patch the holes with a ... product ... and also add an extra process to deal with the failures of that product.
Dead Comment
[1] https://news.ycombinator.com/item?id=41018029
This is a common misunderstanding, an OS that receives frequent security updates is a very good thing. That means attention is being paid to issues being raised, and risks are being mitigated. Security is not a 'checkbox' it's more of a neverending process because the environment is always in a state of flux.
So to flip it, if an OS is not receiving updates, or not being updated frequently, that's not great.
What you want is updates that don't destabilize an OS, and behind that is a huge history and layers of decisions at each 'shop' that runs these machines.
Security is meant to be in layers and needs to be built in.
> but it still doesn't work.
It does work because the 'scene' has been silent for so long, but what we as humans notice is the incident where it didn't.
We've got a bunch of computers that mostly don't make mistakes at the hardware layer. On top of that, we can write any programs we want. Even though the halting problem exists, and is true for arbitrary programs, we know how to prove all sorts of useful security properties over restricted sets of of programs.
Any software security pitch that starts with "when the software starts acting outside of its spec, we have the system ..." is nonsense. In practice, "acting outside its spec" is functionally equivalent to "suffers a security breach".
Ideally, you'd use an operating system that has frequent updates that expand functionality, that is regularly audited for security problems, and that only rarely needs to ship a security patch. OpenBSD comes to mind.
If software has frequent security updates over a long period of time, that implies that the authors of the system will continue to repeat the mistakes that led to the vulnerabilities in the first place.
That's before even addressing mistakes.
Most people know how to use a windows computer.
Most IT desktop support knows how to use and manage windows. Even building facilities folks can help support them.
Microsoft makes it easy to manage a fleet of computers. They also provide first party (along with thousands of 3rd parties) training and certifications for it.
Windows are the de facto Business Machines.
Most signage companies use windows.
Finding someone who knows a BSD is not easy.
A windows computer that relies on cloud services, as an increasing and often nonsensical subset of the functionality on one does, can often only be fixed by Microsoft directly
Microsoft intervenes directly and spends billions of dollars annually on anticompetitive tactics to ensure that other options are not considered by businesses
And with this monopoly, it has shielded itself from having to compete on even crucial dimensions like reliability, maintainability, or security
I know of a very small airport where what is displayed over the HDMI part is essentially Firefox at fullscreen with powersaving disabled so the screen does not blank. Some of them are Intel NUC, some of them are Raspberry Pi with HSM in a box. These devices basically "boot to Firefox" with relevant credentials read off internal TPM/HSM.
Those among airport staff who do not know how to use a computer at all can get them working by just plugging them in.
> Most people know how to use a windows computer.
They know enough to open a browser.
> Most IT desktop support knows how to use and manage windows.
They know how to cope with Windows, at best.
> Finding someone who knows a BSD is not easy.
BSD is everywhere and in far more places than Windows, like almost every car sold after 2014. But you never ever see BSD because it's already-working with nothing for the end customer to do.
Airport staff are not debugging the windows install. They power-cycle it and see what happens, otherwise call the vendor to come in.
So there's no actual reason other than lazyness to build kiosk mode computers on windows.
Another take to be done here is: computers shouldn't have unfiltered internet access all the time.
Whitelist it and once every 3 days open the internet gates.
(Easier said than done)
"Nobody ever gets fired for buying IBM" is as true as ever at least in the corporate world.
I was personally involved in a meeting where my firm's leadership advised a client who did fire their CTO and a bunch of other people for what was ultimately putting what they thought were smart career moves over their actual responsibilities.
Unfortunately, as you did just point out, the CEO, other execs, and board are often just as incompetent as the CTO/CISO who have such shit-brained mindset.
Deleted Comment
I suppose you could build it as a UEFI module that relies on the UEFI firmware to initialize the hardware but then you get a text only interface. But then the UEFI is the OS.
But this outage was not an OS problem. It was an application bug that used invalid pointers. If it was a unikernel it still would have crashed.
What makes you think so?
How is Linux better in that area?
I understand the reasons for it, and why large, billion dollar companies try to create some sort of efficiency by centralising on one "vendor", but, then this happens.
I don't know how to fix the problem of following "Industry Trends" when every layer above me in the organisation is telling me not to spend the time (money) to investigate alternative software choices which don't fit into their nice box.
The same thing crash could happen with any kernel driver in any operating system.
You've never seen Linux crash because of a driver bug?
from the reporting so far, no one has died as a result of the Crowdstrike botch. For my money, that sounds like it's not being used in 'critical industry'.
/unset
There were several 911 service outages included in the news yesterday, so I would definitely say agree those fall into the category. I haven't seen how many hospitals were deeply affected; I know there were several reports of facilities that were deferring any elective procedures.
Nobody's commenting on that because it's the wrong thing to focus on.
1) This fuckup was on CrowdStrike's Falcon tool (basically a rootkit) bricking Windows due to a bad kernel driver they pushed out without proper hygiene, not on Windows's security patches being bad.
2) Linux also needs to get patches all the time to be secure (remember XZ?) It's not just magically secure by default because of the chubby penguin but is only as secure as it's most vulnerable component, and XZ proved it has a lot of components. I'd be scared if a long period goes by and I see no security patches being pushed to my OS. Modern software is complex and vulnerabilities are everywhere. No OS is ever bug-free and fully bullet proof in order to believe it can be secure without regular patches. Other than TempleOS of course.
The lesson is whichever OS you use, don't surrender your security to a single third party vendor who you now have to trust with the keys of your kingdom as that now becomes your single point of failure. Or if you do be sure you can sue them for the damages.
It's just a likely they could crash a Linux machine by releasing an update to their Linux software that also referenced invalid memory.
Am I the only one that's seen drivers in Linux cause a kernel panic?
microkernels, microkernels, microkernels! https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...
1) While CrowdStrike can be run on Linux it is less of a risk to use Linux without it than Windows. I don't think most Linux/BSD boxes would benefit from it. It could be useful for a Linux with remotely accessible software of questionable quality (or a desktop working with untrusted files) but this should not be the case for any critical system.
2) There is a difference between auto-updates (common in Windows world) and updates triggered manually only when it is necessary (and after testing in non-prod environment). Also while Linux is far from being bug-free, remotely exploitable vulnerabilities are rare.
Or even ChromeOS which has insane security.
> but it still doesn't work.
It works momentarily but there will always be 0-days the people who make the exploits intimately know the windows API internals.
ChromeOS is a Linux distro BTW
Minimal software and OS running on linux as a layer between any windows/whatever and internet connectivity. Minimize and control the exact information that gets to the less hardened and trustworthy/complicated computers
We moved to a more frequent update cycle because when a critical vulnerability was found, no one wanted to wait 6-12 months for the service pack.
Deleted Comment
There's an entire industry for guard-railing LLMs now. Go figure.
Because in the non-Silicon-Valley world of software, if you pick Linux and it has issues, fingers will get pointed at you. If you pick Windows and it has issues, fingers will get pointed at Microsoft.
Operating systems that don't require frequent security patches aren't profitable.
Anyway, this is the step of late-phase capitalism that comes after enshittification. Ghost in the Shell 2045 calls it "sustainable war". I'd link to an article, but they're all full of spoilers in the first paragraph.
It probably suffices to say that the series refers to it as capitalism in its most elegant form: It is an economic device that can continue to function without any external inputs, and it has some sort of self-regulatory property that means the collateral damage it causes is just below the threshold where society collapses.
In the case of Cloud Strike, the body count is low enough, and plausible deniability is low enough that the government can get away with not jailing anyone.
Instead, the event will increase the money spent on security theater, and probably lead to a new regulatory framework that leads to yet-another layer of mandatory buggy security crapware (which Cloud Strike apparently is).
In turn, that'll lower the margins of anyone that uses computers in the US by something like 0.1%, and that wealth will be transferred into the industry segment responsible for the debacle in the first place. Ideally, the next layer of garbage will have a bigger blast radius, allowing the computer security complex to siphon additional margins.
Consider the reasons people need this endlessly updated layer of garbage, as you put it. The constant evolution of 0-days and ransomware.
I'm a developer, and also a sysadmin. Do you think I love keeping servers up to the latest versions of every package where a security notice shows up, and then patching whatever that breaks in my code? I get paid for it, but I hate it. However, the need to do that is not a result of "late-stage capitalism" or "enshittification" providing me with convenient cover to charge customers for useless updates. It's a necessary response to constantly evolving security threats that percolate through kernels, languages, package managers, until they hit my software and I either update or risk running vulnerable code on my customers' servers.
Dead Comment
The kiosk display terminal is not something I care about that much.
> We now have an entire industry dedicated to trying to layer security onto Windows
Too bad we have no such layering in our networks, our internet connections, or in our authentication systems.
Thinking about it another way there's actually no specific system in place to ensure your pilot does not show up drunk. We don't give them breathalyzers before the flight. We absolutely could do this even without significant disruption to current operations.
We have no need to actually do this because we've layered so many other systems on top of your pilot that they all serve as redundant checks on their state of mind and current capabilities to safely conduct the flight. These checks are broader and tend to identify a wider range of issues anyways.
This type of thinking is entirely missing at the computer network and human usability layer.
Was there ever such a time? If so then tell me when it was.
"The latest chaos wasn’t caused by an adversary, but it provided a road map of American vulnerabilities at a critical moment."
I've no doubt that road maps of American vulnerabilities are currently being planned, roadmaped and stockpiled for future use by those who aren't on the best terms with the US.
In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
Remember, this happened big-time when Microsoft introduced Windows especially Windows 95. To capture the market Microsoft made everything as easy as possible for nontechnical users—just click on something and it'd happen, things would happen with ease. And all this happened without due consideration to security.
When viruses, vulnerabilities, breaches got out of hand restrictions were introduced which meant users had less freedom to do what they'd gotten used to doing. What Microsoft did was to get the world used to slack operating procedures and efforts reign this in has met with user resistance ever since.
We're now stuck with a major problem that was easily foreseeable even before Microsoft launched Windows 95. Fixing it will be extremely difficult.
"Show me the incentives, and I'll show you the outcomes." - Charlie Munger.
We do not incentivize companies to operate secure, redundant, reliable computer systems. We incentivize companies to make the number at the bottom of the spreadsheet beat the expectations some analyst in Lower Manhattan set 90 days prior. And since companies handle the majority of societal work in the United States, that's how most critical systems are designed.
Now, there's a chance that this will play out in court, and that Crowdstrike will have to be bought out to make up for the damages their customers suffered starting on July 19th. However, that will take years, and the outcome could very well be that the plaintiffs will receive symbolic or even no damages. By then, the market will have hedged, captured regulatory authorities, cut its losses, and just altogether moved on. The assets will be purchased in a firesale by people who see this as "creative destruction" and won't care that peoples' lives were put at risk because of this.
And the cycle will continue.
Except in the gambling industry. As part of a long-standing tradition, companies in the gambling industry are usually contractually required to take financial responsibility for errors. GTECH's annual report, before they were acquired by an Italian company, says "We paid or incurred liquidated damages with respect to our contracts in an amount equal to 0.61%, 0.18%, 0.50%, 0.47% and 0.14% of our annual revenues in fiscal 2006, 2005, 2004, 2003 and 2002, respectively."[1]
So, forcing a transaction process service to take full responsibility for errors cost, at worst, 0.61% of revenue. This is sufficient to force gambling companies to use unusually good security technologies.
The Nevada Gambling Commission has technical rules.[2]
* "On-line slot systems may only communicate with equipment or programs external to the system through a secure interface. This interface will specifically not allow any external connection to directly access the alterable data of the system." Which means no privileged "security" systems such as Crowdstrike.
* "Gaming device application access to the system based game must be logged automatically on the system component of the game and on a computer or other logging device that resides outside the secure area and is not accessible to the individual(s) accessing the secure area." Which means the really important info must not only be logged, the logs have to be kept where the people who run the systems can't get at them. There are more logging requirements. Most things require two logs, one used for normal operation and a remote backup with tamper resistance and secure hashes.
* "Conditions for changing active software on a conventional gaming device or client station that is part of a system supported or system based game: (a) Be in the idle mode with no errors or tilts, no play and no credits on the machine for at least two (2) minutes; (b) Not be participating in an in-house or inter-casino linked payoff schedule..." There's more, but the general idea is that to change anything, you have to take the component being changed down to the idle, fully backed up state. Only then can changes be applied. All of which are logged.
The gaming industry has faced hostile actors for decades. They have reasonably strong defenses. Yet they're still very profitable.
[1] https://www.sec.gov/Archives/edgar/data/857323/0000950123060...
[2] https://gaming.nv.gov/uploadedFiles/gamingnvgov/content/Home...
The question is what can be done, if anything. But I've a solution in my wildest dreams as a dictator. :-)
There were years of cyberattacks against pretty much every peice of critical infrastructure they have. Things went down, there were disruptions, but they adapted. Sometimes by falling back to low-tech solutions, sometimes by developing new systems with robustness into new systems and purging the old (much easier to politically justify when the problem is tangible and immediate).
I seem to recall that one of the first things we did when tensions started ramping up was sending teams of cyber security experts from the NSA to help them lock down and root out infiltrations.
Are they unique in any way? Or is it just yet _another_ case of Windows software being deployed in critical roles and basic 0day vulnerabilities and exploits being applied against it?
If so.. the lesson has been known for decades.
> sending teams of cyber security experts from the NSA
It's nice to know our security agencies have time for games of whack a mole.
My first thought in all this was wondering if there's a business opportunity for a consulting firm or startup that designs and manages offline paper backup systems that can quickly and seamlessly integrate back with digital systems once they come back online.
The 90s and into the early 2000s at least. You would get laughed out the room and then fucking fired if you hooked anything critical up to the internet.
Perhaps this happened where you were, and lucky you it seems you were in a good environment.
But back then I was in IT management and I had precious little power to stop it especially given other senior managers were the culprits. The operation had another function and not IT as its primary role. Moreover, I saw very simular problems in other organizations that I was familiar with.
Also, during that period I was with another outfit whose principal function was surveillance—not of people but of info and physical stuff and I can assure you that whilst the system worked well try as we might it wasn't watertight.
I miss the days when any application that dared to phone home even just to check for updates was considered spyware. Today there is are huge numbers of people who have access to install and run whatever new code they want on our systems whenever they feel like it. If it's not the AV software, it's the browser, or the video card, or the mouse driver, or windows itself. It's totally unmanageable.
It was a goal for a long time, and I'd say we use to be more resilient pre-cloud SaaS auto-update everything. When every software solution installation is on private networks, with fundamentally different architectures (both machine and topology), along with a wide selection of even very poor quality software, was a lot more resilient than what we have today.
Today a single outage in a single service (say AWS) can grind a large number of companies to a halt. A bad update like this one immediately impacts everyone all at once and has a domino effect. That didn't use to happen.
We've been concentrating our collective architecture into a few best practice tools but that all become single points of failure for not only digital attacks, but misconfigurations, mismanagement, company failures, exhausted underpaid engineers, optimizations, etc.
> Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
This isn't necessarily true, and I'd argue quite the opposite direction has been happening in the security industry over the past decade or so. People realized that hard security would only cause users to find simple predictable bypasses that would overall _weaken_ the security posture. You just have to look at the evolution of NIST recommendations around passwords to see this happening.
Must change a password every 90 days that can't be the same as your last 10 passwords and complex password requirements? Well users are going to use the minimum size in predictable patterns and just increment a number at the end. Those old password hashes you have to keep around to check if the user is reusing the password? Those are a liability that, when broken, tell the attacker which pattern each user is using. Not the case anymore and there is a lot more usable security rolled that is entirely transparent to end users or almost entirely transparent.
Think about how prevalent and bad captchas used to be on the website and how easy they were to circumvent. Cloudflare's and Google's captcha solution are pretty transparent and has much greater efficacy than the old ones.
Did Microsoft's general and on-going laxness contribute to bad security practices? Absolutely, but that is one ecosystem that had weird other by the nature of how inherently unstable that environment was and is not and hasn't except for maybe a brief peak ever been a core foundation of the internet infrastructure, just enterprise infrastructure unfortunately. They definitely never got the memo about usable or transparent security. I hope they're at least trying behind the scenes now.
Correct, but on evidence and in practice it's a totally different matter.
Read my other posts here, especially my comment on physical security vs IT security. Unfortunately, the evidence backs my assertions.
> Was there ever such a time? If so then tell me when it was
It seems very plausible that "digital resilience" that this has been buzz phrase repeated often enough in meetings of security-adjacent corporate bureaucrats that some number of people convinced themselves it was a real thing.
And the same divorced-from-specifics approach allows these decision makers to paper over any and all choices that inherently weakened security 'cause the triage needed to partially protect the resulting structurally insecure system can be presented with similar glowing buzz phrases.
In case you don't know, Crowdstrike is hardly the only company with large scale access to this many companies,governments and resources. It takes one rogue employee to deploy a disk wiper that destroys every computer (including linux and macos) and affected systems won't recover at all. it would be months before critical systems are back online, the global economy would come to a halt worse than how it did with COVID in such a scenario.
It isn't "why didn't Crowdstrike do better" (although they should have), it is more, why isn't technology in critical systems more resilient to one vendor screwing up or getting hacked?
For example, let's say it wasn't just a boot loop but a disk wiper erased every boot disk, is there any reason pxe booting a recovery image or a backup image configured already on servers, atms, kiosks, point of sale systems,etc...? even if UEFI and bios were erased, it is technically not impossible to have an auto-recovery mechanism implemented right?
If you have never been in an incident response (IT and security incidents) root cause analysis, I don't blame you for not thinking deeper about the root cause, but that is the type of root cause analysis that has been missing despite over a decade of rampant ransomware, disk wipers, and supply chain risks.
Finding someone to blame and be angry at is easy and doesn't solve the root cause. Making hard technical decisions and not wasting this opportunity (never waste a good crisis) to push for resilient technology investments actually solves the root cause behind this and other repeating problems.
imma take your comment one step further and say that the emphasis on security is coming at the expense of discussions on resilience. and security matters a lot less, especially financially, than resilience.
Backup firmware and boot images can be configured as read only.
Everyone in the industry knows this.
Interesting to see the NYT just catching up.
Maybe it has to do with some major incident that happened yesterday, and the fact they are a news company?
Deleted Comment
[1] https://en.wikipedia.org/wiki/United_States_Foreign_Intellig...
The lengths that the NSA and CIA would go to to implant backdoors (interdicting shipments of laptops/phones and doing the work themselves) further suggests that they cannot compel this sort of action.
If they have a path to covertly compel action as a state secret under National Security / anti-terror laws we will only hear about it from whistleblowers. It won't be something the target can disclose let alone test in court.
FWIW I also don't believe in Apple's nobility as resisting on user's behalf. They happily bow to the state and remove apps for e.g. organising protests, monitoring deaths in US wars, csam scanning etc. IMHO their interest in encryption is to prevent jail-breaking and protect their app-store cash cow.
> interdicting shipments of laptops/phones and doing the work themselves
I don't think that proves anything about their powers. Given the option, I'm sure they would prefer to install things themselves without third-party knowledge or consent.
We have evidence of complicit action e.g. black rooms like Room 641A. I think the nature of "consent" and "obligation" gets pretty grey when it comes to the security agencies. They don't get results using court orders. I'm sure they have assets employed as staff in security sensitive positions.
https://cbr.ru/eng/press/event/?id=18776
https://bank.gov.ua/en/news/all/drugiy-rik-povnomasshtabnoyi...
https://www.bloomberg.com/news/articles/2024-04-10/gaza-face...
Outages are largely limited to physical infrastructure that’s attacked by missiles. Russia isn’t a slouch in digital warfare, either.
No lack hostile hackers.
If more of the critical machines were running different OS's, the damage would be contained.
When we talk about the dangers of "monoculture" it's usually about plants. The same danger applies to computing infrastructure.
I'm sure IBM mainframes are still running critical stuff, too.
Sorry for being harsh with my following statement, but I believe that the companies affected by Crowdstrike share some responsibility on what happened yesterday.
- Whats bad is instant global 0->1 rollout, instead of more gradual, blue/green/canary however you call it. With gradual rollout policy this whole thing could have been caught at their first couple guinea pig customers, and not the whole world
Not if they were running the same CrowdStrike.
"Network effects"? You mean like, "I'd be fine, but I depend on a service from a Windows machine, so I'm still screwed" ?
> The number of platforms you'd need for this much safety is impractically high
I don't see why this becomes an impossible problem. If all the essential services are not provided by a single software infrastructure, then we have the required diversity, right?
Windows is shit.
Mac is more or less.
Linux is best of all.
https://access.redhat.com/solutions/7068083