I'm not a fan of Microsoft, but this is some amazing blame shifting. The root cause of the problem is the government single-sourcing a vendor and being incapable of negotiating with said vendor. The US government is 10% of Microsoft's annual revenue just on security services (if I read the article correctly) but is failing to negotiate. The right answer here is if the situation is that bad, make a very public long-term commitment to shift to something else & up-level your IT department to be able to execute multi-year projects competently. Instead of meeting Microsoft on the business playing field, it's trying to use scary "national security threat" verbiage to try to bully them in the court of public opinion.
Regarding security, at the end of the day, the US government is a huge target for adversaries. You can't outsource your security practices & if MS software is really that much worse they should be fixing their purchasing requirements. The reality though is that whatever software the US government would switch to would become the focus of adversarial research.
Well you can't not outsource your security because gov payscale limits do not match market reality. You have to realise that a ton of people who should be directly employed by NSA etc. are actually working for their contracts for this reason.
The US government isn't in need of a thousand high-skilled hackers. They are in need of a million normal employees with some basic security awareness. Anyone with a modicum of skill can find thousands of areas to improve. The issue is that almost nobody is in a position to get anything changed. Even basic software choices are a multi-year epic.
The government is able to employ tons of smart people that could be making way more money elsewhere.
They might not get the absolute best security people in the world, but they could get good enough - as good as they're getting from MS for a fraction of the price.
Additionally, government salaries aren't terrible when you factor in the pension. Most people want the money now. But if you want financial security in the future - that's a reason a lot of people chose to work for the Fed.
I’d imagine that these proclamations are part of the negotiation.
I wouldn’t want to be a Microsoft account executive on the US govt contract right now; they’re about to have a massive load of additional requirements.
And if they don’t play ball, possibly antitrust to weaken their stranglehold on being the only real enterprise player.
I’m not saying any of this is the right approach, but it’s a tool in the governments toolbox.
any commercial channel filled with this much money will attract hordes of smiling useful idiots filling every possible niche. Now you have another problem, managing useful idiots over time.
> The right answer here is if the situation is that bad, make a very public long-term commitment to shift to something else & up-level your IT department to be able to execute multi-year projects competently.
The problem is that there isn't much in terms of alternatives, especially not if you prefer to have one software / vendor / tech stack.
- In groupware, there used to be Lotus Notes, but that went down the drain years ago. Thunderbird can do everything Outlook can (i.e. provide an email client, calendar and address book), but there is no official(ly supported) Thunderbird server software suite so there's always a potential for subtle bugs between whatever one chooses for directory, email and calendar backends.
- for AD there's obviously Samba but it, again, lacks a management UI that supports all of its features, so yet another potential for issues.
- the Office suite alternatives are even more of a nightmare, both in terms of usability, stability as well as compatibility with the millions of legacy files originating from Office. Or hell, even compatibility with old versions of the same app isn't a given in LibreOffice. (And I'm not sure stuff like MS Access even has a FOSS counterpart)
- And then, there's all the other stuff that integrates with AD for authentication/authorization. In a lot of cases, it's "either use MS AD or you're on your own when you hit issues".
- finally, Windows itself. Essentially, the US Government would have to sink billions of dollars into ReactOS development to make it compatible enough with mainstream Windows versions to run all the legacy software that people use - and no, WINE alone is not enough, not for anything that deals with hardware directly. And I wouldn't assume it's possible to even hire enough developers that are skilled to develop for WINE/ReactOS and fulfill the project requirements of never having been exposed to Windows source code.
Microsoft has achieved an insane amount of vendor lock-in, even Apple with all its financial and technological might or Valve (who invested a huge amount of money and work into getting WINE feature-rich enough to run a ton of AAA games on their Steam Deck) have been able to even come close. They can provide as shitty a service/software as they want, their audience literally has no other choice.
(Me personally, I'll keep my Windows 7 and 10 VMs alive for as long as I can, but no way in hell I'm ever moving to the ad-ridden, bling-bling flashy pseudo-hipster-UI disaster that is Windows 11)
>Thunderbird can do everything Outlook can (i.e. provide an email client, calendar and address book), but there is no official(ly supported) Thunderbird server software suite so there's always a potential for subtle bugs between whatever one chooses for directory, email and calendar backends.
Thunderbird is a lot more reliable than outlook - I use both daily. But yeah there's no 'thunderbird software suite'. I don't think that's really an issue though. Outlook breaks regularly, and our desktop support group had to deploy an 'outlook reset' app because the vendor is unresponsive. Being that unresponsive lessens the attractiveness of a 'software suite'.
>Or hell, even compatibility with old versions of the same app isn't a given in LibreOffice.
If I find MS word can't open a file, Libreoffice always does. I've never had a problem opening a libreoffice writer or MS Word file. I'm no 'power user' though.
The reason businesses and organizations stick to microsoft is because it's what they're used to, not because the apps themselves are better.
As for microsoft as a vendor - they are as obtuse a vendor as I've ever dealt with. Nearly always 100% unresponsive - even with projects involving hundreds of thousands of yearly dollars in licensing fees. I think being unresponsive is some kind of corporate culture thing with them. They're like the McDonalds of IT vendors. You get what's on the menu and that's it. If it doesn't work, well too bad because they know the perceived cost of switching is believed to be too high.
Wine can run apps that access hardware without issues.
The main problem is the catch-22: there's not much point in developing competitors to Microsoft's stack if businesses aren't going to ever consider them, and government departments won't consider alternatives if they can't get everything from a single vendor in a 100% risk free manner thanks to the "nobody ever got fired for buying Microsoft" and "one throat to choke" mentality you get there.
Governments do this to themselves. The USG doesn't even have to pick Microsoft. They could potentially sign contracts with Apple for workstation hardware and services, that would encourage and feed the alternative ecosystem based around Apple, or they could fund Linux, etc. They don't though. It's just soooo much easier to sign one giant contract, because that minimizes work for them and it's not really important to get value for money if you're in the public sector. Signing a great deal won't get you a big bonus or anything like that, but taking risks can get you blocked from promotions. So, why risk anything?
I used to think the way I’d try to tackle a challenge like that would be essentially embracing the ChromeOS model by going web-first for everything so you’d avoid accumulating new accidental dependencies on proprietary software, and trying to shift legacy apps to WINE, terminal servers, etc. Obviously you’d need thousands of exceptions but the important part would be removing the inertia which means that new code starts out being non-portable.
Unfortunately, these days the browser engine market is pretty lopsided so that strategy would need to be paired with something like a requirement to test in Gecko and WebKit because I trust Google’s long-term management even less than Microsoft.
If you want to use Windows outside of an air gapped environment, it’s reckless to use anything less than Windows 10, and even for that you have only about a year left unless you pay for extended support.
Also, as someone who actually uses Windows 11 Professional all day, every working day, I honestly haven’t encountered these adverts everyone is always referring to (maybe it’s because I’m not in the US).
As for the UI, initial release didn’t allow ungrouping of Windows on taskbar and context menus hid most things behind an extra click. This was crap, but a major update a while back has resolved these.
I’ve tried Linux, I occasionally need to use it or macOS for development of our cross platform Electron based product, and I would still choose Windows as it really has consistently “just worked” for me since around Windows 2000.
But if macOS or Linux works well for you, I’m happy you’re happy and perhaps you in turn could be happy for people who are happy with Windows (and ideally without thinking they must be deluded or something).
I assume it's the $20 billion in security services statement compared against their ~$200 billion yearly revenue. I'm not sure those security services are all for the U.S. govt though.
we have a company that has a monopoly that is honestly unimaginable (how does someone monopolize computation of all things...)
and we know that all monopolies require government enforcement to prevent others from competing...
then we know that there's no such thing as a conflict between Microsoft and the us government
the us govt is to serve Microsoft and that's that. any conversation like this about the merits of Microsoft as a market participant are laughable and disingenuous
theres no programmer alive that knows the history of Microsoft that would speak about them as you have just done, as an honest company trying to make a product
Then go after them for monopolistic abuses. Arguably Lina Khan is the first FTC chair to be making noise about it yet the only thing about Microsoft has been their OpenAI partnership because it's sexy & nothing about their traditional marketplace participation. The EU is going after them a bit more aggressively & having some success. Ultimately this would still be the US government's failings - claiming that makes Microsoft a national security threat is a joke & disingenous.
I know the history of MS anti-trust fairly well having grown up during the core part of their monopolist years - now they're part of the familiar oligopaly that's strangling tech.
> and we know that all monopolies require government enforcement to prevent others from competing...
Do we know that? That may be true of government run monopologies, but I'm pretty sure most marketplace monopolies rely on lack of government enforcement. For example, here's some analysis showing how US case law makes the government wary of going after Amazon for driving Quidsi out of business & forcing an acquisition by Amazon: https://cei.org/blog/amazons-private-labels-dont-threaten-co.... Ironically US does enforce rules against dumping from international players which shows that it realizes the harm that such activity can have, it just chooses to allow it for domestic players abusing smaller domestic players.
To be more accurate, the leadership of Microsoft's lack of prioritizing security, aka basic quality of product, is a national security threat.
A similar claim could obviously be made of Boeing. Just imagine what is happening in their military contracts which we are not allowed to hear about. Looking at the projects which we are allowed to know about, airliners and Boeing's Starliner, clearly Boeing management needs to be put out to pasture.
The core issue is cutting corners, for profit. This is not an issue which is easy to handle in our system. It seems that the best we can do is name and shame. Let's do that at least.
The US govt can also do a lot better. The very agency they formed to counter cyber threats and alert against(CISA) itself got hacked because they failed to patch or remediate, and led to a serious leak of sensitive chemical industry information among others. Because they failed to follow their own security advisory. And they won't even put out a report detailing the hack like MS did.
> According to an early report on the breach, an anonymous source said that the compromised systems were the Infrastructure Protection (IP) Gateway, which houses critical information about the interdependency of U.S. infrastructure, and the Chemical Security Assessment Tool (CSAT), which houses private sector chemical security plans.
> CSAT is an online portal that contains highly sensitive information that determines which facilities are considered high-risk under the Chemical Facility Anti-Terrorism Standards (CFATS).
> CISA declined to confirm or deny which of their systems were taken offline.
> In late February, CISA had already issued a warning that cyber threat actors are exploiting previously identified vulnerabilities in Ivanti Connect Secure and Ivanti Policy Secure gateways. Ivanti Connect Secure is a widely deployed SSL VPN, while Ivanti Policy Secure (IPS) is a network access control (NAC) solution.
> Now, CISA itself has fallen victim to a cyberattack involving Ivanti products.
CSRB's report on the Exchange Online breach that dropped a couple weeks ago was pretty damning. Microsoft had a situation where a threat actor had access to the entirety of Exchange Online, and possibly their entire cloud. CSRB describes the entire incident as completely avoidable, and resulting from Microsoft's inadequate security culture, and it calls Microsoft out for making public statements about the breach and its response that it knew to be inaccurate.
The ONLY way that breach got detected was because the State department bought the premium package with extra logging that let them see when mailboxes get opened. It turned out, Microsoft had a signing key that could create access tokens for anything in their cloud, and it was stolen by Storm-0558. (More precisely, the key was only supposed to be useful for a portion of their services, but a bug allowed Storm-0558 to bypass that scope limitation.) And they used that to go read the e-mails of the State department and a bunch of other organizations, and private individuals. There was nothing customers could do to prevent the attack, and apparently no other indication in their logs that it was taking place, besides this category of entry that was gatekept behind a premium subscription package.
Microsoft generated the key in 2016 and discontinued it years prior to the incident, but it was never revoked. Microsoft didn't even bother with key rotations anymore after 2021 because one time they fucked it up and it caused an outage, so they decided to just not do that anymore. Also, Microsoft apparently didn't have any means of detecting the obvious use of a zombie key.
Also, Microsoft still doesn't really know how they got the key. They made a blog post about their theory, representing it as something they were highly confident in based on the evidence. After 6 months of pressure from the government, Microsoft finally updated the post to admit that they had no evidence of critical parts of what they claimed, and several key points in their narrative were factually incorrect.
Then earlier this year, Microsoft got hacked AGAIN because they had an unused-but-active test account with a guessable password and no MFA, and it was authorized for access to e-mail boxes of (at a minimum) numerous members of Microsoft senior leadership.
> Microsoft didn't even bother with key rotations anymore after 2021 because one time they fucked it up and it caused an outage, so they decided to just not do that anymore.
Key rotation is almost like restoring from backups. It's an absolutely necessary capability and practice.
You'd be surprised at how little cloud vendors give a shit about security internally. Story time: I recently went ahead and implemented key rotation for one of our authz services, since it had none, and was reprimanded for "not implementing it like Google". Fun fact: Google's jwks.json endpoint claims to be "certs" from the path (https://www.googleapis.com/oauth2/v3/certs). They are not certs - there is no X.509 wrapper, no stated expiration, no trust hierarchy. Clients are effectively blind when performing token validation with this endpoint, and it's really shitty.
Other nonsense I've seen: leaking internally signed tokens for external use (front-channel), JWTs being validated without a kid claim in the header - so there's some sketchy coupling going on, skipping audience validation, etc...
Not much surprises me anymore when it comes to this kinda stuff - internally, I suspect most cloud providers operate like "feature factories" and security is treated as a CYA/least-concern thing. Try pushing for proper authz infrastructure inside your company and see what kinda support you'll get.
Not necessarily. Scheduled key rotation has a lot of conceptual problems:
1. As MS found, revoking old keys is very risky because doing so creates outages. But if you don't do it then changing keys is useless. This isn't a problem specific to Microsoft. Lots of companies have learned this lesson the hard way.
2. It assumes that attackers don't just use stolen keys immediately (e.g. to issue more keys, change passwords, create new accounts etc). In practice they usually do.
3. It assumes that if you change the keys the attackers can't just immediately re-steal the new keys.
So it's only really a useful practice in one very specific scenario: you do something that boots undetected attackers out of your network without realising that's what it did, and the attackers need ongoing access that only that key can provide, and they can't use that key to elevate permissions in a more permanent way like by creating a new account on the system or stealing a user password. Pretty specific scenario.
Unfortunately, key rotation also comes with big downsides. Any software that works with keys has to be built to tolerate a change silently, because now it's a regular occurrence instead of a rare one (where maybe a bit of disruption can be absorbed). That creates complexity and therefore bugs. And because it's a repetitive piece of fiddly and complex work that can break your entire service if you get it wrong it inevitably gets automated, and that in turn means that you end up with a large collection of highly privileged subsystems that have the power to silently change keys in ways admins won't notice because they are expecting it: exactly the sort of thing attackers will immediately focus on.
Overall it's not an obviously winning move. Opportunity cost matters too. Whilst you're setting up all the infrastructure to do this, ironing out the bugs, cleaning up after the outages etc, your competitors might be investing in other kinds of security best practices that are more effective. It's especially useless here because MS don't know how the key was stolen to begin with, so there's no reason to think that if they changed it that would have had any effect. Most likely it could have just been immediately restolen and all the effort would have been theatre.
I've been told by current military folks that they are forced to use outdated windows (the ones without security updates) on official military computers on base. So this is where they access emails, surf the web, and all of that. They had to use IE instead of the evergreen edge browser.
It's well known among the people who serve that it's a joke.
Reminds me of Gary McKinnon, who "hacked" US military computers by using default passwords.... what difference does.it make to use newer OSes if ITops sucks?
Not just on base. I had a conversation with a three star general. He remarked that while our warships had separate software and hardware for systems and fire control, the rest of of the ship’s IT ran on Windows. Supposedly, there was no connection between the two, but LOL.
Why LOL? These are separate systems/networks. Even warships need boring admin things like email, internet, identity management, etc. Plus the systems that go on these ships are heavily customized not just straight out of the box.
It would definitely be possible for government to be good at things ... in the olden days of tech development, very good people were employed and empowered at government positions with technology roles. I'm thinking back to later 90s when I filled out my financial student aid application. That was an _extremly_ complicated web product for the time, built entirely by the government, and it completely worked and was easy to use. Commercial products like turbotax on the web didn't get parity of complexity and robustness for a good decade more than that.
The 'outsource everything' 'not allowed to compete with private industry' mentalities are what has the made the government unable to function in a quality manner ... Its virtually impossible for the government to just hire some people to a team to build some shit -- instead they are _required_ to create bids for contractors to bid on and then incredibly formal contract management processes that are just incredibly disfunctional by design ...
It doesn't have to be this way -- its a political result going back to the 'small government' movement which was ultimately about proving that government has to be bad at everything by imposing rules to ensure that result in as many places as possible ...
More control yes but also: more transparency, more vendor choices across the lifecyle. Also more control over telemetry: the current MS stack uploads far more than necessary to only keep things running; government users should not be surveilled, their data should not be sold, etc.
I'd guess that, before the US ever does anything about its Microsoft Achilles heel, other countries will realize that they face even more threats from depending on Microsoft.
(This applies to a number of reckless tech companies upon which countries and other companies create dependencies. But MS is one of the most concerning to me.)
The federal government is merely experiencing what everyone else does: The IT industry has very low standards:
Look at the vast amount of fraud, theft, slander, cons, mis/disinformation, etc. on the Internet, not to mention consumers and small businesses getting run over by legal corporate behavior such as not providing customer service, locking accounts, holding money, holding data, surveillance and selling data, etc.
It's been said before, but IT needs professional standards that provide actual, real-life security, privacy, etc., just like professional standards for building developers provide actual, safe buildings - imagine a fraud equivalent to cryptocurrency in building architecture and engineering; imagine an SBF. SBF was celebrated and still is by some. Lawyers have professional standards, doctors, accountants, scientists, engineers in most fields, ... the list goes on in every field of human endeavor but IT.
And with standards comes liability. Microsoft shouldn't be able to just say, 'sorry about those hackers'.
Readit News
Regarding security, at the end of the day, the US government is a huge target for adversaries. You can't outsource your security practices & if MS software is really that much worse they should be fixing their purchasing requirements. The reality though is that whatever software the US government would switch to would become the focus of adversarial research.
The government is able to employ tons of smart people that could be making way more money elsewhere.
They might not get the absolute best security people in the world, but they could get good enough - as good as they're getting from MS for a fraction of the price.
Additionally, government salaries aren't terrible when you factor in the pension. Most people want the money now. But if you want financial security in the future - that's a reason a lot of people chose to work for the Fed.
Naturally government pay would lag behind even the more mediocre H1Bs.
State/federal govts receive 10-20% of every single citizens salary on top of being able to literally create money. They aren't hurting for money.
This is entirely caused by corruption and incompetence.
I wouldn’t want to be a Microsoft account executive on the US govt contract right now; they’re about to have a massive load of additional requirements.
And if they don’t play ball, possibly antitrust to weaken their stranglehold on being the only real enterprise player.
I’m not saying any of this is the right approach, but it’s a tool in the governments toolbox.
source: knows some idiots
The problem is that there isn't much in terms of alternatives, especially not if you prefer to have one software / vendor / tech stack.
- In groupware, there used to be Lotus Notes, but that went down the drain years ago. Thunderbird can do everything Outlook can (i.e. provide an email client, calendar and address book), but there is no official(ly supported) Thunderbird server software suite so there's always a potential for subtle bugs between whatever one chooses for directory, email and calendar backends.
- for AD there's obviously Samba but it, again, lacks a management UI that supports all of its features, so yet another potential for issues.
- the Office suite alternatives are even more of a nightmare, both in terms of usability, stability as well as compatibility with the millions of legacy files originating from Office. Or hell, even compatibility with old versions of the same app isn't a given in LibreOffice. (And I'm not sure stuff like MS Access even has a FOSS counterpart)
- And then, there's all the other stuff that integrates with AD for authentication/authorization. In a lot of cases, it's "either use MS AD or you're on your own when you hit issues".
- finally, Windows itself. Essentially, the US Government would have to sink billions of dollars into ReactOS development to make it compatible enough with mainstream Windows versions to run all the legacy software that people use - and no, WINE alone is not enough, not for anything that deals with hardware directly. And I wouldn't assume it's possible to even hire enough developers that are skilled to develop for WINE/ReactOS and fulfill the project requirements of never having been exposed to Windows source code.
Microsoft has achieved an insane amount of vendor lock-in, even Apple with all its financial and technological might or Valve (who invested a huge amount of money and work into getting WINE feature-rich enough to run a ton of AAA games on their Steam Deck) have been able to even come close. They can provide as shitty a service/software as they want, their audience literally has no other choice.
(Me personally, I'll keep my Windows 7 and 10 VMs alive for as long as I can, but no way in hell I'm ever moving to the ad-ridden, bling-bling flashy pseudo-hipster-UI disaster that is Windows 11)
Thunderbird is a lot more reliable than outlook - I use both daily. But yeah there's no 'thunderbird software suite'. I don't think that's really an issue though. Outlook breaks regularly, and our desktop support group had to deploy an 'outlook reset' app because the vendor is unresponsive. Being that unresponsive lessens the attractiveness of a 'software suite'.
>Or hell, even compatibility with old versions of the same app isn't a given in LibreOffice.
If I find MS word can't open a file, Libreoffice always does. I've never had a problem opening a libreoffice writer or MS Word file. I'm no 'power user' though.
The reason businesses and organizations stick to microsoft is because it's what they're used to, not because the apps themselves are better.
As for microsoft as a vendor - they are as obtuse a vendor as I've ever dealt with. Nearly always 100% unresponsive - even with projects involving hundreds of thousands of yearly dollars in licensing fees. I think being unresponsive is some kind of corporate culture thing with them. They're like the McDonalds of IT vendors. You get what's on the menu and that's it. If it doesn't work, well too bad because they know the perceived cost of switching is believed to be too high.
The main problem is the catch-22: there's not much point in developing competitors to Microsoft's stack if businesses aren't going to ever consider them, and government departments won't consider alternatives if they can't get everything from a single vendor in a 100% risk free manner thanks to the "nobody ever got fired for buying Microsoft" and "one throat to choke" mentality you get there.
Governments do this to themselves. The USG doesn't even have to pick Microsoft. They could potentially sign contracts with Apple for workstation hardware and services, that would encourage and feed the alternative ecosystem based around Apple, or they could fund Linux, etc. They don't though. It's just soooo much easier to sign one giant contract, because that minimizes work for them and it's not really important to get value for money if you're in the public sector. Signing a great deal won't get you a big bonus or anything like that, but taking risks can get you blocked from promotions. So, why risk anything?
Unfortunately, these days the browser engine market is pretty lopsided so that strategy would need to be paired with something like a requirement to test in Gecko and WebKit because I trust Google’s long-term management even less than Microsoft.
Also, as someone who actually uses Windows 11 Professional all day, every working day, I honestly haven’t encountered these adverts everyone is always referring to (maybe it’s because I’m not in the US).
As for the UI, initial release didn’t allow ungrouping of Windows on taskbar and context menus hid most things behind an extra click. This was crap, but a major update a while back has resolved these.
I’ve tried Linux, I occasionally need to use it or macOS for development of our cross platform Electron based product, and I would still choose Windows as it really has consistently “just worked” for me since around Windows 2000.
But if macOS or Linux works well for you, I’m happy you’re happy and perhaps you in turn could be happy for people who are happy with Windows (and ideally without thinking they must be deluded or something).
How are you able to conclude that?
Cheaper just to pay up.
Could be the opposite. Every provider is an attack vector of the government's network.
The problem is MS fucked up big time by not only losing a key but also that this key is a master key due to an error.
Not to mention they downplayed the problem afterwards and tried to charge customers for the logs needed to identify the attack.
https://www.military.com/daily-news/2024/02/24/marine-corps-...
and we know that all monopolies require government enforcement to prevent others from competing...
then we know that there's no such thing as a conflict between Microsoft and the us government
the us govt is to serve Microsoft and that's that. any conversation like this about the merits of Microsoft as a market participant are laughable and disingenuous
theres no programmer alive that knows the history of Microsoft that would speak about them as you have just done, as an honest company trying to make a product
actually sick to see how HN has been corrupted
I know the history of MS anti-trust fairly well having grown up during the core part of their monopolist years - now they're part of the familiar oligopaly that's strangling tech.
> and we know that all monopolies require government enforcement to prevent others from competing...
Do we know that? That may be true of government run monopologies, but I'm pretty sure most marketplace monopolies rely on lack of government enforcement. For example, here's some analysis showing how US case law makes the government wary of going after Amazon for driving Quidsi out of business & forcing an acquisition by Amazon: https://cei.org/blog/amazons-private-labels-dont-threaten-co.... Ironically US does enforce rules against dumping from international players which shows that it realizes the harm that such activity can have, it just chooses to allow it for domestic players abusing smaller domestic players.
Dead Comment
A similar claim could obviously be made of Boeing. Just imagine what is happening in their military contracts which we are not allowed to hear about. Looking at the projects which we are allowed to know about, airliners and Boeing's Starliner, clearly Boeing management needs to be put out to pasture.
The core issue is cutting corners, for profit. This is not an issue which is easy to handle in our system. It seems that the best we can do is name and shame. Let's do that at least.
Deleted Comment
> According to an early report on the breach, an anonymous source said that the compromised systems were the Infrastructure Protection (IP) Gateway, which houses critical information about the interdependency of U.S. infrastructure, and the Chemical Security Assessment Tool (CSAT), which houses private sector chemical security plans.
> CSAT is an online portal that contains highly sensitive information that determines which facilities are considered high-risk under the Chemical Facility Anti-Terrorism Standards (CFATS).
> CISA declined to confirm or deny which of their systems were taken offline.
https://securityintelligence.com/news/cisa-hackers-key-syste...
> In late February, CISA had already issued a warning that cyber threat actors are exploiting previously identified vulnerabilities in Ivanti Connect Secure and Ivanti Policy Secure gateways. Ivanti Connect Secure is a widely deployed SSL VPN, while Ivanti Policy Secure (IPS) is a network access control (NAC) solution.
> Now, CISA itself has fallen victim to a cyberattack involving Ivanti products.
The ONLY way that breach got detected was because the State department bought the premium package with extra logging that let them see when mailboxes get opened. It turned out, Microsoft had a signing key that could create access tokens for anything in their cloud, and it was stolen by Storm-0558. (More precisely, the key was only supposed to be useful for a portion of their services, but a bug allowed Storm-0558 to bypass that scope limitation.) And they used that to go read the e-mails of the State department and a bunch of other organizations, and private individuals. There was nothing customers could do to prevent the attack, and apparently no other indication in their logs that it was taking place, besides this category of entry that was gatekept behind a premium subscription package.
Microsoft generated the key in 2016 and discontinued it years prior to the incident, but it was never revoked. Microsoft didn't even bother with key rotations anymore after 2021 because one time they fucked it up and it caused an outage, so they decided to just not do that anymore. Also, Microsoft apparently didn't have any means of detecting the obvious use of a zombie key.
Also, Microsoft still doesn't really know how they got the key. They made a blog post about their theory, representing it as something they were highly confident in based on the evidence. After 6 months of pressure from the government, Microsoft finally updated the post to admit that they had no evidence of critical parts of what they claimed, and several key points in their narrative were factually incorrect.
Then earlier this year, Microsoft got hacked AGAIN because they had an unused-but-active test account with a guessable password and no MFA, and it was authorized for access to e-mail boxes of (at a minimum) numerous members of Microsoft senior leadership.
Microsoft has got serious problems.
edit: I keep futzing with my phrasing. Those wanting a much better account should just read the report, since it has a great deal more nuance and information. https://www.cisa.gov/sites/default/files/2024-04/CSRB_Review...
https://www.theverge.com/2022/3/25/22995144/microsoft-foreig...
https://en.wikipedia.org/wiki/Microsoft_licensing_corruption...
https://www.wsj.com/articles/BL-CJB-17439
Key rotation is almost like restoring from backups. It's an absolutely necessary capability and practice.
Other nonsense I've seen: leaking internally signed tokens for external use (front-channel), JWTs being validated without a kid claim in the header - so there's some sketchy coupling going on, skipping audience validation, etc...
Not much surprises me anymore when it comes to this kinda stuff - internally, I suspect most cloud providers operate like "feature factories" and security is treated as a CYA/least-concern thing. Try pushing for proper authz infrastructure inside your company and see what kinda support you'll get.
1. As MS found, revoking old keys is very risky because doing so creates outages. But if you don't do it then changing keys is useless. This isn't a problem specific to Microsoft. Lots of companies have learned this lesson the hard way.
2. It assumes that attackers don't just use stolen keys immediately (e.g. to issue more keys, change passwords, create new accounts etc). In practice they usually do.
3. It assumes that if you change the keys the attackers can't just immediately re-steal the new keys.
So it's only really a useful practice in one very specific scenario: you do something that boots undetected attackers out of your network without realising that's what it did, and the attackers need ongoing access that only that key can provide, and they can't use that key to elevate permissions in a more permanent way like by creating a new account on the system or stealing a user password. Pretty specific scenario.
Unfortunately, key rotation also comes with big downsides. Any software that works with keys has to be built to tolerate a change silently, because now it's a regular occurrence instead of a rare one (where maybe a bit of disruption can be absorbed). That creates complexity and therefore bugs. And because it's a repetitive piece of fiddly and complex work that can break your entire service if you get it wrong it inevitably gets automated, and that in turn means that you end up with a large collection of highly privileged subsystems that have the power to silently change keys in ways admins won't notice because they are expecting it: exactly the sort of thing attackers will immediately focus on.
Overall it's not an obviously winning move. Opportunity cost matters too. Whilst you're setting up all the infrastructure to do this, ironing out the bugs, cleaning up after the outages etc, your competitors might be investing in other kinds of security best practices that are more effective. It's especially useless here because MS don't know how the key was stolen to begin with, so there's no reason to think that if they changed it that would have had any effect. Most likely it could have just been immediately restolen and all the effort would have been theatre.
https://securityintelligence.com/news/cisa-hackers-key-syste...
What incentive do cloud providers have to add extra security to the government stuff, while not also adding that extra security to the public stuff?
"Oh, yeah, we have more security over in the thing you can't use, but you can trust us with your data anyway."
It's well known among the people who serve that it's a joke.
Source: was IT on a destroyer
Might be very expensive initially, but then they'd have complete control.
The 'outsource everything' 'not allowed to compete with private industry' mentalities are what has the made the government unable to function in a quality manner ... Its virtually impossible for the government to just hire some people to a team to build some shit -- instead they are _required_ to create bids for contractors to bid on and then incredibly formal contract management processes that are just incredibly disfunctional by design ...
It doesn't have to be this way -- its a political result going back to the 'small government' movement which was ultimately about proving that government has to be bad at everything by imposing rules to ensure that result in as many places as possible ...
(This applies to a number of reckless tech companies upon which countries and other companies create dependencies. But MS is one of the most concerning to me.)
Look at the vast amount of fraud, theft, slander, cons, mis/disinformation, etc. on the Internet, not to mention consumers and small businesses getting run over by legal corporate behavior such as not providing customer service, locking accounts, holding money, holding data, surveillance and selling data, etc.
It's been said before, but IT needs professional standards that provide actual, real-life security, privacy, etc., just like professional standards for building developers provide actual, safe buildings - imagine a fraud equivalent to cryptocurrency in building architecture and engineering; imagine an SBF. SBF was celebrated and still is by some. Lawyers have professional standards, doctors, accountants, scientists, engineers in most fields, ... the list goes on in every field of human endeavor but IT.
And with standards comes liability. Microsoft shouldn't be able to just say, 'sorry about those hackers'.