> Once Microsoft's eBPF support for Windows becomes production-ready, Windows security software can be ported to eBPF as well.
This doesn’t seem grounded in reality. If you follow the link to the “hooks” that Windows eBPF makes available [1], it’s just for incoming packets and socket operations. IOW, MS is expecting you to use the Berkeley Packet Filter for packet filtering. Not for filtering I/O, or object creation/use, or any of the other million places a driver like Crowdstrike’s hooks into the NT kernel.
In addition, they need to be in the kernel in order to monitor all the other 3rd party garbage running in kernel-space. ELAM (early-launch anti-malware) loads anti-malware drivers first so they can monitor everything that other drivers do. I highly doubt this is available to eBPF.
If Microsoft intends eBPF to be used to replace kernel-space anti-malware drivers, they have a long, long way to go.
Yes, we know eBPF must attach to equivalent events to Linux, but given there are already many event sources and consumers in Windows, the work is to make eBPF another consumer -- not to invent instrumentation frameworks from scratch.
Just to use an analogy: Imagine people do their banking on JavaScript websites with Google Chrome, but if they use Microsoft Edge it says "JavaScript isn't supported, please download and run this .EXE". I'm not sure we'd be asking "if" Microsoft would support JavaScript (or eBPF), but "when."
This assumes eBPF becomes the standard. It's not clear Microsoft wants that. They could create something else which integrates with dot net and push for that instead.
Also this problem of too much software running in the kernel in an unbounded manner has long existed. Why should Microsoft suddenly invest in solving it on Windows?
Windows development on eBPF is slower than Linux development on eBPF, so it will never be supported. A source code user licensee could develop it faster, but who licenses Windows source and already has great eBPF experience?
Microsoft already has an extensible file system filter capability in place, which is what current AV uses. Does it make sense to add eBPF on top of that and if so, are there any performance downsides, like we see with file system filters?
They've done a technology transition once already from legacy file system filter drivers to the minifilter model. If they see enough benefit to another change, it wouldn't be unprecedented.
Mind you, it looks like after 20-ish years Windows still supports loading legacy filter drivers. Given the considerable work that goes into getting even a simple filesystem minifilter driver working reliably, it's safe to assume that we'd be looking at a similarly protracted transition period.
As to the performance, I don't think the raw infrastructure to support minifilters is the major performance hit. The work the drivers themselves end up doing tends to be the bigger hit in my experience.
Keep in mind they don't just allow any old code to execute in the kernel.
They do have rigorous tests (WHQL), it's just Crowdstrike decided that was too burdensome for their frequent updates, and decided to inject code from config files (thus bypassing the control).
Doubt it. Microsoft is clearly over Windows. They continue to produce it but every release feels like "Ugh, fine, since you are paying me a ton of money."
Internally, Microsoft is running more and more workloads on Linux and externally, I've had .Net team tell me more than once that Linux is preferred environment for .Net. SQL Server team continues to push hard for Linux compatibility with every release.
EDIT: Windows Desktop gets more love because they clearly see that as important market. I'm talking more Windows Server.
I hate to dispute with someone like Brendan Gregg, but I'm hoping vendors in this space take a more holistic approach to investigating the complete failure chain. I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure. It may be true, but if we don't do the analysis we could leave ourselves open to blindspots. There may also be plenty of alternative approaches that should be considered and appropriately discarded.
I think the part I specifically dispute is the only negative outcome is wasted CPU cycles. That's likely the case for the class of bug, but there are plenty of failure modes where a bad ruleset could badly brick a system and make it hard to recover.
That's not to say eBPF based security modules isn't the right choice for many vendors, just that let's understand what risks they do and do not avoid, and what part of the failure chain they particularly address.
Just because you have not been aware of the discussions on this topic that have been happening for years, doesn't mean that they haven't been happening. This isn't some new analysis formed 3 days after an incident, this is the generally accepted consensus among many experts who have been working in the space, introducing these new APIs specifically to improve stability, security, etc. of systems.
> I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure.
Microsoft has been working on eBPF for a few years at least.
If you're really concerned, they have discussions and communication channels where you're invited to air your concerns. They're listed on their github:
This isn't right. If I need a system to run with a piece of code, then it shouldn't run at all if that piece of code is broken. Ignoring the failure is perverse. Let's say that the driver code ensures that some medical machine has safety locks (safeguards) in place to make sure that piece of equipment won't fry you to a crisp; I'd prefer that the whole thing not run at all rather than blithely operate with the safeguards disabled.
It's turtles all the way down.
I think the premise is false? It's up to the eBPF implementor what to do in the case of invalid input; the kernel could choose to perform a controlled shutdown in that case. (I have no idea what e.g. Linux actually does here, but one could imagine worlds where the action it takes on invalid input is configurable.)
Also your statement is sometimes not true, although I certainly sympathise in the mainline case. In some contexts you really do need to keep on trucking. The first example to spring to mind is "the guidance computers on an automated Mars lander"; the round-trip to Earth is simply too long to defer responsibility in that case. If you shut down then you will crash, but if you do your best from a corrupted state then you merely probably crash, which is presumably better.
> I have no idea what e.g. Linux actually does here
If you attempt to load an eBPF program that the verifier rejects, the syscall to load it fails with EINVAL or E2BIG. What your user-space program then does is up to you, of course.
The medical machine software should just refuse to run with an error message if a critical driver was not loaded.
The OS bricking is causing way more trouble where an IT technician now needs to fix something where it otherwise would just be updating the faulty driver...
Also does your car not start if you are missing water for the wiper?
3rd party hooking into kernel is 3rd party responsibility. It is like equipping your car with LPG - THAT hooks into engine (kernel). And When I had a faulty gas pressure sensor then my car actually halted (BSOD if you will) instead of automatically failing over to gasoline as it is by design.
You can argue that car had no means to continue execution but kernel has, however invalid kernel state can cause more corruption down the road. Or as parent even points out - carry out lethal doses of something.
I agree that some system components should be treated as critical no matter what, but the software at issue in this case (Falcon Sensor or Antivirus more generally) is precautionary and only best effort anyways. I would wager the vast majority of the orgs affected on Friday would have preferred the marginally increased risk of a malware attack or unauthorized use over a 24 hour period instead of the total IT collapse they experienced. Further, there's no reason the bug HAD to cause a BSOD, it's possible the systems could have kept on trucking but with an undefined state and limitless consequences. At least with eBPF you get to detect a subset of possible errors and make a risk management decision based on the result.
I'm with you.
What's critical, and what's not? Is it a big thing, or not a big thing? Is this particular machine more critical than the one over there?
Security systems need to be at the lowest level, or else some shifty bastard will find a path around them. If it's at the lowest level, the downside of a failure is catastrophic, as we experienced last Friday. The carnage here is ultimately on CrowdStrike. The testing must have been slapdash at best, and missing at worst. eBPF changes nothing. The question is: should we fail, or carry on? eBPF doesn't help with that decision, it only determines the outcome from a system perspective. Any decision is a value judgement; it might be right or wrong, and its outcome either benign or deadly.
Choices!
I like how Unison works for this reason. You call functions by cryptographic hash, so you have some assurance that you're calling the same function you called yesterday.
Updates would require the caller to call different functions which means putting the responsibility in the hands of the caller, where it should be, instead of on whoever has a side channel to tamper with the kernel.
You end up with the work-perfectly-or-not-at-all behavior that you're after because if the function that goes with the indicated hash is not present, you can't call it, and if it is present you can't call it in any way besides how it was intended
The system clearly already behaves that way (i.e. ignores failure) - after all, the fix was to simply delete the offending file. If that's an option, then loader can do that too. It can and perhaps even is smarter, such as "fallback onto previous version".
Furthermore, the reaction to a malformed state need not be "ignore". It could disable restricted user login; or turn off the screen.
If the worry is that this is viable to abuse by malware, well, if the malware can already rewrite the on-disk files for the AV, I wonder whether it's really a good idea to trust the system itself to be able to deal with that. It'd probably be safer to just report that up the security foodchain, and potentially let some external system take measures such as disable or restrict network access. Better yet, such measures don't even require the same capabilities to intervene in the system, merely to observe - which makes the AV system less likely to serve as a malware vector itself or to cause bugs like this.
If the failed system is a security module, I think that's absolutely correct. If the system runs, without the security module, well, that's like forgetting to pack condoms on Shore Leave. You'll likely be bringing something back to the ship with you.
Someone needs to be testing the module, and the enclosing system, to make sure it doesn't cause problems.
I suspect that it got a great deal of automated unit testing, but maybe not so much fuzz and monkey (especially "Chaos Monkey"-style) testing.
Interesting analogy, but yes. If the module *is* necessary, well, it's necessary and nothing should work without it. Testing must have been a mess here.
> In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code.
eBPF is fantastic, and it can be used for many purposes and improve a lot of things, but this is IMO overselling it. Assuming that BPF itself it free of bugs, it’s still a rather large sprawl of kernel hooks, and those hooks invoke eBPF code, which can call right back into the kernel. Here’s a list:
bpf_probe_read_kernel() is particularly heavily used, and it is not safe. It tries fairly hard not to OOPS or crash, but it is definitely not perfect.
The rest of that list contains plenty of this that will easily take down a system, even if it doesn’t actually oops or panic in the process.
And, of course, any tool that detects userspace “malicious behavior” and stops it can start calling everything malicious, and the computer becomes unusable.
Meanwhile, eBPF has no real security model on the userspace side. Actual attachment of an eBPF program goes through the bpf() syscall, not through sensibly permissioned operations on the underlying kernel objects being attached to, and there is nothing whatsoever that confines eBPF to, say, a container that uses it. (See bpf_probe_read_kernel() -- it's fundamentally able to read all kernel memory.)
So, IMO, most of the benefit of eBPF over ordinary kernel C code is that eBPF is kind of like writing code in a safe language with a limited unsafe API surface. It's a huge improvement for this sort of work, but it is not perfect by any means.
> The verifier is rigorous -- the Linux implementation has over 20,000 lines of code
The verifier is absurdly complex. I'd rather see something based on formal methods than 20kLOC of hand-written logic.
I'm not sure that "panic" is the right word here. bpf_probe_read_kernel boils down to copy_from_kernel_nofault, which checks for an "allowed" address and then does the access. Any page faults turn into error returns instead of OOPSes. x86 disallows user addresses, the vsyscall page, and non canonical addresses.
Doing this from bpf assumes that all "allowed" addresses are side-effect-free and will either succeed or cleanly fault. Off the top of my head, MMIO space (including, oddities like the APIC page on CPUs that still have that) and TDX memory are not in this category.
> eBPF programs cannot crash the entire system because they are safety-checked by a software verifier and are effectively run in a sandbox.
Isn’t one of the purposes of an OS to police software? I get that this has to do with the OS itself, but what does watching the watchers accomplish other than adding a layer which must then be watched?
Why not reduce complexity instead of naively trusting that the new complexity will be better long term?
eBPF isn't "watching the watchers" it's just a tool that lets other tools access low-level things in the kernel via a very picky sandbox. Think of it like this:
Old way: Load kernel driver, hook into bazillions of system calls (doing whatever it is you want to do), pray you don't screw anything up (otherwise you can get a panic though not necessarily--Linux is quite robust).
eBPF way: Just ask eBPF to tell you what you want by giving it some eBPF-specific instructions.
Right? I might spend a few minutes seeing if an AI chatbot can explain all the justifications that lead to using something like CrowdStrike in the first place.
This sounds like a cool technology, but this was the really egregious problem:
> There are other ways to reduce risks during software deployment that can be employed as well: canary testing, staged rollouts, and "resilience engineering" in general
You don't need a new technology to implement basic industry-standard quality control
Maybe we should start taking Fridays off to commemorate the event, which probably would have been less bad if more people spent less time with their nose to the grindstone and had more time to stop and think about how it all was shaping up and how they could influence that shape.
```The verifier is rigorous -- the Linux implementation has over 20,000 lines of code -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). The safety this provides is a key benefit of eBPF, along with heightened security and lower resource usage.
```
Wow, 20k is not exactly encouraging. Besides the extra attack surface, who can vouch for such a large code base?
I had exactly the same thought. I don’t know if that 20k number was supposed to inspire confidence, but for me it did the opposite. It would have inspired confidence if it was 300 lines of code.
My impression is that the WebAssembly verifier is much simpler.
Readit News
This doesn’t seem grounded in reality. If you follow the link to the “hooks” that Windows eBPF makes available [1], it’s just for incoming packets and socket operations. IOW, MS is expecting you to use the Berkeley Packet Filter for packet filtering. Not for filtering I/O, or object creation/use, or any of the other million places a driver like Crowdstrike’s hooks into the NT kernel.
In addition, they need to be in the kernel in order to monitor all the other 3rd party garbage running in kernel-space. ELAM (early-launch anti-malware) loads anti-malware drivers first so they can monitor everything that other drivers do. I highly doubt this is available to eBPF.
If Microsoft intends eBPF to be used to replace kernel-space anti-malware drivers, they have a long, long way to go.
[1]: https://microsoft.github.io/ebpf-for-windows/ebpf__structs_8...
Just to use an analogy: Imagine people do their banking on JavaScript websites with Google Chrome, but if they use Microsoft Edge it says "JavaScript isn't supported, please download and run this .EXE". I'm not sure we'd be asking "if" Microsoft would support JavaScript (or eBPF), but "when."
Also this problem of too much software running in the kernel in an unbounded manner has long existed. Why should Microsoft suddenly invest in solving it on Windows?
Mind you, it looks like after 20-ish years Windows still supports loading legacy filter drivers. Given the considerable work that goes into getting even a simple filesystem minifilter driver working reliably, it's safe to assume that we'd be looking at a similarly protracted transition period.
As to the performance, I don't think the raw infrastructure to support minifilters is the major performance hit. The work the drivers themselves end up doing tends to be the bigger hit in my experience.
Some background for the curious:
https://www.osr.com/nt-insider/2019-issue1/the-state-of-wind...
They do have rigorous tests (WHQL), it's just Crowdstrike decided that was too burdensome for their frequent updates, and decided to inject code from config files (thus bypassing the control).
The fault here is entirely with Crowdstrike.
Internally, Microsoft is running more and more workloads on Linux and externally, I've had .Net team tell me more than once that Linux is preferred environment for .Net. SQL Server team continues to push hard for Linux compatibility with every release.
EDIT: Windows Desktop gets more love because they clearly see that as important market. I'm talking more Windows Server.
I think the part I specifically dispute is the only negative outcome is wasted CPU cycles. That's likely the case for the class of bug, but there are plenty of failure modes where a bad ruleset could badly brick a system and make it hard to recover.
That's not to say eBPF based security modules isn't the right choice for many vendors, just that let's understand what risks they do and do not avoid, and what part of the failure chain they particularly address.
Microsoft has been working on eBPF for a few years at least.
https://opensource.microsoft.com/blog/2021/05/10/making-ebpf...
https://lwn.net/Articles/857215/
If you're really concerned, they have discussions and communication channels where you're invited to air your concerns. They're listed on their github:
https://github.com/microsoft/ebpf-for-windows
Who knows, maybe they already have answers to your concerns. If not, they can address them there.
Also your statement is sometimes not true, although I certainly sympathise in the mainline case. In some contexts you really do need to keep on trucking. The first example to spring to mind is "the guidance computers on an automated Mars lander"; the round-trip to Earth is simply too long to defer responsibility in that case. If you shut down then you will crash, but if you do your best from a corrupted state then you merely probably crash, which is presumably better.
If you attempt to load an eBPF program that the verifier rejects, the syscall to load it fails with EINVAL or E2BIG. What your user-space program then does is up to you, of course.
3rd party hooking into kernel is 3rd party responsibility. It is like equipping your car with LPG - THAT hooks into engine (kernel). And When I had a faulty gas pressure sensor then my car actually halted (BSOD if you will) instead of automatically failing over to gasoline as it is by design.
You can argue that car had no means to continue execution but kernel has, however invalid kernel state can cause more corruption down the road. Or as parent even points out - carry out lethal doses of something.
Updates would require the caller to call different functions which means putting the responsibility in the hands of the caller, where it should be, instead of on whoever has a side channel to tamper with the kernel.
You end up with the work-perfectly-or-not-at-all behavior that you're after because if the function that goes with the indicated hash is not present, you can't call it, and if it is present you can't call it in any way besides how it was intended
Furthermore, the reaction to a malformed state need not be "ignore". It could disable restricted user login; or turn off the screen.
If the worry is that this is viable to abuse by malware, well, if the malware can already rewrite the on-disk files for the AV, I wonder whether it's really a good idea to trust the system itself to be able to deal with that. It'd probably be safer to just report that up the security foodchain, and potentially let some external system take measures such as disable or restrict network access. Better yet, such measures don't even require the same capabilities to intervene in the system, merely to observe - which makes the AV system less likely to serve as a malware vector itself or to cause bugs like this.
If the failed system is a security module, I think that's absolutely correct. If the system runs, without the security module, well, that's like forgetting to pack condoms on Shore Leave. You'll likely be bringing something back to the ship with you.
Someone needs to be testing the module, and the enclosing system, to make sure it doesn't cause problems.
I suspect that it got a great deal of automated unit testing, but maybe not so much fuzz and monkey (especially "Chaos Monkey"-style) testing.
It's a fuzzy, monkey-filled world out there...
eBPF is fantastic, and it can be used for many purposes and improve a lot of things, but this is IMO overselling it. Assuming that BPF itself it free of bugs, it’s still a rather large sprawl of kernel hooks, and those hooks invoke eBPF code, which can call right back into the kernel. Here’s a list:
https://www.man7.org/linux/man-pages/man7/bpf-helpers.7.html
bpf_probe_read_kernel() is particularly heavily used, and it is not safe. It tries fairly hard not to OOPS or crash, but it is definitely not perfect.
The rest of that list contains plenty of this that will easily take down a system, even if it doesn’t actually oops or panic in the process.
And, of course, any tool that detects userspace “malicious behavior” and stops it can start calling everything malicious, and the computer becomes unusable.
Meanwhile, eBPF has no real security model on the userspace side. Actual attachment of an eBPF program goes through the bpf() syscall, not through sensibly permissioned operations on the underlying kernel objects being attached to, and there is nothing whatsoever that confines eBPF to, say, a container that uses it. (See bpf_probe_read_kernel() -- it's fundamentally able to read all kernel memory.)
So, IMO, most of the benefit of eBPF over ordinary kernel C code is that eBPF is kind of like writing code in a safe language with a limited unsafe API surface. It's a huge improvement for this sort of work, but it is not perfect by any means.
> The verifier is rigorous -- the Linux implementation has over 20,000 lines of code
The verifier is absurdly complex. I'd rather see something based on formal methods than 20kLOC of hand-written logic.
Doing this from bpf assumes that all "allowed" addresses are side-effect-free and will either succeed or cleanly fault. Off the top of my head, MMIO space (including, oddities like the APIC page on CPUs that still have that) and TDX memory are not in this category.
Isn’t one of the purposes of an OS to police software? I get that this has to do with the OS itself, but what does watching the watchers accomplish other than adding a layer which must then be watched?
Why not reduce complexity instead of naively trusting that the new complexity will be better long term?
Old way: Load kernel driver, hook into bazillions of system calls (doing whatever it is you want to do), pray you don't screw anything up (otherwise you can get a panic though not necessarily--Linux is quite robust).
eBPF way: Just ask eBPF to tell you what you want by giving it some eBPF-specific instructions.
There's a rundown on how it works here: https://ebpf.io/what-is-ebpf/
> …via a very picky sandbox…
When the eBPF is a CrowdStrike mechanism, and eBPF is “picky,” it is clearly “watching the watchers.”
> There are other ways to reduce risks during software deployment that can be employed as well: canary testing, staged rollouts, and "resilience engineering" in general
You don't need a new technology to implement basic industry-standard quality control
My impression is that the WebAssembly verifier is much simpler.
Deleted Comment