For anybody unfamiliar with this, as I was, this appears to refer to Intel's Indirect Branch Tracking feature[1] (and the equivalent on ARM, BTI). The idea is that an indirect branch can only pass control to a location that starts with an "end branch" instruction. An indirect branch is one that jumps to a location whose value is loaded or computed from either a register or memory address: think calling a function pointer in C.
Without IBT, you'd have this equivalence between C and assembly:
main() {
void (*f)();
f = foo;
f();
}
void foo() { }
---
main:
movl $foo, %edx
call *%edx
ret
foo:
ret
If IBT is enabled, the above code triggers an exception because foo doesn't begin with an "end branch" instruction. When IBT is enabled by the compiler, the above code gets assembled as:
main:
endbr64
movl $foo, %edx
call *%edx
ret
foo:
endbr64
ret
Now the compiler inserts endbr64 at the start of each function prologue. The reason for this feature, is to use as a defense in depth against JOP, and COP attacks, as it means that the only "widgets" available to you are entire functions, which can be far harder to exploit and chain.
There's a good question in the comments there that I still don't see the answer to. How does this work if there's an interrupt between the branch and the endbranch? Does the OS need to save/restore the "branchness" bit?
Various architectures do other interesting things with NOPs, IIRC one convention on PowerPC had something vaguely related to debugging or tracing (I can't remember the details or find any references right now).
It was an old joke that the opposite of "goto" is "come from", or that if goto is considered harmful, nobody said anything about a "come from". Marking something as a branch target reminds me of this.
Interesting. Seems like enforcement on Intel CPUs is supported since Tiger Lake (so ~2020). Windows has basically the same feature implemented in software since 2015, called Control Flow Guard [1]. I wonder what the story there is, and if Windows has any plans to (get everyone to) switch to the hardware version once those CPUs have sufficient market share.
Windows also recently implemented a far better version of this called Extended Flow Guard (XFG) that not only checks whether the location is a valid destination, but also whether it's a valid destination for that specific source.
For example, for any virtual function call or function pointer call, the destination must have a correct tag with the hash of the arguments. It's much more secure, and also faster, since loading the tag from memory can be merged with loading the actual code after it.
It looks like endbr64 is a 4-byte instruction. That could be a significant code size overhead for jump tables with lots of targets: https://godbolt.org/z/xTPToaddh
OpenBSD disables jump tables in Clang on amd64 due to IBT, some architectures also had jump tables disabled as part of the switch to --execute-only ("xonly") binaries by default, e.g: powerpc64/sparc64/hppa.
As to why they're not always called directly, imagine some code like this:
int FooWithoutChecks(void *p);
int Foo(void *p) {
if (p == NULL) return -1;
return FooWithoutChecks(p);
}
In general the caller is expected to call Foo if they aren't sure if the pointer is nullable, or if they already know that pointer is not null (e.g. because they already checked it themselves) they can call FooWithoutChecks and avoid a null check that they know will never be true.
The naive way to emit assembly for this is to actually emit two separate functions, and have Foo call FooWithoutChecks the usual way. But notice that the FooWithoutChecks function call is a tail call, so the compiler can use tail call optimization. To do this it would inline FooWithoutChecks into Foo itself, so the compiler just emits code for Foo with the logic in FoowithoutChecks inlined into Foo. This is nice because now when you call Foo, you avoid a call/ret instruction, so you save two instructions on every call to Foo. But what if someone calls FooWithoutChecks? Simple, you just call at the offset into Foo just past the pointer comparison. This actually just works because Foo already has a ret instruction, so the call to FooWithoutChecks will just reuse the existing ret. This optimization also saves some space in the binary which has various benefits in and of itself.
The example here with the null pointer check is kind of contrived, but this kind of pattern happens a LOT in real code when you have a small wrapper function that does a tail call to another function, and isn't specific to pointer checks.
> Why should every function start with endbr64 command? Aren't functions usually called directly?
They're usually called directly, but unless the compiler can prove that they always are (e.g., if they're static and nothing in the same file takes the address), endbr64 is required.
> Also, is it required to insert endbr64 command after function calls (for return address)?
No, IBT is only for jmp and call. SS is the equivalent mechanism for ret.
C allows for any function to be called via a function pointer, and functions can be in different translation units, so the compiler can't simply assume that a function will never be called indirectly and has to pessimistically insert endbr64 in order to maintain a reasonable ABI.
And no, as I understand it, this is only for branch/calls not returns.
Theo had to get his digs in against Linux in that announcement. Why not just focus on what OpenBSD is doing, and maybe contrast it to what Linux does without the speculation that they will still be doing the same thing in 20 years.
He's unquestionably brilliant, but I've had a few encounters with him on the mailing lists and he is so quick to take offense where none was meant and drop into name-calling and insults. I don't really get it. He may have some deep insecurities.
It's an important comparison of the mechanisms, even in 2023, you can still find binaries on modern Linux distributions with executable stacks due to the fail-open design, 20 years later.
The fact that Linux hasn't learned the right lessons in 20 years, and has chosen to "double down" in respect to IBT/BTI, does not inspire confidence that they will ever fix it. I'd say his 20 year estimate was in fact being pretty generous given the evidence available.
It's the price you pay for never-break-userspace. OpenBSD is fine with the very small probability that an executable which doesn't do branch tracking will fail to run under the enforced rules. The answer to that is to recompile because you've still got the source, and if not, well, tough cookies.
> It's an important comparison of the mechanisms, even in 2023, you can still find binaries on modern Linux distributions with executable stacks due to the fail-open design, 20 years later.
Unfortunately, for C code using GCC’s nested functions extension (or for languages that want to be ABI-compatible with C and support nested functions, like that paragon of advanced features called Pascal /s ), there’s no other compilation strategy in current ABIs. The patches to switch C (and not just Ada) to function descriptors[1] with an ABI break have been sitting on the GCC mailing list since approximately forever[2], but it doesn’t seem like there’s been any progress.
[1] The strategy is basically to compile (*fp)() not as
thus essentially inlining the (currently stack-allocated) closure calling thunk at all indirect call sites. It is ABI-compatible on x86 and x86-64 with all code that does not involve nested functions, place functions at odd addresses, or tag function pointers itself (and I think with all arm64 and riscv code, although arm32’s usage of the low pointer bit for Thumb interworking is bound to make this trickier).
The funny thing is that this attitude towards breaking changes is one of the reasons why Theo is able to make this comment at all. If he would allow breaking changes then OpenBSD adoption likely would be higher and that in turn would cause him to resist the kind of things that Linux would not be able to get away with.
It's clearly different philosophies leading to different outcomes with neither of them clearly better than the other, it just depends on what you need. It would be possible to make that statement in a more graceful way.
That part doesn't look like a "dig" or an insult to me.
It seems like a reasonable, relevant, and plausible assessment of how the long-term outcomes may likely differ between OpenBSD's stricter approach versus a looser approach, specifically when it comes to the degree of security offered (which is one of OpenBSD's main focuses), based on a past situation that's similar.
How do you know that you aren't being, to use your words, "quick to take offense where none was meant" in this case?
That's the problem with many brilliant people: what they perceive as their interlocutors being deliberately obtuse on some completely obvious point is actually their interlocutors being just as smart as they always are on some point that is not obvious at all to them.
Perception of relative intelligence or sensible decision making is irrelevant. Just because you think you're doing a better job doesn't mean you need to shit on the other person.
You could not mention Linux at all, or you could even say "we think this is better than Linux's approach because of X" and it would be a great improvement.
I have always found it interesting that Rust purposefully avoided doing language comparisons - "we're better than Python like this and better than C like that". Their message purposefully avoided any positioning of it as a competition, instead focusing just on articulating Rust's value. It was an eye opening approach given our instinct is normally to pit things against each other.
This is my main takeaway too. As a one time OpenBSD enthusiast (and still admirer), now I'm a bit older I find the continual smugness starts to grate.
Truth is, Linux has a lot more constraints on how it can implement something because it has users. Users that have all sorts of different ways they need it to work.
I think it's great that he's calling Linux's choices out. TBH Theo's attitude is borne of Linux's culture - most older school sec people have learned that this is the only way to get things to improve.
I’d just like to interject for a moment. What you’re referring to as Linux, is in fact, NotOpenBSD/Linux, or as I’ve recently taken to calling it, Linux as opposed to OpenBSD…
I still run OpenBSD where I can, especially where security is more important. Yes, it's still missing A LOT of functionally compared to other UNIX-like systems, but security bases tend to be well covered.
I find OpenBSD's hardware support especially lacking. It doesn't really work that well on at least 3 devices where I tried it on (all Dell laptops from various generations, 3-10 years old), whereas Linux runs perfectly out-of-the-box on all three.
Which is sad, as I kinda like the *BSD approach to things
Not my experience at all, it works very well with a new Acer laptop I own: the graphics work (Intel Xe - 12th gen processor), audio, touchpad, keyboard (and special keyboard keys like brightness), wifi... All I had to do is to download the firmware with fw_update, nothing more.
Also I was pleasantly surprised to hear they support Apple M1/M2 Macs. Asahi Linux gets a lot of press around here but I had no idea OpenBSD supported it.
Sure. Poor SMP support (but this has improved heavily over the years), ancient file system, no Bluetooth (not important if you don't need this), reduced performance (due to a lack of optimizations and security mitigations overhead), limited Wi-Fi support (this is for numerous reasons, but it's better than other BSDs)...
I could go on, but, for my needs, it works very well and some of its simplicities are a godsend.
I don't really buy their approach to security honestly. Trying to fix all bugs is great, but they provide little to prevent unknown bugs bing exploited (pledge is nice for software that opts in to use it, but otherwise not so much). I'd love to see them implement something like AppArmor with their approach, it would probably be amazing.
I actually think NetBSD is a pretty interesting alternative, it has some nice security features like veriexec that don't get talked about much.
I think in the past they tried to fix all the bugs, and realized they couldn't, so they started to build all sorts of mitigations in the same vein as the one you see posted here today. As for pledge, and the related mitigations, yes, they're not useful if you don't use them, but I see this as them innovating in the space and giving application developers more tools to build hardened applications.
I see tools like AppArmor as band-aids to fix problems that shouldn't exist in the first place. The problem with these approaches are the band-aids tend to break things in unexpected ways and when that happens they simply get removed and unused.
Is this protection really all that helpful? Surely there are functions you can call into the top of to do your diabolical deeds for you.
It would be more helpful if callers would store some machine specific hash of the function prototype and the function itself would check the hash, so that you could only redirect to calling a function with the right signature.
But that would also increase the overhead further. Already this is bad enough that it makes jump tables unattractive (which is too bad, considering the usually jump tables have little to no risk of control flow redirection).
Windows does this in software, since approximately 8 years.
An advantage of the software solution is that you don't need to have the feature compiled into every library for it to work, you just lose protection in those parts. That makes for a much quicker rollout. Also faster iteration times, in the Windows Insider Preview you can get the extended version that also checks that the hashed function signature matches.
Nope. Here's the actual problem, in these crappy languages it's really easy for mistakes to result in a stack smash, so, these types of hacks aim to make it harder for the bad guys to turn that into arbitrary remote code execution. Not impossible, just harder. Specifically in this case the idea is that they won't be able to abuse arbitrary bits of function without calling the whole function, at a cost of some hardware changes and emitting unnecessary code. So maybe they can't find a whole function which works for them and they give up.
Using better languages makes the entire problem disappear. You don't get a stack smash, the resulting opportunities for remote code execution disappear.
It suggests that maybe the "C magically shouldn't have Undefined Behaviour" people were onto something after all. Maybe C programmers really are so wedded to this awful language that just being much slower than Python wouldn't deter them. There is still the problem that none of them can agree how this should work, but if they'll fund it maybe it's worth pursuing to find out how much they will put up with to keep writing C.
It's only irrelevant if the hardware solution is available on all the supported architectures/systems. As long as it's not, the software version must be maintained anyway, and might suffer from bitrot if it's no longer exercised on the major architectures.
Readit News
Without IBT, you'd have this equivalence between C and assembly:
If IBT is enabled, the above code triggers an exception because foo doesn't begin with an "end branch" instruction. When IBT is enabled by the compiler, the above code gets assembled as: Now the compiler inserts endbr64 at the start of each function prologue. The reason for this feature, is to use as a defense in depth against JOP, and COP attacks, as it means that the only "widgets" available to you are entire functions, which can be far harder to exploit and chain.[1]: https://www.intel.com/content/dam/develop/external/us/en/doc...
https://en.m.wikipedia.org/wiki/COMEFROM
COMEFROM considered harm-mitigating
It ingeniously makes Return Oriented Programming (ROP) a lot harder.
1: https://learn.microsoft.com/en-us/windows/win32/secbp/contro...
For example, for any virtual function call or function pointer call, the destination must have a correct tag with the hash of the arguments. It's much more secure, and also faster, since loading the tag from memory can be merged with loading the actual code after it.
I wish this was the one implemented in hardware..
It looks like endbr64 is a 4-byte instruction. That could be a significant code size overhead for jump tables with lots of targets: https://godbolt.org/z/xTPToaddh
https://marc.info/?l=openbsd-cvs&m=168254711511764&w=2
E.g: https://marc.info/?l=openbsd-cvs&m=167337396024167&w=2
Also, is it required to insert endbr64 command after function calls (for return address)?
The naive way to emit assembly for this is to actually emit two separate functions, and have Foo call FooWithoutChecks the usual way. But notice that the FooWithoutChecks function call is a tail call, so the compiler can use tail call optimization. To do this it would inline FooWithoutChecks into Foo itself, so the compiler just emits code for Foo with the logic in FoowithoutChecks inlined into Foo. This is nice because now when you call Foo, you avoid a call/ret instruction, so you save two instructions on every call to Foo. But what if someone calls FooWithoutChecks? Simple, you just call at the offset into Foo just past the pointer comparison. This actually just works because Foo already has a ret instruction, so the call to FooWithoutChecks will just reuse the existing ret. This optimization also saves some space in the binary which has various benefits in and of itself.
The example here with the null pointer check is kind of contrived, but this kind of pattern happens a LOT in real code when you have a small wrapper function that does a tail call to another function, and isn't specific to pointer checks.
They're usually called directly, but unless the compiler can prove that they always are (e.g., if they're static and nothing in the same file takes the address), endbr64 is required.
> Also, is it required to insert endbr64 command after function calls (for return address)?
No, IBT is only for jmp and call. SS is the equivalent mechanism for ret.
And no, as I understand it, this is only for branch/calls not returns.
He's unquestionably brilliant, but I've had a few encounters with him on the mailing lists and he is so quick to take offense where none was meant and drop into name-calling and insults. I don't really get it. He may have some deep insecurities.
The fact that Linux hasn't learned the right lessons in 20 years, and has chosen to "double down" in respect to IBT/BTI, does not inspire confidence that they will ever fix it. I'd say his 20 year estimate was in fact being pretty generous given the evidence available.
https://news.ycombinator.com/item?id=21554975
Unfortunately, for C code using GCC’s nested functions extension (or for languages that want to be ABI-compatible with C and support nested functions, like that paragon of advanced features called Pascal /s ), there’s no other compilation strategy in current ABIs. The patches to switch C (and not just Ada) to function descriptors[1] with an ABI break have been sitting on the GCC mailing list since approximately forever[2], but it doesn’t seem like there’s been any progress.
[1] The strategy is basically to compile (*fp)() not as
but as (untested) thus essentially inlining the (currently stack-allocated) closure calling thunk at all indirect call sites. It is ABI-compatible on x86 and x86-64 with all code that does not involve nested functions, place functions at odd addresses, or tag function pointers itself (and I think with all arm64 and riscv code, although arm32’s usage of the low pointer bit for Thumb interworking is bound to make this trickier).[2] https://gcc.gnu.org/legacy-ml/gcc-patches/2019-01/msg00735.h...
It's clearly different philosophies leading to different outcomes with neither of them clearly better than the other, it just depends on what you need. It would be possible to make that statement in a more graceful way.
It seems like a reasonable, relevant, and plausible assessment of how the long-term outcomes may likely differ between OpenBSD's stricter approach versus a looser approach, specifically when it comes to the degree of security offered (which is one of OpenBSD's main focuses), based on a past situation that's similar.
How do you know that you aren't being, to use your words, "quick to take offense where none was meant" in this case?
Past knowledge about Theo?
You could not mention Linux at all, or you could even say "we think this is better than Linux's approach because of X" and it would be a great improvement.
I have always found it interesting that Rust purposefully avoided doing language comparisons - "we're better than Python like this and better than C like that". Their message purposefully avoided any positioning of it as a competition, instead focusing just on articulating Rust's value. It was an eye opening approach given our instinct is normally to pit things against each other.
Truth is, Linux has a lot more constraints on how it can implement something because it has users. Users that have all sorts of different ways they need it to work.
Is a Canadian kernel developer more like a POSIX operating system than a POSIX operating system is like a POSIX operating system?
I'm not sure I understand. Perhaps you meant to write "Linus" since Linus is also a kernel developer? That seems more like apples to apples.
Explains why he spends all his time developing mitigations
Which is sad, as I kinda like the *BSD approach to things
Also I was pleasantly surprised to hear they support Apple M1/M2 Macs. Asahi Linux gets a lot of press around here but I had no idea OpenBSD supported it.
Could you give some examples/samples of things you have ran into off the top of your head?
I could go on, but, for my needs, it works very well and some of its simplicities are a godsend.
I actually think NetBSD is a pretty interesting alternative, it has some nice security features like veriexec that don't get talked about much.
I see tools like AppArmor as band-aids to fix problems that shouldn't exist in the first place. The problem with these approaches are the band-aids tend to break things in unexpected ways and when that happens they simply get removed and unused.
They provide plenty of mitigations (https://www.openbsd.org/innovations.html). In fact OP's article is for preventing unknown bugs from being exploited.
Deleted Comment
It would be more helpful if callers would store some machine specific hash of the function prototype and the function itself would check the hash, so that you could only redirect to calling a function with the right signature.
But that would also increase the overhead further. Already this is bad enough that it makes jump tables unattractive (which is too bad, considering the usually jump tables have little to no risk of control flow redirection).
Deleted Comment
An advantage of the software solution is that you don't need to have the feature compiled into every library for it to work, you just lose protection in those parts. That makes for a much quicker rollout. Also faster iteration times, in the Windows Insider Preview you can get the extended version that also checks that the hashed function signature matches.
1: https://learn.microsoft.com/en-us/windows/win32/secbp/contro...
Using better languages makes the entire problem disappear. You don't get a stack smash, the resulting opportunities for remote code execution disappear.
It suggests that maybe the "C magically shouldn't have Undefined Behaviour" people were onto something after all. Maybe C programmers really are so wedded to this awful language that just being much slower than Python wouldn't deter them. There is still the problem that none of them can agree how this should work, but if they'll fund it maybe it's worth pursuing to find out how much they will put up with to keep writing C.