It's fascinating watching someone write code for macOS while only referring to Linux and FreeBSD man pages.
On macOS, 'man ip' gives all the necessary info about raw sockets and IP_HDRINCL:
> Outgoing packets automatically have an IP header prepended to them (based on the destination address and the protocol number the socket is created with), unless the IP_HDRINCL option has been set.
> Unlike previous BSD releases, the program must set all the fields of the IP header
> Note that the ip_off and ip_len fields are in host byte order.
In a lot of cases, the macos man pages leave a lot to be desired, or don't come up in a google search. I know I'm personally not in the habit of remembering apropros exists, so I can imagine that the first step to manpage discovery being a google search.
And most of the time the docs are interchangable enough. Until they're not. At the raw socket layer, I'd still expect the macos docs to be consulted _eventually_.
You can do the same on Linux, just not easily from the command line; ebpf however can do all this and more. Once you have built it you can just create a program that you can use from the command line. There are guides out there to show you how.[0]
I've been meaning to try out eBPF. I read a tutorial on writing a simple perf application in Rust, but I kind of lack the intuition to know what it's good for.
Do you know a good overview of what can be done with eBPF?
Speaking of sockets on Mac, anyone knows if there is a documentation about how the network stack on MacOs works? (Routing, firewall, resolving, kernel extensions filtering/offloading, etc).
I guess I was tunnel visioned and lost clarity (and also, common sense). While searching for reasons why the code (from Graham King’s blog) had worked on Linux and not on macOS, a stack overflow answer [1] from 2015, from a person who had also been on the same journey, directed me to that page [2] — that talked about raw socket peculiarities in FreeBSD. So I dived straight into FreeBSD docs.
OP, you may find this [1] “trick” useful. It allows you to dynamically determine the correct byte order for the various IPv4 headers for the platform and thus avoid the need to statically decide on the byte ordering for each platform you intend to target.
You may also find this [2] table useful, it shows which platforms allow the combination of IPPROTO_ICMP + IP_HDRINCL so it may be used without elevated privileges.
In general, my experience of raw sockets is that they are not very “raw” at all, the OS can and does still perform a variety of modifications and additions to what you send and receive, in highly platform specific and often poorly documented ways. In particular, TCP and raw sockets should generally be avoided.
Apple has a relatively new (I think introduced in 2018) "Network.framework" Swift API [1] for working with raw sockets.
Maybe this post is dealing with the layer below - I'm out of my comfort zone with networking - but I recently built a custom protocol using Network and it's working well for me so far.
This is why i f**ing hate macOS. Developers wax poetic about 'oh its a UNIX', but NOTHING works like Linux once you dig into it. Zsh is different from bash, subtle API differences, different filesystem, esp with stuff like /dev or /proc or /etc, no systemd, some cli utils have different flags, subtle differences like this.
At least Windows is completely different and there's no expectation of compatibility, but with the mac you can start off thinking things would work, but they don't.
Some of the things you point out here are things you cannot rely on in Linux consistently, either.
You can’t rely on Systemd existing in Linux. You can’t rely on bash being the shell in Linux. Depending on the users system and distribution these tools may or may not exist.
Also the difference in some of the tools CLI flags wind up boiling down to being the BSD versions of them as opposed to the GNU versions of them. Which, again, isn’t a problem isolated to MacOS.
If you’re considering all Unixes everything you said basically still applies even if you remove MacOS from the equation. Unix isn’t just Linux.
The only practically relevant Linux environments you can't expect these things to exist are stripped down distros used for docker images, like Alpine (where you don't expect much from the system anyways) and Android (not sure about the latter, haven't used it in production in quite a while, but afair it's as non-standard as it gets).
That's a weird objection. Linux isn't UNIX, technically. All of the current and former UNIXes had various incompatibilities. /proc was never standardized. /dev has some standard names, but not many. Not sure what you mean about /etc, that's up to whatever's installed in userland. systemd is more or less Linux only; it's weird to blame other OSes for not having it. (The funny thing is that systemd has functionality in part inspired by macOS's launchd.) The ext* family of filesystems (and certainly stuff like btrfs) is a Linux thing; other UNIXes used different file systems.
CLI utils are an interesting thing: the BSDs have always(?) had their own, and don't use GNU coreutils (which is what Linux uses). There wasn't really much coordination over the decades, so they went in different directions. Most (all?) of them should be POSIX-compliant (possibly after setting an env var, at least in GNUs case), so if you restrict yourself to functionality specified in POSIX, you should be fine.
I don't really like macOS, but your specific criticism of it doesn't really hold water. If you're going to hate on macOS for this stuff, then you also have to hate on all the BSDs, which seems a little silly and unwarranted.
> with the mac you can start off thinking things would work, but they don't.
If you're expecting things to work on macOS just like on Linux, then your expectations were way off in the first place. If you want cross-platform compat in these areas, you'll have to stick with POSIX. Which sucks, especially since no one really implements POSIX correctly, but that's just how it is, and has always been.
Having used hp-ux, sunos, solaris, aix, plan 9, various flavors of linux and a bunch of embedded OSs, VMS, and various Windoze systems I formally welcome you to the Real World.
The Real World - it's got more stuff than you can imagine.
This is false plurality - while some bank or legacy datacenter might have these OSes in production, 99% of the time, you'll only encounter Linux in a professional server environment (or the Real World as you call it).
Indeed, I've been Linux-only for a bit over 10 years now, and when I have to do some work on a Windows or Mac for work, Windows with WSL feels a lot better than modern macos. You'll encounter the occasional alien thing when Windows details leak through, but for the most part it just behaves like you expect. If I had to pick a mac or windows pc for daily driver, Windows would be an easy choice now (although as coming of age in the 90s when MS was evil, it's mind-blowing to hear me say this all!)
Readit News
On macOS, 'man ip' gives all the necessary info about raw sockets and IP_HDRINCL:
> Outgoing packets automatically have an IP header prepended to them (based on the destination address and the protocol number the socket is created with), unless the IP_HDRINCL option has been set.
> Unlike previous BSD releases, the program must set all the fields of the IP header
> Note that the ip_off and ip_len fields are in host byte order.
And most of the time the docs are interchangable enough. Until they're not. At the raw socket layer, I'd still expect the macos docs to be consulted _eventually_.
Becasue BSD raw socket rules.
That's how you get Sniffer to be performing waaaaay more intuitively on macos than at all on Linux.
[0] https://www.datadoghq.com/blog/ebpf-guide/
Do you know a good overview of what can be done with eBPF?
https://wiki.nftables.org/wiki-nftables/index.php/Matching_p...
https://developer.apple.com/library/archive/documentation/Ne...
[1]- https://stackoverflow.com/a/32599757/3728336
[2] - https://cseweb.ucsd.edu/~braghava/notes/freebsd-sockets.txt
You may also find this [2] table useful, it shows which platforms allow the combination of IPPROTO_ICMP + IP_HDRINCL so it may be used without elevated privileges.
In general, my experience of raw sockets is that they are not very “raw” at all, the OS can and does still perform a variety of modifications and additions to what you send and receive, in highly platform specific and often poorly documented ways. In particular, TCP and raw sockets should generally be avoided.
[1] https://github.com/fujiapple852/trippy/blob/master/crates/tr...
[2] https://github.com/fujiapple852/trippy/issues/101#issuecomme...
Maybe this post is dealing with the layer below - I'm out of my comfort zone with networking - but I recently built a custom protocol using Network and it's working well for me so far.
[1] https://developer.apple.com/documentation/network
CFSocket would be the Swift/Obj-C API.
https://developer.apple.com/documentation/corefoundation/cfs...
The sysctls are net.link.generic.system.hwcksum_tx and net.link.generic.system.hwcksum_rx on macos.
Deleted Comment
At least Windows is completely different and there's no expectation of compatibility, but with the mac you can start off thinking things would work, but they don't.
Also the difference in some of the tools CLI flags wind up boiling down to being the BSD versions of them as opposed to the GNU versions of them. Which, again, isn’t a problem isolated to MacOS.
If you’re considering all Unixes everything you said basically still applies even if you remove MacOS from the equation. Unix isn’t just Linux.
CLI utils are an interesting thing: the BSDs have always(?) had their own, and don't use GNU coreutils (which is what Linux uses). There wasn't really much coordination over the decades, so they went in different directions. Most (all?) of them should be POSIX-compliant (possibly after setting an env var, at least in GNUs case), so if you restrict yourself to functionality specified in POSIX, you should be fine.
I don't really like macOS, but your specific criticism of it doesn't really hold water. If you're going to hate on macOS for this stuff, then you also have to hate on all the BSDs, which seems a little silly and unwarranted.
> with the mac you can start off thinking things would work, but they don't.
If you're expecting things to work on macOS just like on Linux, then your expectations were way off in the first place. If you want cross-platform compat in these areas, you'll have to stick with POSIX. Which sucks, especially since no one really implements POSIX correctly, but that's just how it is, and has always been.
The Real World - it's got more stuff than you can imagine.
Windows and WSL feels better to me at least.
I'd call that an excellent feature and more true to Linux than Linux is to itself today.
Even Linux is not a true UNIX,it is UNIX-like. You can't expect macOS to be like Linux, it should be quite the other way around.