The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand.
The whole IT ecosystem has become a hail mary. Even admins usually have no idea what a certain program actually wants to do. If the admin knows how to install the app so that it actually runs, you call them a good admin.
From a security point of view, an application is like a nuclear power plant. It's good if it works as planned, but if something blows up it endangers your whole enterprise.
The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl. That way the radiation hopefully stays in, but history has shown that it really doesn't. Also, the wheel of history has just turned one more iteration and now admins just view the sarcophagus as something you deploy as you previously deployed the app. Who is responsible that it is air tight? Well, uh, nobody, really.
You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.
Obscure files like /etc/localtime, /etc/resolv.conf, /etc/ld.so.conf, /dev/zero ... how can you expect devs to build well documented and well sandboxable applications if they don't know which files their library functions will open?
You may have heard of /etc/resolv.conf ... but have you heard of /etc/gai.conf? /etc/nsswitch.conf? /etc/host.conf? Wouldn't it be great if the man page of getaddrinfo mentioned those (mine only mentions gai.conf)
> The problem is not so much that selinux is too complicated (it is as complicated as it needs to be)
I disagree.
> but that we all run software we don't understand.
I fully agree.
My disagreement lies in the fact that you've described the problem, but are proposing that some software (SELinux) that fails to solve the problem is somehow good.
SELinux might be a perfect tool in an ideal utopia where everyone understands all the software they run, but that isn't the real world, and having a tool that works in a theoretical world isn't particularly useful.
Either SELinux is applicable to the real world, or it's not useful and not really fit for purpose.
This isn't a "blame game" - it's not about figuring out why things are bad (no-one understands their software components), nor blaming SELinux (it hasn't caused these problems, it's just failing to mitigate them). It's about figuring out how to improve that situation. Does SELinux do that effectively?
>My disagreement lies in the fact that you've described the problem, but are proposing that some software (SELinux) that fails to solve the problem is somehow good.
My opinion is that there's a cultural and policy problem and you're simply not going to solve it with technology. I don't think SE Linux is bad, it just isn't the answer here. One (clearly not the only) main reasons people just run a bunch of software they don't understand is due to expectations in productivity. In an idealistic world with caution built in, we are willing to sacrifice some productivity for security, stability, understanding, etc. I think many engineers like these ideas because they like to work with and build solid systems they understand very well--otherwise they wouldn't be practicing any form of engineering.
What we instead have is a world of engineering with no sort of guard rails. Unlike Civil Engineering, there's no licensing, no sorts of inspections, no oversight, liability is completepy waived in a 300 page agreement people just click through, etc. Even when things go horribly wrong (e.g. Equifax) the repercussions are limited. We see abuse in other engineering fields where actual life and death are factors and they're still just amortized in a cost analysis.
Ultimately though engineers simply don't lead the world (and arguably,
maybe they shouldn't) , people with wealth lead the world (arguably, they shouldn't either) because it underpins pretty much all other systems anymore. As such, productivity is always prioritized over safety and security. This issue isn't limited to software, it happens everywhere almost ubiquitously at this point. Software is one of those areas where most people just care less, so you can get away with more and more of this. The issue is that I believe we're building a house of cards that will be difficult to revert to a reasonably stable state once it finally collapses.
SELinux is a perfect tool in its niche — where that niche is "secure-installation system integrators." It's not a tool for regular people; it's for e.g. defense subcontractors who want to certify their virtual appliance as meeting certain standards demanded by certain customers; where meeting those standards requires them to actually go through their virtual appliance from top to bottom and prove the whole thing secure and sandboxed.
It's not a tool for application developers (who at most can supply SELinux templates to start with); nor is it a tool for sysadmins (who really don't have the time or the incentives to deal with it.) It's a tool to be wielded specifically by people occupying a role that, in most organizations, doesn't exist.
SELinux is applicable in the real world, but you must choose if you want it to be Fast (use globs of rules written by unknown people who don’t know your requirements), Cheap (start from zero and develop policies specifically for your app), or Good (do both while learning a substantial amount about SELinux).
If you’re using SELinux rules off the shelf for a handful of Fedora boxes for personal use, maybe that use case doesn’t work. But is that more “real world” than an institution like the NSA trying to completely lock down its production systems?
Honestly, the bigger issue is that most SWEs just aren’t very good. It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…
System design, though, is the actual point of SW ENGINEERING. That’s the part that is responsible for creating a foundation of quality to build on.
The other side is that sysadmins have largely become DevOps or SRE roles, which means most folks left as a sysadmin are those that couldn’t hack it in the other roles. In the end, as we grow the number of people in tech, the number of software and systems, and the complexity we experience a regression to the mean across the board.
It’s always been a pet peeve of mine that I can count on my hand the number of SWEs in my ~20yr career that had similar understanding of the system they worked with as their SRE counterparts… but this should be table stakes.
> It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…
Yes. It's absurd that anyone that can talk about this stuff at a basic level is seen as some sort of ultra-skilled professional. See also: the relatively small amount of chatter about this stuff in the blogosphere, where there's a million tutorials that amount to "here's how you run create-react-app" and about 3 that talk about even moderate system design topics.
> Honestly, the bigger issue is that most SWEs just aren’t very good. It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…
I think this is a bit unfair, the software industry is so big now there are lots of different sub-sectors. Big Tech companies have huge scale problems that are kind of unique. Most developers work on products that only have a few hundred users at a time.
Your 20 years experience is a bit telling. I'm the same age, back then SWEs usually had full access to production environments. Now everything is segregated and devs can't experiment in prod and end up getting detached.
Maybe we're just in different circles, or perhaps have different definitions of system design, but that's not been my experience. Usually I see developers being very excited about system design and quite good at understanding system design questions - nothing's more fun than architecting a new system and thinking about stuff like scalability.
Where people tend to fail is when they have to implement these systems, and they've planned for all these high level building blocks but then actually have to configure them. Redis, RabbitMQ, GraphQL, Docker, NGINX, the list goes on - there's so much to configure and you really don't truly know how they work unless you dive into the source code, so you inevitably get bitten by subtle bugs and errors where your expectations of how something should work based on the docs are violated by how it actually works.
Should a SWE have a deep level understanding of all this stuff? I'm not sure. Usually you can still hack it into a working system after a little bit of firefighting, and that's likely more time efficient than understanding source code or rolling your own infrastructure. But I definitely think there's room for more comprehensive and transparent documentation to bridge the gap between the source code and barebones docs.
> It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…
> System design, though, is the actual point of SW ENGINEERING.
I mean, I would say I'm pretty good at engineering and system design (10 YoE, senior/principal for most of them). But I dread the system design interview questions because (in my experience) they bear almost no relation to the system design experience I have and almost no relation to the actual requirements of the job. Things like "design an ETL flow" or "build a distributed wikipedia downloader" are just not in my area of experience. (I work mostly in the lower layers of single-host operating systems and file systems.)
I think most devs see that the jobs are asking for overloaded responsibilities from multiple other jobs and they're trying to live up to that. It also became the norm to see job postings which would have been 3-4 distinct positions a decade back.
True, but even in the 90's and early 2000's, before cloud took off, most developers weren't sysadmins, and they didn't want to be. That wasn't their specialty. The "Unix wizard" developer was always a rare breed.
> System design, though, is the actual point of SW ENGINEERING
Software Engineering is about problem solving using software and hardware (we'll just say computers for simplicity). Whatever solution system you propose has inherent constraints and tradeoffs. That's the Engineering. System design is a catchall phrase for a subjective problem scale.
Speaking personally, the reason I dread system design questions is because it's asking me to give an off-the-cuff answer to something I would prefer to take weeks to consider and research.
> The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand.
There are many many problems.
One of the biggest problem with SELinux is that it is trying to graft Mandatory Access Controls on a userland that is not designed for it.
Unix, frankly, is not designed for security. It is designed to get work done by writing a bunch of little buggy C programs that you string together in novel ways.
Security is something that was grafted on it. And it shows.
How many decades of security vulnerabilities have occurred because of a shared /tmp space? 40 years?
How do you graft access controls on a system designed with no access controls and potentially billions of combinations of programs, paths, and various other resources without breaking anything?
The answer is: You don't. You can't.
Were as you have a system designed for security, like Android, and literally hundreds of millions of fully SELinux-enabled fully locked down user-facing Linux devices are out there being used by people who haven't the faintest clue what "audit2allow" is and wouldn't understand it if you tried to explain it to them.
So it's less of an issue of "we all use software that we don't understand". SELinux is complicated enough that you can devote your life to trying to understand it and still fail to craft good rules for other people.
It's more of an issue of "Linux userland follows the basic Unix design from the 1970s which is kinda shit if your goal is security".
It is just bad design. Pure and simple. That is all there really is.
However there is a way out.
The way out is to give each process and each user their own little itty bitty special Unix environment where they can do whatever they want. And then you use strong divisions to keep them mostly unaware of each other. Use a default deny policy and only poke holes in it when required.
My android phone came without a calculator app. I recently realized this in a setting where I needed a calculator. I commented out loud about the lack of calculator and got some responses from the group I was in at the time.
"Just download one from the app store," I was told. "But be prepared to grant it network access, file access, contacts access, camera access, and email privileges."
Single page of instructions from google apps has me download a tarball, add a hello world command, and Presto! I have a webapp on my computer! Then change this one line and PRESTO! My webapp is in the cloud!!
No telling how, or with what components, or what the dependencies or security implications are, or how to see the logging for the app. Just a single paragraph of instructions and you can be the next internet startup IPO!
'Hail Mary' is the best way to describe the situation I've yet heard.
I fully agree with your last point. In my experience, the all-versions-packaged approach of containers actually leads to a worse patch state than a simple Ubuntu or Debian installation with unattended-upgrades.
I would argue that the fundamental problem is that the companies selling software "engineering" products do not actually take any responsibility that the product they have engineered works as intended. (see: https://www.snopes.com/fact-check/car-balk/ )
And of course, the main reason they do not take the responsibility is that the customers won't pay for it.
It is kind of interesting. We are very good at making bridges that do not fail unexpectedly, even if there is unlimited amount of unknown failure modes when working with physical materials. And on the software side, well, managing to come up with a fizzbuzz without failure modes is used to screen people on interviews. What would be similar for bridge engineers? Here is a hammer, nail and two pieces of wood. Can you make the two pieces of wood stick together?
The engineer building the bridge is liable. Hence, you get engineers with qualifications (instead of a random person with X years in Y), and those qualifications certify they know what they are doing.
A bridge, even a fancy modern one, is pretty simple compared with the software running on a 90s-era computer, much less a modern computer. Fizzbuzz is probably about equivalent to building a simple engine out of a box of parts.
The problem is that we have built all our software on Unix. The Unix security model is based on an attack model where users have to be protected from each other on a terminal server. Code is implicitly trusted and exploits were an unknown unknown.
That security model is almost completely useless now. Terminal servers are an extreme edge case. Services implement their own security model between users. Special “Users” for services is just a hack that tries to use an inadequate system as much as possible.
The Unix security model is deeply baked in everywhere and it’s nearly impossible to tack on a security model fitting to todays requirements afterwards.
I agree with your points about the Unix threat model being designed around protecting users from each other, but for completeness I'd like to point out that while modern Linux/BSD systems ship with Discretionary Access Control (DAC) by default, SELinux implements a Mandatory Access Control (MAC) system which is much more fine grained. SELinux is not limited to the traditional Unix security model.
Short simplified example highlighting DAC/MAC differences: DAC asks "is user Alice allowed to read Bob's files?" while MAC asks "is the SMTP server (subject user/role) process allowed to read private keys (object type) of the HTTP server (object user/role)?"
And if you're really motivated (read: want to have fun diagnosing unexpected file permission issues), you can associate files and processes with different security categories and levels (MCS/MLS), implementing horizontal and vertical separation. For example, "is this software update service allowed to read confidential files owned by the accounting database or keys used to encrypt classified information?"
There are other MAC systems besides SELinux as well.
Is Windows much better with its VMS legacy kernel? It has a complex permission model with ACLs and everything but that trades inadequate permissions for something like the SELinux problem. There are too many knobs. (For this discussion leave aside the backward compatibility baggage which is another issue.)
The only approach that I see as viable without rebuilding the entire compute universe is VMs, either the web browser (JS and WASM sandboxes) approach or the qubes type approach.
I agree with you, but if we look past the hack of dynamically creating new users per process, isn’t the security model “sufficient”? E.g. I believe android’s security is quite elegant with the different user per-process model, communicating only through IPC and with heavy use of SELinux (which is much more useful in this model).
Some of the problem is that historically we've built systems badly engineered for security.
Take for instance something like xscreensaver. Something in there needs to be setuid so that it can verify your password to unlock the screen. That something is fortunately a dedicated binary, and not every single screensaver, but still, that's bad. Writing that executable is a delicate thing. Get one of them wrong, and it's a glaring hole.
And /usr/libexec/xscreensaver/xscreensaver-auth is quite the thing. It links to a whole bunch of graphical libraries and talks to X11 to read your password, so that thing has a huge potential attack surface. Far more than seems comfortable.
What we should have instead is some sort of authentication service. A program confined by SELinux to only interact with a socket and the password database, speaking a simple but well designed protocol.
With that in place we'd have much simpler security. Policies get simpler because random stuff doesn't link to PAM anymore, and so doesn't end up touching a whole bunch of critical security related files. There's one thing on the system that deals with that, and it ends up with a small, understandable policy.
And the policy for the users is now expressed in terms of "Do we want this program to be able to validate user passwords?", which is also far simpler.
With that kind of redesign over time we could have much smaller and more understandable policies, and a more secure system as a result.
I get your overall point, but your example is not a good one: on Linux, at least, xscreensaver does not have a component that needs to be setuid root.
xscreensaver-auth uses PAM to authenticate, which (typically) runs a small program called unix_chkpwd, which is setgid shadow (not setuid root), as /etc/shadow is owned by root:shadow and is readable by group.
xscreensaver is one of the very few screen lockers that is reasonably secure, because the bit that locks the screen is pretty minimal and only links with a few fairly-low level X11 libraries, not a full-scale GUI toolkit.
> What we should have instead is some sort of authentication service. A program confined by SELinux to only interact with a socket and the password database, speaking a simple but well designed protocol.
That sounds pretty overengineered. All you need is a small binary that is setgid shadow, that can take username/password on stdin, and exit 0 if auth succeeds, 1 if auth fails. But we already have that: unix_chkpwd.
Well I have been playing around with Wayland + Sway the last month. I like it but there are some caveats. For example I end up modularly picking the tools I want to use with it. So by default I think there is swaylock and then there is waylock but a few times I got back to my seat and the locking application had crashed, exposing my session to my co-workers. That ain't the desired effect. I resorted to physlock which swaps to tty and allows to disable things like echoing of kernel messages and sysrq. It has yet to crash. Also, all of this relies on PAM and according to the OpenBSD devs PAM is a mess.
What's the difference between a service that you send a string to and get a string back and a binary that you execute with a string argument and it prints a string argument back? I quite like the latter, as you're free from keeping state and thus have a smaller attack surface or potential to leak resources. Of course one difference is the execution environment, with a systemd service you can have it set up exactly as it should be so no changes to LD_LIBRARY_PATH &c can poke holes. I wonder if its socket activation feature can be used like a sort of CGI server - for each connection on a socket, run the binary in a controlled environment and connect its stdin/out to the socket, with stderr going to a log file.
> The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl. […] Who is responsible that it is air tight? Well, uh, nobody, really.
The people who designed the container/sarcophagus system. If it's not secure don't sell it as secure: see the difference between Linux containers and FreeBSD jails or Solaris zones.
> You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.
As a developer / software "engineer" it is your responsibility to know how the components you choose work and how much you can rely on them. When a structural engineer selects certain types of steel or concrete to be used in the construction of a bridge, it is their responsibility to know the characteristics of the raw materials.
When you ship a software product, enable SELinux or AppArmor and audit what it does: then ship an SELinux/AppArmor profiles.
See also OpenBSD's pledge(2) framework where the developer embeds system calls in their (C) code to promise to only access certain OS features:
> it is your responsibility to know how the components you choose work and how much you can rely on them
Unfortunately that is not always your choice and arguing against it most commonly costs you more than you like. My experience is quite often timelines get underestimated massively with software so you end up doing what you can with the resources given. If you pull this into the bridge metafore the bridge still stands but can collapse any time and as long as it doesn't collapse during rush hour and can be rebuilt before the next morning it's all fine.
> When you ship a software product, enable SELinux or AppArmor and audit what it does: then ship an SELinux/AppArmor profiles.
It's not just sysamins -- developers don't understand SELinux, either.
As a dev, I find least-privilege sandboxing (e.g., look at how sshd is structured) and capabilities (e.g. FreeBSD capsicum) somewhat non-intuitive, but at least tractable to understand. SELinux is a blackhole.
>> The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl. […] Who is responsible that it is air tight? Well, uh, nobody, really.
> The people who designed the container/sarcophagus system.
Original comment misses two nuances with the gripe.
1. Container creators and software creators are not the same people. So they can (and should!) have different goals and priorities. For containers, security and reliability. For software, make it work.
2. There is a nuanced, different interface between software-system and container-system, created by encapsulating a system-like environment inside the container. Much of what software does can be encapsulated. Which allows a more controlled (and much smaller) subset to exit the container.
Ideally all apps & libraries would ship their own selinux policies with a common framework for combining them & base layer for privilege sets (eg, "let me open any file the current user can open" or whatever). If that was the case then your concern wouldn't be an issue. You'd just say "I use libc sockets" and you'd inherit whatever file path permissions are necessary for that to work, as defined & exported by the libc in question.
But that's not a thing. So distros are attempting to add it later themselves, which is a disaster.
> Wouldn't it be great if the man page of getaddrinfo mentioned those (mine only mentions gai.conf)
That's a huge part of the problem. Was looking for a monograph to pass on my successor that wasn't so familiar with linux - modern linux - as in systemd, docker, nssswitch, pam_homed, network-manager is penetrable but more often than not I'm just looking at the c source in github and I am in my 30ies, starting in my teens running linux nonstop and I'm still throwing up my hands every other day...
Add time constraints and pressure to deliver (don't waste your time understanding this, just do xyz) and here we are - on the other hand it's not okay that you have to devote years of trail and error to get a comprehensive undestanding of the system.
I guess a nuclear power plant has at least a training plan and complete reference book - that still needs to be read and grokked and trained upon but modern linux is often just undestandable by reading the source if you hit a problem - for some projects like things in the freedesktop ecosystem and partly systemd even that is kind of difficult because i.e. this bug https://github.com/systemd/systemd/issues/19118 explains the problem - there is no overview documentation, no clear way to look up what's happening, not even a good way to introspect and it's several components that interact with each other fail subtly. I've chosen this one because I've also hit it, not because I want to blame systemd, which is not so bad there are much worse things out there but it's part of a trend to introduce complexity - not sure what is necessary complexity and what is unnecessary - went to uni in compsci without ever someone slapping out of the tar pit (http://curtclifton.net/papers/MoseleyMarks06a.pdf) in my face and maybe that's part of the problem.
Exactly. ‘ps ax’ on a fresh OpenBSD install gives you about 10-12 processes, each of whose purpose is either obvious or easy to discover.
Try doing the same thing on macOS or Ubuntu. Last time I looked carefully at the former I discovered, for example, daemons for classrooms (??) It’s a free-for-all.
> The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand.
I don't think these statements are meaningfully different. "too complicated" implies "...for humans to manage". Maybe that's sort of your point?
> You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.
Agreed. It's too burdensome for software developers to understand exactly what syscalls their program needs to make and the security implications of permitting those syscalls. It also doesn't help that Linux naming conventions and concepts are very counterintuitive (yes, dear Kernel Hacker, I'm sure they're very intuitive to you, but we lowly mortals struggle).
And unfortunately the SELinux policies are tightly coupled to the application such that you can't make SELinux policies the purview of a dedicated sysadmin expert and leave the appdev to the development teams. They have to collaborate which is swimming against the current of Conway's Law or else you make SELinux policies the responsibility of appdev and suffer their lack of expertise.
We had similar problems with operations in general, but containers largely solved this problem by allowing sysadmins to focus on a platform while developers focus on developing and operating the application. We need something similar for security. This is probably a rephrasing of your "sarcophagus" point?
> Obscure files like /etc/localtime, /etc/resolv.conf, /etc/ld.so.conf, /dev/zero ... how can you expect devs to build well documented and well sandboxable applications if they don't know which files their library functions will open?
Who the fuck invented that convention that fine-grained permissions must be file-based? It's insane. No, no developer will anticipate that he needs to read /etc/nsswitch.conf. No developer should. A developer should anticipate that the software needs permission to connect on network hosts.
As much as it is too granular, file based permissions aren't fine-grained enough. Asking to connect on random hosts is absurdly wide, most programs only need to connect in to a few of them, or connect to an user-supplied host (that can be a permission by itself).
Anyway, yes, the manpages should interlink better. What is a different issue.
> The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl
Reminder: containers are not meant to be security tools.
Fine-grained sandboxing (e.g. seccomp) is. And it can be layered upon OS-level VMs.
Additionally, bundling tons of stuff together with an application (like docker but also flatpak do) is not good for security. Same for static linking. They all increase the workload of updating vulnerable dependencies.
I've been recently thinking along similar lines about libraries. In web dev, it's common practice when you have a problem to first look for a library that solves it for you.
I get the "don't reinvent the wheel" sentiment but, I think we take it too far sometimes. I've been looking at the source code for some dependencies at work lately, and many of them actually don't hold up to our own code quality standards. Kind of subjective, yes, but many of our dependencies would probably fail code review if actually reviewed by us.
Then when there is a bug in a dependency, nobody actually understands how it works and the code is often tucked away and not easily changed.
when it comes to JS libraries I'm usually a lot more interested in how good their testing is. Code quality can be very subjective in some areas. But if the dang library doesn't have a lot of good testing then I can't really trust any updates from it :/
Used to be, you could update it manually. Then NetworkManager (or whatever its successor is) came along and would change it back on you. Oh well, you could still edit it for a quick test.
Then I set up a Wireguard tunnel. And it changed /etc/resolv.conf. Oh, but an entry is missing, I'll just add that and test. Nope, read-only. But I'm root! What gives?
Turns out the wg-quick script mounts a file on top of /etc/resolv.conf! I didn't even know that was possible until I saw it. Nobody messes with wg-quick's resolv.conf, and that's final! Until some other tool ups the ante and gets code to undo that.
I have, in moments of desperation, `chattr +i /etc/resolv.conf` before. I understand the rationale behind managing that with a more robust service, but in my experience that's much more prone to the black-box effect.
>>The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand
There is truth to this, and it nails the fundamental asymmetry of the bad guys vs good guys in the security war. To program selinux you need to understand the software you run at the syscall level. and potentially have a deep understanding of its file usage, particularly if it’s doing ipc.
In general, I think that is a good goal. More understanding is more understanding and that is Good. In practice? I equate it to the problem of writing secure and robust code in C, I don’t know how good you have to be todo it and I basically assume that anyone who says they do is full of shit. I have contributed to the Linux kernel, I have decades of UNIX and specifically Linux experience as a software engineer, and I am still surprised when I fire up strace from time to time. You look at something like the recent Dirty Pipe bug, and I have a difficult time accepting that many people can fully grasp it all. The cost of a fairly simple system interface is all the subtlety and edge cases.
I view the container movement as a way of managing interactions. It’s really hard to manage dependencies in such a heterogenous systems. Containers simplify it all, at the cost of owning ones full dep tree. But with a good release process, one can keep on top of that and put less effort into the combinatorial growth when interacting with all the other dependencies other programs require.
containerization is less about security these days, and is being used to work around the broken package/dependency management situation on linux which is caused by the wild west of libraries that make little effort to assure any kind of forward/backwards compatibility. Given the fact that distro package maintainers cant maintain 30x different versions of any given package to satisfy all the applications stuck on a particular library version, means that larger applications which can't be constantly churning their code bases to replace foo(x,y) with foo_bar(y,x) are stuck shipping their own pinned version of a library.
So, I don't think anyone seriously thought that containers provided significantly more security than simply running an application and trusting the kernel syscall and filesystem permissions were bulletproof. More software layers are unlikely to improve security.
It's the forward march of progress. As things get more complex they also get more fragile and inscrutable, and harder to solve problems for.
Containers are a reaction to the flaws in the system. Fixing the root cause of those flaws is hard, because they're systemic. We would need to re-design everything to fix those flaws. Containers are a stop-gap measure that makes things easier to deal with.
The thing that I've learned about systems is they are all crap, but humans constantly munge them to keep them working. We don't end up with the best systems, we end up with systems easiest to munge. Every once in a while, a spasm of collective organization redesigns the system, and then we keep going with that one. It's evolution. So however crap our current system is, eventually it will change, probably for the better. It just takes a really long time.
This seems like something that WebAssembly (misnomer) is set to resolve, to some degree. Sandboxing capabilities is easy because all of the wasm module imports must be provided by the process that initializes the module, so you can view imports as a kind of list of "requests for capabilities".
I think that's leaning even more into the parents point.
We make a sandbox (in this case a very inefficient one) -- we assume the sandbox is the only way to deploy software, but mostly it's not because there will be times you need more than the sandbox or the software doesn't work, and then you're running "privileged" WASM routines.
The sandbox itself might even require some kind of leaky behaviour, such as accessing files or shared memory.
The wheel keeps turning. Some new sandbox will come along.
Why would a completely new compilation target be needed for that when “old” sandboxes can work just fine by hijacking syscalls? Sure, memory safety is somewhat improved by Wasm, but valgrind and the like are effectively giving you the same thing.
Wasm addresses memory and execution isolation from the rest of the system, however anything running in it maintains it's exploitable logic internally. If the application needs to read/write files, and someone exploits that logic, wasm won't help. The only thing that could is limiting the filesystem access in the first place and building secure software to begin with.
> "The problem is not so much that selinux is too complicated (it is as complicated as it needs to be)..."
Completely disagree, if the target users are advising each other to disable it... then the tool is definitely more complicated than it needs to be.
Using tools that don't hide any complexity are very painful to use. It feels like the creator doesn't care about the user and put no thought into the display of information or workflows of the tool.
Solving a complex problem with a perceived complex tool is relatively easy... don't hide any complexity.
Solving a complex problem with a perceived simple tool is difficult... hiding complexity at the right time, revealing functionality it based on the user's intentions and experience is not easy but greatly appreciated.
if the target users are advising each other to disable it... then the tool is definitely more complicated than it needs to be.
This does not follow. I you ask a 5-year to repaint the Sistine Chapel and the result is somehow less than stellar, are you similarly going to blame the paintbrushes used? There exists no situation where the task at hand is too complicated for the user?
Uhh isn't this precisely what the whole snap/flatpak movement is trying to fix?
I can talk about snaps, but at least they are offering a specific permission model which is usable for users to understand. i.e. Via the interfaces, I can know software x accesses my password-manager, home directory, camera etc... I can disconnect access to the given permission and have it enforced via the kernel and apparmor.
The applications themselves bundle only libraries that themselves are sandboxed/snapped.
>> It doesn’t help that no one wants to be told: “stop what you’re doing, spend a week to learn this other thing, make a tiny change, and then return to your real job”. They’re looking for solutions — preferably quick ones — and not a week’s worth of study.
If you're a system administrator running SeLinux why dont you already know SeLinux? It's not like some obscure command, it's a big part of the infrastructure you're running.
As an experienced RHEL admin, a few years ago I probably would have said this is very bad advice in any professional context, and you should spend the time to learn it because it will save you one day.
Now, I think my advice would be: Put everything in a container, and learn how to run Docker or Podman (or k8s) in a secure way (ie no root containers, be very careful with volume mounts, etc). Yes, they aren’t as mature as SELinux, but containers aim to provide many of the same benefits that SELinux does (and even more) except in a way that’s much easier to manage. Even better is that these container runtimes often come with SELinux and AppArmor policies out of the box on good Linux distros.
I agree that containers are easier to manage than SELinux, but still not easy enough. Linux provides many low level primitives for restricting applications and SELinux, Apparmor, Docker, Flatpak, and systemd all provide high level abstractions for those. But IMHO none of them really finds a sweet spot between flexibility and usability. `systemd-analyze security` for example lists 80 (!) different settings, even though some of them are very high level such as `ProtectSystem`.
Containers have made the conceptual shift from allow/deny to isolate/share. Somehow this feels better even though it is effecitvely the same.
I am still waiting for an abstraction that uses all the low level features and wraps them in a high-level interface that puts usability front and center.
I am not sure if this is even possible though because many applications are not built with sandboxing in mind. Adding another file somewhere on the system that needs to be accessed is not considered a breaking change by most. So maybe we need a more fundamental shift.
> I am still waiting for an abstraction that uses all the low level features and wraps them in a high-level interface that puts usability front and center.
Bubblewrap? FlatPak's sandboxing features are built on it already.
In my opinion, if an application requires more access than a docker container gives by default, then that application should probably just run in a VM. If the application needs more access because it needs to manage or control some hardware, then it should be tailored to the O/S and have a small core service that runs naked under systemd or whatever. If fancy management of that core service is needed, it can expose a port that an application can talk to, that's safely isolated in a docker container.
Maybe that's just daydreaming about a perfect world, but I agree that it should be easy to reason about the access levels.
As a lightweight alternative to Docker-based (or any container-based) solutions: Try firejail. You can set up a directory that will be the "home" of the sandboxed application you're running, then you can do something like `firejail --private="${HOME}/my_firefox_jail" firefox`. There are built-in profiles for many applications already, and you can customize them (by adding `.local` files, not editing the existing `.profile` files). See the following link for details.
Convenience matter. Without it being automatically on everywhere, it doesn’t protect from much. Sure it can be good for the occasional random software you trust the least, but how many exploitable bugs were found in any of your completely trustable tools?
Also, afaik firejail runs as suid, making any possible escape much more serious.
I'd generally agree with this, but for production use (where it'll be CRI-O or ContainerD in all likelihood) it's worth noting that using containers does increase your attack surface.
We've had container breakout CVEs in the Kernel+ContainerD+CRI-O this year, and the ContainerD/CRI-O ones were not complex at all, so you do need to stay on top of patching as well as the other good practices mentioned.
I was about to argue with you until I read to your last point. Totally agree one of the great things about containers is indeed they integrate very well with various Mandatory Access Control implementations SELinux included and don't generally require specialized policies for each app.
AppArmor is great. It can do much of what SELinux does, but it uses plain configuration files as opposed to incredibly obscure and clunky filesystem-level hacks.
AppArmour assumes everything is allowed unless it is explicitly denied by policy, SELinux assumes everything is banned unless it is allowed by policy. This makes SELinux fundamentally more secure because unlike apparmour you cant forget to deny some access that leaves you wide open, instead you forget to allow things and your app breaks. It also makes SELinux more complicated to implement because you have to allow everything you need or your app breaks whereas apparmour lets you deny the things you're worried about and ignore everything else.
SELinux also supports interesting things like applying security levels and contexts to data, so you can have data that is only accessible to appropriatly cleared users in one department but not to people with similar clearances in other departments. Very few people use this stuff though.
Selinux uses extended attributes to store its labels which is a pretty standard way of associating metadata with objects, and is supported by most standard unix commands (via the -Z flag) so I'm not sure what you mean by "filesystem hacks". Fundamentally they are both implemented as security modules in the kernel and do pretty much the same thing, which is best depends on whether the added security of SELinux is worth it to you.
I was daemonizing a uvicorn (Python) process using systemd and could not get the Python script to write its log file anywhere. Not the current directory owned by the user, not elsewhere, and no errors showing up in the system logs.
After a few hours banging my head against the wall I discovered systemd's ReadWritePaths. Once I set that, it would log to any directory I specified.
It's the only service I've created where this has been necessary, and others write logs, so I have no idea why this one caused errors. If I did something wrong, let me know how this should be handled.
Container approach is not always feasible, for example when you need to stay on bare metal. There are Linux applications where selinux definitely gets in your way and there are legitimate reasons to turn it off completely.
edit: I like how I'm getting down-voted for sharing factual information. No-one in high frequency trading employs containers because it doesn't help soft-realtime, low-latency goals. There are other performance-sensitive domains where containers aren't an option. Sometimes I feel like this place is turning into Reddit.
What cases can linux containers not handle? Containers can access GPUs, /dev/kvm, block devices... I'm having trouble thinking of anything they can't do. After all, they're just processes in a glorified chroot, not that different from processes on the linux host.
And when everything is in docker you can use SELinux again: A docker container is not allowed to access the host system or access the filesystem of other containers. That‘s the default. SELinux protection without any weird problems.
As a completely new admin, could you elaborate on "be very careful with volume mounts"?
In the k8s env I run, the filesystem is backed by ceph, which creates persistent volumes that the different pods can claim. Is it just "be careful not to mount docker.sock"?
With IllumOS I can agree on zones, I can partially agree on lpar on AiX, I can accept GNU/Linux limits with cgroups. Enlarging the attack surface to follow the current wannabe Gurgle mania... No. Like no in the recent past to full-stack virtualization on x86 to enrich VMWare, buy big desktops sold as powerful servers just to play matryoshka doll for someone else business...
Of course, it's hard to satisfy hw "subdivision" vs sw subdivision, in the past we have tried the hw path with blades for the small side, infiniband and co for the large side and both proved to be not very scalable but the issue is in actual OS design and can only be solved changed that: An OS mush be a single application, with internal namespacing, networking. We have seen it in various past OSes. It's about time to make them again accepting that the IBM model is good only for IBM, witch means for selling stuff to people having no choice but buy or ignoring something better can be done.
I would say it's not necessarily easier to manage, but it's much easier to understand, because containers create natural borders between the different software running on the machine.
Docker actually comes with an apparmor config that disallows you to interact with any files outside of the container. It's not a so strong guarantee compares to inode based approach (selinix), but is still much better than don't have one.
And if you enables uid mapping, your permission in the host system is just as high as nobody if there isn't a kernel exploit happened somewhere.
But the fact it is nobody is also problematic, because dir mount will be almost alwyas read-only now.
It comes with both an apparmor profile and an SELinux profile. What gets used depends on what's on the system... and I guess you need to actually set the flag on dockerd to enable SELinux.
I've always preferred apparmor. SELinux has always seemed radically more complex for very little benefit, unless you have a tightly constrained OS (like Android, where the VM does most of the work and every app has the same sort of security policy) or a team of admins working full time to maintain it (again, like Android).
Apparmor is weak in all the same ways SELinux is weak, at least in terms of the ways that actually matter - that is to say, a kernel exploit is the simplest way out of either. But anyone can write an apparmor profile for any program in an hour, and if you actually know wtf you're doing you can build very strong profiles and trivially test and maintain them.
SELinux is "good" in that if you are building a system like Android, great, consumers get very tight policies for free, and SELinux is ultimately more expressive. But I think 99% of companies looking at an LSM MAC should probably roll apparmor.
I think my main problems with AppArmor are the feature set isn't coherent across all distros that use it and it isn't integrated too well with audit tools. OpenSUSE ships distro patches that are a major hack to make ausearch properly show AA denials, while others need to look at dmesg and remember to disable printk ratelimit. The supported feature set is fun, the kernel mainline vs Ubuntu vs SUSE feature sets differ a lot. Ubuntu supports fine grained socket mediation (as in specific ops) & dbus. OpenSUSE supports some of the socket mediation, but as life has demonstrated some rules that work on Ubuntu are incorrectly downgraded and for instance all unix socket access gets blocked (though thanks to the AppArmor folks this got fixed now). Arch which I'm considering to be close to mainline has no dbus and no fine socket mediation. In theory one can use the same policy file on all systems, but the way it gets downgraded is not always predictable. At least the features are somewhat discoverable though /sys/kernel/security/apparmor tree, but I don't know what happens when say the kernel reports dbus mediation, but dbus-daemon was built without it. Then there are also some missing features, I think the abstract socket mediation is still not there. Lastly, the parser is super slow and memory hungry, especially when throwing some AARE heavy profiles at it. It happens that Ubuntu Core makes heavy use of apparmor profiles to lock down snaps on embedded Linux devices, and all the timings data that we collect show calling out to apparmor parser taking the largest chunk of the snap installation time.
FWIW mediation of dbus & network is already supported by SELinux in all implementations I've tried, but then SELinux has a separate set of warts. Eg. trying to use the same source policy and building it on hosts with different tooling/refpolicy. Using `optional_policy` doesn't cut it if types are not known at all, and as debugged with good folks from #selinux, the only way to work around this was using `ifndef` and passing defines to m4 interpreter.
IMHO most RedHat systems tend to be overengineered. Even the config file split is overengineered in RedHat compared to many other distros. And documentation, although existing, tends not to be detailed enough to understand how to deal with the systems. You need trial and error.
Well, unnecessary complexity full of YAGNI is exactly what the E in RHEL stands for. Red Hat products need to be complex enough so that somebody will pay for support. :)
> There’s nowhere on the system where you can view the policies and look up why something might or might not work.
I always thought I had to be missing something with SELinux because this is what it seemed like to me and that can't be right. My impression is that the documentation for SELinux is extensive in all of the areas that aren't affecting you but it's really hard to nail down exactly what the policies are, what labels are available, which labels you should be using, and generally how everything interacts. Is there a SELinux Wiki somewhere that I've missed that has a simple breakdown of each and every possible label with interactions? Some tool I can use to generate said list?
Which says that things (i.e. processes) with the httpd_t label are allowed to perform getattr, ioctl, lock, open, read, and search operations on directories with the httpd_t label.
ps -Z and ls -Z will then list the labels of processes, and the labels of files and directories. If your processes label does not have an allow rule for your file label for the correct action you are denied.
The only trick really is that when it reports a label as say "system_u:system_r:httpd_t:s0" thats 4 different colon separated fields and the only one you really care about is the type field "httpd_t", because thats what the rules are defined for (unless you're the NSA or doing Very Interesting things (the other fields are user, role, and security level, but the default Red Hat etc selinux policies dont use them))
There is no wiki with definitive list of labels because they are just strings and the policy writer can call them what they like, even the _t suffix is just a convention to denote the type.
I have to say, for sysadmin work I find this view sorta overblown.
My experience is with RHEL and derivatives, not Fedora, and for the most part those are very straightforward and while selling does rear it's head every once in a while, it's usually not that problematic to work around.
When you have something failing that you need to fix and isn't an upstream problem (i.e. your own app or some third party vendor), you set the system to permissive mode and let it run with the app long enough that you think you've hit all the use cases, and then you use ausearch and audit2allow to generate a module for it. And bi, while the selinux audit log entries (which ausearch shows) aren't super clear, you can learn to interpret the gist of them, and piping them to audit2why and audit2allow gives more context.
The real "trick" in the above is to make sure you run the system in permissive mode for a while. Otherwise you'll usually only see the first deny in a code path which the app will then fail on, and you won't see subsequent ones. Permissive mode means nothing is actually denied, but all thing that would be denials are logged as such.
For things that ship with the OS repos for RHEL and derivatives, my experience is that very rarely are there problems or things that are not obvious to fix (running errors through audit2why will tell you when there's a boolean to toggle to allow what you want, such as outbound tcp connections from the http context that is causing some PHP app to malfunction).
In my experience creating and applying new policies with audit2allow is almost always the wrong thing to do. Most SELinux issues are either a SELinux boolean that needs enabling, or SELinux file type that needs changing.
The few times where you do need to create new policies then audit2allow is useful but still requires you to review and understand the changes it's going to make before applying them.
> In my experience creating and applying new policies with audit2allow is almost always the wrong thing to do. Most SELinux issues are either a SELinux boolean that needs enabling, or SELinux file type that needs changing.
I think that's very dependent on what you're doing, and how supported the thing your doing is by the OS. We write a lot of custom software. We also end up installing a lot of applications that didn't ship with good (or any) rulesets, or that we packaged into RPMs ourselves. Using audit2allow (after first checking audit2why and booleans) has become standard enough in those cases to generate a ruleset that makes sense (either for immediate installation or adding to a package we're building) that my experience seems to be somewhat different than yours. Or maybe we just run a lot more servers and while the percentages are the same the total numbers look higher, I don't know.
There's also the cases where we basically threw our hands up and said screw it, run in permissive mode indefinitely, because the alternative it too annoying and error prone (I'm looking at you, Jira).
> The few times where you do need to create new policies then audit2allow is useful but still requires you to review and understand the changes it's going to make before applying them.
Yep, and I noted that I don't think the audit logs and autdit2allow output aren't actually all that hard to understand if you take the time to look.
Exactly this. I’ve run SELinux on for various workloads for over a decade and have barely ever had to touch audit2allow. And what I have done is codified in ansible so you can just copy and adjust for new cases.
> and then you use ausearch and audit2allow to generate a module for it. And bi, while the selinux audit log entries (which ausearch shows) aren't super clear,
That still sounds pretty complicated for most people. I mean, I know a lot of otherwise capable people who think NoScript is too much to manage and that's something I've heard repeated online many times. That's just clicks, and only when you actually need to allow something. I would think that if even the people who own websites where they write entire articles about linux find something over-complicated and poorly documented most computer users wont stand a chance.
The three paragraphs starting with "In the best-case scenario" address exactly what you're talking about. And in my brief (hobbyist) usage of CentOS 7 with trying to run some self-written software, what TFA says was exactly my experience.
As it turned out, the hardening provided by systemd (private tmp, restricted syscalls, etc) was sufficient, and easier to understand, for my use case.
Sure, but it also sounds like your usage isn't really sysadmin level, which is what I prefaced everything I said with to put it into that context.
I don't expect random users to run with selinux on and deal with problems for all the stuff they might do. I do think it's probably a good idea for system administrators, and if those sysadmins have problems getting some custom in-house software running through selinux, that seems like a good place for them to focus some learning to rectify that issue.
You can also write a stub policy which is permissive for all actions, with logging turned on. This means you can run just the one application in permissive mode, get a sense of its behavior, and then lock it down.
Like others here, I had a similar experience. I setup a simple minecraft server on a SELinux secured OS. So far, so good. I wanted to setup a systemd service to startup and shutdown the minecraft server.
After ~1 hour of work later, I came to the conclusion that I was going to disable SELinux. Another hour later, I disabled SELinux.
Much as the article mentions, there didn't seem to be much good help, especially w.r.t. learning what the incantations meant and how to use them properly outside of a narrow path. Similarly, I did not have any decent way of introspecting what was going on in there. The error messages were of the "you must google this to even have a remote chance of figuring out what it means at all."
I understand SELinux is probably designed for enterprise or organizational specialists and not for normies to touch. It just seemed a bit too extreme towards that end.
You hit the main issue with selinux: most documentation is aimed at policy developers and not end users. Your main issue was that you were trying to adapt the reference policy without knowing its inner workings.
I was in a similar situation. It always takes me way to long to find out that SELinux is responsible for a seemingly impossible to understand error behavior, until I give up, ask a colleague, he mentions SELinux and then it's suddenly obvious. So my conclusion is that the fail-behaviour is just not good. I would prefer my process to be killed with a nice clear error message somewhere. I think it all about giving the user the right information instead of failing silently.
Apparmor is -much- easier to write policies for; even more so if you use one of the ubuntu based distros that have decent rules already set up for most apps you'd use.
the linux fiefdoms have a serious UX problem. SELinux being a prime example. As the article articulates, no wonder why people just turn it off. If your subsystems are not consistent, discoverable, palpable, and most important logical your setting yourself up for lousy adoption. And just "reading the docs" does not solve this problem. Your subsystem does not get to consume my professional time slice.
The reason docker became the de facto entry point into containerization in yesteryear is because if you were dealing with 'containers' you were dealing with the 'docker' cli entry point. Everything you did with linux containers in the (mainstream) came from 'docker' and you can '--help' to your hearts content -or- google as much as you required with others that had the same shared experience with 'docker'. We've moved on in recent years but its important to remember the power of a well described, but imperfect interface.
SELinux has none of this mindshare. What is my canonical entrypoint to SELinux on any particular distro? There is none. I have to specifically know to install support packages for 'audit2allow' or 'audit2why' to do any reasonable troubleshooting on why a processes wont start. Why? Because any raw logs are so chocked with implementation details as a administrator I cannot make a real-world decision on what is broken on the system. Sysadmins do not start every day thinking about SELinux and memorizing its maze of tools and procedures. Something is starting to smell here...
For SELinux I need to know about, and sometimes explicitly install, half a dozen cli tools to administer selinux. Most of which don't follow any particular naming convention or entry point. I now need to learn a completely new markup for policy AND compile them AND install them using other esoteric tools . I need to explicitly refresh system state after making any changes, and return to my blunt 'audit2why' what-is-this tool to figure out if I did anything right.
The principles of SELinux are fine. The UX of SELinux in terms of getting shit done day to day is not.
when you acknowledge this, then continuing your advice with
setenforce 0
is a spectacularly bad idea. You can make individual domains permissive using
semanage permissive -a ${context}
and then run the failing test using only that permissive domain.
Of course, the story becomes much more complicated if the failing test requires type transitions -- but blanket advising people to put an entire PROD system in permissive mode is not a good idea.
SELinux has a horrible misfeature called dontaudit, that lets policies using it deny actions without any evidence being logged anywhere. Because of the existence of this, the only reliable way to know if a problem is being caused by SELinux is to temporarily disable it and see if the problem goes away.
After years of setting permissive on my personal machine I wanted to do the right thing and set it up properly. I watched the presentations and all seemed so reasonable. It's there to help you, audit2allow will help you out even further if you need to allow something. Look at how detailed those error messages are explaining the problem and the solutions.
I thought I was going crazy that things weren't working and there were no denials in the log. After much pulling of hair I finally found out about dontaudit, I felt so cheated, it wasn't even playing fair at that point.
For anyone else hitting this, you can use semodule to disable dontaudit in step 4 [1]
Not only that, but the official policies rely on dontaudit rules because there are known policy violations that are in fact harmless (like processes scanning entire directories, or attempting to open a file in various locations). So disabling dontaudit means your logs will get cluttered with non-related error messages that were up to then silently ignored.
Readit News
The whole IT ecosystem has become a hail mary. Even admins usually have no idea what a certain program actually wants to do. If the admin knows how to install the app so that it actually runs, you call them a good admin.
From a security point of view, an application is like a nuclear power plant. It's good if it works as planned, but if something blows up it endangers your whole enterprise.
The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl. That way the radiation hopefully stays in, but history has shown that it really doesn't. Also, the wheel of history has just turned one more iteration and now admins just view the sarcophagus as something you deploy as you previously deployed the app. Who is responsible that it is air tight? Well, uh, nobody, really.
You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.
Obscure files like /etc/localtime, /etc/resolv.conf, /etc/ld.so.conf, /dev/zero ... how can you expect devs to build well documented and well sandboxable applications if they don't know which files their library functions will open?
You may have heard of /etc/resolv.conf ... but have you heard of /etc/gai.conf? /etc/nsswitch.conf? /etc/host.conf? Wouldn't it be great if the man page of getaddrinfo mentioned those (mine only mentions gai.conf)
I disagree.
> but that we all run software we don't understand.
I fully agree.
My disagreement lies in the fact that you've described the problem, but are proposing that some software (SELinux) that fails to solve the problem is somehow good.
SELinux might be a perfect tool in an ideal utopia where everyone understands all the software they run, but that isn't the real world, and having a tool that works in a theoretical world isn't particularly useful.
Either SELinux is applicable to the real world, or it's not useful and not really fit for purpose.
This isn't a "blame game" - it's not about figuring out why things are bad (no-one understands their software components), nor blaming SELinux (it hasn't caused these problems, it's just failing to mitigate them). It's about figuring out how to improve that situation. Does SELinux do that effectively?
My opinion is that there's a cultural and policy problem and you're simply not going to solve it with technology. I don't think SE Linux is bad, it just isn't the answer here. One (clearly not the only) main reasons people just run a bunch of software they don't understand is due to expectations in productivity. In an idealistic world with caution built in, we are willing to sacrifice some productivity for security, stability, understanding, etc. I think many engineers like these ideas because they like to work with and build solid systems they understand very well--otherwise they wouldn't be practicing any form of engineering.
What we instead have is a world of engineering with no sort of guard rails. Unlike Civil Engineering, there's no licensing, no sorts of inspections, no oversight, liability is completepy waived in a 300 page agreement people just click through, etc. Even when things go horribly wrong (e.g. Equifax) the repercussions are limited. We see abuse in other engineering fields where actual life and death are factors and they're still just amortized in a cost analysis.
Ultimately though engineers simply don't lead the world (and arguably, maybe they shouldn't) , people with wealth lead the world (arguably, they shouldn't either) because it underpins pretty much all other systems anymore. As such, productivity is always prioritized over safety and security. This issue isn't limited to software, it happens everywhere almost ubiquitously at this point. Software is one of those areas where most people just care less, so you can get away with more and more of this. The issue is that I believe we're building a house of cards that will be difficult to revert to a reasonably stable state once it finally collapses.
It's not a tool for application developers (who at most can supply SELinux templates to start with); nor is it a tool for sysadmins (who really don't have the time or the incentives to deal with it.) It's a tool to be wielded specifically by people occupying a role that, in most organizations, doesn't exist.
If you’re using SELinux rules off the shelf for a handful of Fedora boxes for personal use, maybe that use case doesn’t work. But is that more “real world” than an institution like the NSA trying to completely lock down its production systems?
System design, though, is the actual point of SW ENGINEERING. That’s the part that is responsible for creating a foundation of quality to build on.
The other side is that sysadmins have largely become DevOps or SRE roles, which means most folks left as a sysadmin are those that couldn’t hack it in the other roles. In the end, as we grow the number of people in tech, the number of software and systems, and the complexity we experience a regression to the mean across the board.
It’s always been a pet peeve of mine that I can count on my hand the number of SWEs in my ~20yr career that had similar understanding of the system they worked with as their SRE counterparts… but this should be table stakes.
Yes. It's absurd that anyone that can talk about this stuff at a basic level is seen as some sort of ultra-skilled professional. See also: the relatively small amount of chatter about this stuff in the blogosphere, where there's a million tutorials that amount to "here's how you run create-react-app" and about 3 that talk about even moderate system design topics.
I think this is a bit unfair, the software industry is so big now there are lots of different sub-sectors. Big Tech companies have huge scale problems that are kind of unique. Most developers work on products that only have a few hundred users at a time.
Your 20 years experience is a bit telling. I'm the same age, back then SWEs usually had full access to production environments. Now everything is segregated and devs can't experiment in prod and end up getting detached.
Where people tend to fail is when they have to implement these systems, and they've planned for all these high level building blocks but then actually have to configure them. Redis, RabbitMQ, GraphQL, Docker, NGINX, the list goes on - there's so much to configure and you really don't truly know how they work unless you dive into the source code, so you inevitably get bitten by subtle bugs and errors where your expectations of how something should work based on the docs are violated by how it actually works.
Should a SWE have a deep level understanding of all this stuff? I'm not sure. Usually you can still hack it into a working system after a little bit of firefighting, and that's likely more time efficient than understanding source code or rolling your own infrastructure. But I definitely think there's room for more comprehensive and transparent documentation to bridge the gap between the source code and barebones docs.
> System design, though, is the actual point of SW ENGINEERING.
I mean, I would say I'm pretty good at engineering and system design (10 YoE, senior/principal for most of them). But I dread the system design interview questions because (in my experience) they bear almost no relation to the system design experience I have and almost no relation to the actual requirements of the job. Things like "design an ETL flow" or "build a distributed wikipedia downloader" are just not in my area of experience. (I work mostly in the lower layers of single-host operating systems and file systems.)
Software Engineering is about problem solving using software and hardware (we'll just say computers for simplicity). Whatever solution system you propose has inherent constraints and tradeoffs. That's the Engineering. System design is a catchall phrase for a subjective problem scale.
Deleted Comment
Deleted Comment
Deleted Comment
SWE = Software Engineer (obviously)
I thought system design was the point of system engineering?
There are many many problems.
One of the biggest problem with SELinux is that it is trying to graft Mandatory Access Controls on a userland that is not designed for it.
Unix, frankly, is not designed for security. It is designed to get work done by writing a bunch of little buggy C programs that you string together in novel ways.
Security is something that was grafted on it. And it shows.
How many decades of security vulnerabilities have occurred because of a shared /tmp space? 40 years?
How do you graft access controls on a system designed with no access controls and potentially billions of combinations of programs, paths, and various other resources without breaking anything?
The answer is: You don't. You can't.
Were as you have a system designed for security, like Android, and literally hundreds of millions of fully SELinux-enabled fully locked down user-facing Linux devices are out there being used by people who haven't the faintest clue what "audit2allow" is and wouldn't understand it if you tried to explain it to them.
So it's less of an issue of "we all use software that we don't understand". SELinux is complicated enough that you can devote your life to trying to understand it and still fail to craft good rules for other people.
It's more of an issue of "Linux userland follows the basic Unix design from the 1970s which is kinda shit if your goal is security".
It is just bad design. Pure and simple. That is all there really is.
However there is a way out.
The way out is to give each process and each user their own little itty bitty special Unix environment where they can do whatever they want. And then you use strong divisions to keep them mostly unaware of each other. Use a default deny policy and only poke holes in it when required.
"Just download one from the app store," I was told. "But be prepared to grant it network access, file access, contacts access, camera access, and email privileges."
No telling how, or with what components, or what the dependencies or security implications are, or how to see the logging for the app. Just a single paragraph of instructions and you can be the next internet startup IPO!
'Hail Mary' is the best way to describe the situation I've yet heard.
I always felt containers were always about packaging and deployment, not security. Any "security" was a byproduct of isolation, not an end goal.
And of course, the main reason they do not take the responsibility is that the customers won't pay for it.
It is kind of interesting. We are very good at making bridges that do not fail unexpectedly, even if there is unlimited amount of unknown failure modes when working with physical materials. And on the software side, well, managing to come up with a fizzbuzz without failure modes is used to screen people on interviews. What would be similar for bridge engineers? Here is a hammer, nail and two pieces of wood. Can you make the two pieces of wood stick together?
The equivalent to fizzbuzz would be something like a free body diagram.
That security model is almost completely useless now. Terminal servers are an extreme edge case. Services implement their own security model between users. Special “Users” for services is just a hack that tries to use an inadequate system as much as possible.
The Unix security model is deeply baked in everywhere and it’s nearly impossible to tack on a security model fitting to todays requirements afterwards.
Short simplified example highlighting DAC/MAC differences: DAC asks "is user Alice allowed to read Bob's files?" while MAC asks "is the SMTP server (subject user/role) process allowed to read private keys (object type) of the HTTP server (object user/role)?"
And if you're really motivated (read: want to have fun diagnosing unexpected file permission issues), you can associate files and processes with different security categories and levels (MCS/MLS), implementing horizontal and vertical separation. For example, "is this software update service allowed to read confidential files owned by the accounting database or keys used to encrypt classified information?"
There are other MAC systems besides SELinux as well.
The only approach that I see as viable without rebuilding the entire compute universe is VMs, either the web browser (JS and WASM sandboxes) approach or the qubes type approach.
Take for instance something like xscreensaver. Something in there needs to be setuid so that it can verify your password to unlock the screen. That something is fortunately a dedicated binary, and not every single screensaver, but still, that's bad. Writing that executable is a delicate thing. Get one of them wrong, and it's a glaring hole.
And /usr/libexec/xscreensaver/xscreensaver-auth is quite the thing. It links to a whole bunch of graphical libraries and talks to X11 to read your password, so that thing has a huge potential attack surface. Far more than seems comfortable.
What we should have instead is some sort of authentication service. A program confined by SELinux to only interact with a socket and the password database, speaking a simple but well designed protocol.
With that in place we'd have much simpler security. Policies get simpler because random stuff doesn't link to PAM anymore, and so doesn't end up touching a whole bunch of critical security related files. There's one thing on the system that deals with that, and it ends up with a small, understandable policy.
And the policy for the users is now expressed in terms of "Do we want this program to be able to validate user passwords?", which is also far simpler.
With that kind of redesign over time we could have much smaller and more understandable policies, and a more secure system as a result.
xscreensaver-auth uses PAM to authenticate, which (typically) runs a small program called unix_chkpwd, which is setgid shadow (not setuid root), as /etc/shadow is owned by root:shadow and is readable by group.
xscreensaver is one of the very few screen lockers that is reasonably secure, because the bit that locks the screen is pretty minimal and only links with a few fairly-low level X11 libraries, not a full-scale GUI toolkit.
> What we should have instead is some sort of authentication service. A program confined by SELinux to only interact with a socket and the password database, speaking a simple but well designed protocol.
That sounds pretty overengineered. All you need is a small binary that is setgid shadow, that can take username/password on stdin, and exit 0 if auth succeeds, 1 if auth fails. But we already have that: unix_chkpwd.
The people who designed the container/sarcophagus system. If it's not secure don't sell it as secure: see the difference between Linux containers and FreeBSD jails or Solaris zones.
> You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.
As a developer / software "engineer" it is your responsibility to know how the components you choose work and how much you can rely on them. When a structural engineer selects certain types of steel or concrete to be used in the construction of a bridge, it is their responsibility to know the characteristics of the raw materials.
When you ship a software product, enable SELinux or AppArmor and audit what it does: then ship an SELinux/AppArmor profiles.
See also OpenBSD's pledge(2) framework where the developer embeds system calls in their (C) code to promise to only access certain OS features:
* https://awesomekling.github.io/pledge-and-unveil-in-Serenity...
* https://man.openbsd.org/pledge.2
Unfortunately that is not always your choice and arguing against it most commonly costs you more than you like. My experience is quite often timelines get underestimated massively with software so you end up doing what you can with the resources given. If you pull this into the bridge metafore the bridge still stands but can collapse any time and as long as it doesn't collapse during rush hour and can be rebuilt before the next morning it's all fine.
It's not just sysamins -- developers don't understand SELinux, either.
As a dev, I find least-privilege sandboxing (e.g., look at how sshd is structured) and capabilities (e.g. FreeBSD capsicum) somewhat non-intuitive, but at least tractable to understand. SELinux is a blackhole.
> The people who designed the container/sarcophagus system.
Original comment misses two nuances with the gripe.
1. Container creators and software creators are not the same people. So they can (and should!) have different goals and priorities. For containers, security and reliability. For software, make it work.
2. There is a nuanced, different interface between software-system and container-system, created by encapsulating a system-like environment inside the container. Much of what software does can be encapsulated. Which allows a more controlled (and much smaller) subset to exit the container.
But that's not a thing. So distros are attempting to add it later themselves, which is a disaster.
That's a huge part of the problem. Was looking for a monograph to pass on my successor that wasn't so familiar with linux - modern linux - as in systemd, docker, nssswitch, pam_homed, network-manager is penetrable but more often than not I'm just looking at the c source in github and I am in my 30ies, starting in my teens running linux nonstop and I'm still throwing up my hands every other day...
Add time constraints and pressure to deliver (don't waste your time understanding this, just do xyz) and here we are - on the other hand it's not okay that you have to devote years of trail and error to get a comprehensive undestanding of the system.
I guess a nuclear power plant has at least a training plan and complete reference book - that still needs to be read and grokked and trained upon but modern linux is often just undestandable by reading the source if you hit a problem - for some projects like things in the freedesktop ecosystem and partly systemd even that is kind of difficult because i.e. this bug https://github.com/systemd/systemd/issues/19118 explains the problem - there is no overview documentation, no clear way to look up what's happening, not even a good way to introspect and it's several components that interact with each other fail subtly. I've chosen this one because I've also hit it, not because I want to blame systemd, which is not so bad there are much worse things out there but it's part of a trend to introduce complexity - not sure what is necessary complexity and what is unnecessary - went to uni in compsci without ever someone slapping out of the tar pit (http://curtclifton.net/papers/MoseleyMarks06a.pdf) in my face and maybe that's part of the problem.
Documentation on Linux has always been shit.
This almost looks like a hint to OpenBSD's pledge and unveil system calls.
I'm just a hobbyist, but regarding
we all run software we don't understand
what I like (again) in OpenBSD is that I feel I can largely understand/control what's happening in my system, it quite fits in my head.
Just my 2 cents.
Try doing the same thing on macOS or Ubuntu. Last time I looked carefully at the former I discovered, for example, daemons for classrooms (??) It’s a free-for-all.
I don't think these statements are meaningfully different. "too complicated" implies "...for humans to manage". Maybe that's sort of your point?
> You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.
Agreed. It's too burdensome for software developers to understand exactly what syscalls their program needs to make and the security implications of permitting those syscalls. It also doesn't help that Linux naming conventions and concepts are very counterintuitive (yes, dear Kernel Hacker, I'm sure they're very intuitive to you, but we lowly mortals struggle).
And unfortunately the SELinux policies are tightly coupled to the application such that you can't make SELinux policies the purview of a dedicated sysadmin expert and leave the appdev to the development teams. They have to collaborate which is swimming against the current of Conway's Law or else you make SELinux policies the responsibility of appdev and suffer their lack of expertise.
We had similar problems with operations in general, but containers largely solved this problem by allowing sysadmins to focus on a platform while developers focus on developing and operating the application. We need something similar for security. This is probably a rephrasing of your "sarcophagus" point?
Who the fuck invented that convention that fine-grained permissions must be file-based? It's insane. No, no developer will anticipate that he needs to read /etc/nsswitch.conf. No developer should. A developer should anticipate that the software needs permission to connect on network hosts.
As much as it is too granular, file based permissions aren't fine-grained enough. Asking to connect on random hosts is absurdly wide, most programs only need to connect in to a few of them, or connect to an user-supplied host (that can be a permission by itself).
Anyway, yes, the manpages should interlink better. What is a different issue.
Reminder: containers are not meant to be security tools.
Fine-grained sandboxing (e.g. seccomp) is. And it can be layered upon OS-level VMs.
Additionally, bundling tons of stuff together with an application (like docker but also flatpak do) is not good for security. Same for static linking. They all increase the workload of updating vulnerable dependencies.
I get the "don't reinvent the wheel" sentiment but, I think we take it too far sometimes. I've been looking at the source code for some dependencies at work lately, and many of them actually don't hold up to our own code quality standards. Kind of subjective, yes, but many of our dependencies would probably fail code review if actually reviewed by us.
Then when there is a bug in a dependency, nobody actually understands how it works and the code is often tucked away and not easily changed.
Used to be, you could update it manually. Then NetworkManager (or whatever its successor is) came along and would change it back on you. Oh well, you could still edit it for a quick test.
Then I set up a Wireguard tunnel. And it changed /etc/resolv.conf. Oh, but an entry is missing, I'll just add that and test. Nope, read-only. But I'm root! What gives?
Turns out the wg-quick script mounts a file on top of /etc/resolv.conf! I didn't even know that was possible until I saw it. Nobody messes with wg-quick's resolv.conf, and that's final! Until some other tool ups the ante and gets code to undo that.
There is truth to this, and it nails the fundamental asymmetry of the bad guys vs good guys in the security war. To program selinux you need to understand the software you run at the syscall level. and potentially have a deep understanding of its file usage, particularly if it’s doing ipc.
In general, I think that is a good goal. More understanding is more understanding and that is Good. In practice? I equate it to the problem of writing secure and robust code in C, I don’t know how good you have to be todo it and I basically assume that anyone who says they do is full of shit. I have contributed to the Linux kernel, I have decades of UNIX and specifically Linux experience as a software engineer, and I am still surprised when I fire up strace from time to time. You look at something like the recent Dirty Pipe bug, and I have a difficult time accepting that many people can fully grasp it all. The cost of a fairly simple system interface is all the subtlety and edge cases.
So, I don't think anyone seriously thought that containers provided significantly more security than simply running an application and trusting the kernel syscall and filesystem permissions were bulletproof. More software layers are unlikely to improve security.
Containers are a reaction to the flaws in the system. Fixing the root cause of those flaws is hard, because they're systemic. We would need to re-design everything to fix those flaws. Containers are a stop-gap measure that makes things easier to deal with.
The thing that I've learned about systems is they are all crap, but humans constantly munge them to keep them working. We don't end up with the best systems, we end up with systems easiest to munge. Every once in a while, a spasm of collective organization redesigns the system, and then we keep going with that one. It's evolution. So however crap our current system is, eventually it will change, probably for the better. It just takes a really long time.
We make a sandbox (in this case a very inefficient one) -- we assume the sandbox is the only way to deploy software, but mostly it's not because there will be times you need more than the sandbox or the software doesn't work, and then you're running "privileged" WASM routines.
The sandbox itself might even require some kind of leaky behaviour, such as accessing files or shared memory.
The wheel keeps turning. Some new sandbox will come along.
Completely disagree, if the target users are advising each other to disable it... then the tool is definitely more complicated than it needs to be.
Using tools that don't hide any complexity are very painful to use. It feels like the creator doesn't care about the user and put no thought into the display of information or workflows of the tool.
Solving a complex problem with a perceived complex tool is relatively easy... don't hide any complexity.
Solving a complex problem with a perceived simple tool is difficult... hiding complexity at the right time, revealing functionality it based on the user's intentions and experience is not easy but greatly appreciated.
This does not follow. I you ask a 5-year to repaint the Sistine Chapel and the result is somehow less than stellar, are you similarly going to blame the paintbrushes used? There exists no situation where the task at hand is too complicated for the user?
I can talk about snaps, but at least they are offering a specific permission model which is usable for users to understand. i.e. Via the interfaces, I can know software x accesses my password-manager, home directory, camera etc... I can disconnect access to the given permission and have it enforced via the kernel and apparmor.
The applications themselves bundle only libraries that themselves are sandboxed/snapped.
>> It doesn’t help that no one wants to be told: “stop what you’re doing, spend a week to learn this other thing, make a tiny change, and then return to your real job”. They’re looking for solutions — preferably quick ones — and not a week’s worth of study.
If you're a system administrator running SeLinux why dont you already know SeLinux? It's not like some obscure command, it's a big part of the infrastructure you're running.
If you want a real security on linux where root is not God !
Go to this site.
https://www.rsbac.org/
It's little difficult to implement (by kernel customisation) but there is a learning mode to secure all Linux structure
Be "root" is not be "God" after implementation. You will must ask to Security Officer (SecOff)
You can speak of "Evaluation Assurance Level" with this security solution and push SELinux into a trash.
Now, I think my advice would be: Put everything in a container, and learn how to run Docker or Podman (or k8s) in a secure way (ie no root containers, be very careful with volume mounts, etc). Yes, they aren’t as mature as SELinux, but containers aim to provide many of the same benefits that SELinux does (and even more) except in a way that’s much easier to manage. Even better is that these container runtimes often come with SELinux and AppArmor policies out of the box on good Linux distros.
Containers have made the conceptual shift from allow/deny to isolate/share. Somehow this feels better even though it is effecitvely the same.
I am still waiting for an abstraction that uses all the low level features and wraps them in a high-level interface that puts usability front and center.
I am not sure if this is even possible though because many applications are not built with sandboxing in mind. Adding another file somewhere on the system that needs to be accessed is not considered a breaking change by most. So maybe we need a more fundamental shift.
Bubblewrap? FlatPak's sandboxing features are built on it already.
Maybe that's just daydreaming about a perfect world, but I agree that it should be easy to reason about the access levels.
https://wiki.archlinux.org/title/Firejail
Also, afaik firejail runs as suid, making any possible escape much more serious.
We've had container breakout CVEs in the Kernel+ContainerD+CRI-O this year, and the ContainerD/CRI-O ones were not complex at all, so you do need to stay on top of patching as well as the other good practices mentioned.
1) The article author is Testing in PROD
2) selinux debugging relies on auditd, so sanity checks required.
After which the selinux debugging experience boils down to:SELinux also supports interesting things like applying security levels and contexts to data, so you can have data that is only accessible to appropriatly cleared users in one department but not to people with similar clearances in other departments. Very few people use this stuff though.
Selinux uses extended attributes to store its labels which is a pretty standard way of associating metadata with objects, and is supported by most standard unix commands (via the -Z flag) so I'm not sure what you mean by "filesystem hacks". Fundamentally they are both implemented as security modules in the kernel and do pretty much the same thing, which is best depends on whether the added security of SELinux is worth it to you.
I was daemonizing a uvicorn (Python) process using systemd and could not get the Python script to write its log file anywhere. Not the current directory owned by the user, not elsewhere, and no errors showing up in the system logs.
After a few hours banging my head against the wall I discovered systemd's ReadWritePaths. Once I set that, it would log to any directory I specified.
It's the only service I've created where this has been necessary, and others write logs, so I have no idea why this one caused errors. If I did something wrong, let me know how this should be handled.
edit: I like how I'm getting down-voted for sharing factual information. No-one in high frequency trading employs containers because it doesn't help soft-realtime, low-latency goals. There are other performance-sensitive domains where containers aren't an option. Sometimes I feel like this place is turning into Reddit.
What cases can linux containers not handle? Containers can access GPUs, /dev/kvm, block devices... I'm having trouble thinking of anything they can't do. After all, they're just processes in a glorified chroot, not that different from processes on the linux host.
In the k8s env I run, the filesystem is backed by ceph, which creates persistent volumes that the different pods can claim. Is it just "be careful not to mount docker.sock"?
Absolutely not. They are not designed as security tools in the first place. Especially docker, with its huge attack surface.
You are confusing containers with sandboxes.
Furthermore they don't implement anything resembling RBAC.
Or just use Qubes OS: https://qubes-os.org.
With IllumOS I can agree on zones, I can partially agree on lpar on AiX, I can accept GNU/Linux limits with cgroups. Enlarging the attack surface to follow the current wannabe Gurgle mania... No. Like no in the recent past to full-stack virtualization on x86 to enrich VMWare, buy big desktops sold as powerful servers just to play matryoshka doll for someone else business...
Of course, it's hard to satisfy hw "subdivision" vs sw subdivision, in the past we have tried the hw path with blades for the small side, infiniband and co for the large side and both proved to be not very scalable but the issue is in actual OS design and can only be solved changed that: An OS mush be a single application, with internal namespacing, networking. We have seen it in various past OSes. It's about time to make them again accepting that the IBM model is good only for IBM, witch means for selling stuff to people having no choice but buy or ignoring something better can be done.
I would say it's not necessarily easier to manage, but it's much easier to understand, because containers create natural borders between the different software running on the machine.
And if you enables uid mapping, your permission in the host system is just as high as nobody if there isn't a kernel exploit happened somewhere.
But the fact it is nobody is also problematic, because dir mount will be almost alwyas read-only now.
That is, it does nothing to address security problems latent to each container binary.
Apparmor is weak in all the same ways SELinux is weak, at least in terms of the ways that actually matter - that is to say, a kernel exploit is the simplest way out of either. But anyone can write an apparmor profile for any program in an hour, and if you actually know wtf you're doing you can build very strong profiles and trivially test and maintain them.
SELinux is "good" in that if you are building a system like Android, great, consumers get very tight policies for free, and SELinux is ultimately more expressive. But I think 99% of companies looking at an LSM MAC should probably roll apparmor.
To expand a little, for a vertically integrated OS like Android or RHEL (official packages only!) SELinux can work.
For a sysadmin trying to configure a custom service or anything outside official repositories then AppArmor is actually usable where SELinux isn't.
But what you probably want is a Docker (or Podman) container, which gets you 90% of the hardening with 1% of the effort.
If a container isn't viable for whatever reason, then a systemd "contained" service is also very easy to configure. See `man systemd.exec`.
FWIW mediation of dbus & network is already supported by SELinux in all implementations I've tried, but then SELinux has a separate set of warts. Eg. trying to use the same source policy and building it on hosts with different tooling/refpolicy. Using `optional_policy` doesn't cut it if types are not known at all, and as debugged with good folks from #selinux, the only way to work around this was using `ifndef` and passing defines to m4 interpreter.
I always thought I had to be missing something with SELinux because this is what it seemed like to me and that can't be right. My impression is that the documentation for SELinux is extensive in all of the areas that aren't affecting you but it's really hard to nail down exactly what the policies are, what labels are available, which labels you should be using, and generally how everything interacts. Is there a SELinux Wiki somewhere that I've missed that has a simple breakdown of each and every possible label with interactions? Some tool I can use to generate said list?
sesearch -A will show you every thing that is allowed, and gives you (a lot of) lines like:
allow httpd_t httpd_t:dir { getattr ioctl lock open read search };
Which says that things (i.e. processes) with the httpd_t label are allowed to perform getattr, ioctl, lock, open, read, and search operations on directories with the httpd_t label.
ps -Z and ls -Z will then list the labels of processes, and the labels of files and directories. If your processes label does not have an allow rule for your file label for the correct action you are denied.
The only trick really is that when it reports a label as say "system_u:system_r:httpd_t:s0" thats 4 different colon separated fields and the only one you really care about is the type field "httpd_t", because thats what the rules are defined for (unless you're the NSA or doing Very Interesting things (the other fields are user, role, and security level, but the default Red Hat etc selinux policies dont use them))
There is no wiki with definitive list of labels because they are just strings and the policy writer can call them what they like, even the _t suffix is just a convention to denote the type.
My experience is with RHEL and derivatives, not Fedora, and for the most part those are very straightforward and while selling does rear it's head every once in a while, it's usually not that problematic to work around.
When you have something failing that you need to fix and isn't an upstream problem (i.e. your own app or some third party vendor), you set the system to permissive mode and let it run with the app long enough that you think you've hit all the use cases, and then you use ausearch and audit2allow to generate a module for it. And bi, while the selinux audit log entries (which ausearch shows) aren't super clear, you can learn to interpret the gist of them, and piping them to audit2why and audit2allow gives more context.
The real "trick" in the above is to make sure you run the system in permissive mode for a while. Otherwise you'll usually only see the first deny in a code path which the app will then fail on, and you won't see subsequent ones. Permissive mode means nothing is actually denied, but all thing that would be denials are logged as such.
For things that ship with the OS repos for RHEL and derivatives, my experience is that very rarely are there problems or things that are not obvious to fix (running errors through audit2why will tell you when there's a boolean to toggle to allow what you want, such as outbound tcp connections from the http context that is causing some PHP app to malfunction).
The few times where you do need to create new policies then audit2allow is useful but still requires you to review and understand the changes it's going to make before applying them.
I think that's very dependent on what you're doing, and how supported the thing your doing is by the OS. We write a lot of custom software. We also end up installing a lot of applications that didn't ship with good (or any) rulesets, or that we packaged into RPMs ourselves. Using audit2allow (after first checking audit2why and booleans) has become standard enough in those cases to generate a ruleset that makes sense (either for immediate installation or adding to a package we're building) that my experience seems to be somewhat different than yours. Or maybe we just run a lot more servers and while the percentages are the same the total numbers look higher, I don't know.
There's also the cases where we basically threw our hands up and said screw it, run in permissive mode indefinitely, because the alternative it too annoying and error prone (I'm looking at you, Jira).
> The few times where you do need to create new policies then audit2allow is useful but still requires you to review and understand the changes it's going to make before applying them.
Yep, and I noted that I don't think the audit logs and autdit2allow output aren't actually all that hard to understand if you take the time to look.
That still sounds pretty complicated for most people. I mean, I know a lot of otherwise capable people who think NoScript is too much to manage and that's something I've heard repeated online many times. That's just clicks, and only when you actually need to allow something. I would think that if even the people who own websites where they write entire articles about linux find something over-complicated and poorly documented most computer users wont stand a chance.
As it turned out, the hardening provided by systemd (private tmp, restricted syscalls, etc) was sufficient, and easier to understand, for my use case.
I don't expect random users to run with selinux on and deal with problems for all the stuff they might do. I do think it's probably a good idea for system administrators, and if those sysadmins have problems getting some custom in-house software running through selinux, that seems like a good place for them to focus some learning to rectify that issue.
Deleted Comment
After ~1 hour of work later, I came to the conclusion that I was going to disable SELinux. Another hour later, I disabled SELinux.
Much as the article mentions, there didn't seem to be much good help, especially w.r.t. learning what the incantations meant and how to use them properly outside of a narrow path. Similarly, I did not have any decent way of introspecting what was going on in there. The error messages were of the "you must google this to even have a remote chance of figuring out what it means at all."
I understand SELinux is probably designed for enterprise or organizational specialists and not for normies to touch. It just seemed a bit too extreme towards that end.
It’s just that SELinux documentation (something like a cookbook) is (or was) very hard to find on the internet.
The reason docker became the de facto entry point into containerization in yesteryear is because if you were dealing with 'containers' you were dealing with the 'docker' cli entry point. Everything you did with linux containers in the (mainstream) came from 'docker' and you can '--help' to your hearts content -or- google as much as you required with others that had the same shared experience with 'docker'. We've moved on in recent years but its important to remember the power of a well described, but imperfect interface.
SELinux has none of this mindshare. What is my canonical entrypoint to SELinux on any particular distro? There is none. I have to specifically know to install support packages for 'audit2allow' or 'audit2why' to do any reasonable troubleshooting on why a processes wont start. Why? Because any raw logs are so chocked with implementation details as a administrator I cannot make a real-world decision on what is broken on the system. Sysadmins do not start every day thinking about SELinux and memorizing its maze of tools and procedures. Something is starting to smell here...
For SELinux I need to know about, and sometimes explicitly install, half a dozen cli tools to administer selinux. Most of which don't follow any particular naming convention or entry point. I now need to learn a completely new markup for policy AND compile them AND install them using other esoteric tools . I need to explicitly refresh system state after making any changes, and return to my blunt 'audit2why' what-is-this tool to figure out if I did anything right.
The principles of SELinux are fine. The UX of SELinux in terms of getting shit done day to day is not.
Even after mastering all the fundamentals of SELinux, six month later a different audit-related problem surfaced and “what was that command again?”
This link often saves me:
https://access.redhat.com/documentation/en-us/red_hat_enterp...
In fact, I condensed it to the following steps (outlined elsewhere in this OP by patrck, new HN user):
couple sysadm red flags:
1) The article author is Testing in PROD
2) selinux debugging relies on auditd, so sanity checks required.
After which the selinux debugging experience boils down to:when you acknowledge this, then continuing your advice with
is a spectacularly bad idea. You can make individual domains permissive using and then run the failing test using only that permissive domain.Of course, the story becomes much more complicated if the failing test requires type transitions -- but blanket advising people to put an entire PROD system in permissive mode is not a good idea.
I thought I was going crazy that things weren't working and there were no denials in the log. After much pulling of hair I finally found out about dontaudit, I felt so cheated, it wasn't even playing fair at that point.
For anyone else hitting this, you can use semodule to disable dontaudit in step 4 [1]
[1] https://access.redhat.com/documentation/en-us/red_hat_enterp...