Purpose of first-define is the rule: In placing configuration files higher than user-defined configuration but Only with SSH client, can want user to have control from their config files: Remove from config files Place a couple under Match/MatchGroup using deny/accept.
SSHD (server/non-client) still support admin-defined by having system-wide settings done firstly. For those who have multi-file SSHD configurations, breakdown of the many config file locations and scopes here as it covers default user,
system-wide,
specific user:
Also I broken out each and every SSHD and SSH options along with their ordering by execution by using file name and numbering as well as its various state machine, dispatch, CLI equivalence, network context, and function nesting, all in:
With multiple config files overriding each other in an predictable order you can effectively allow users to change some settings while ignoring (overriding) whatever they set on others.
This is perhaps old sysadmin knowledge, but different tools have very different heuristics about how they parse configuration, and you have to check every time and not assume. Among the consequences to not checking are gaping security holes.
That's why I erase sshd_config and put what I really meant there. You may say "but isn't it better to patch it properly?". It is not. Yet another vps hoster –> yet another /etc/ssh directory template that may have all sorts of access issues in it. Better to replace it and make it do exactly what you have planned.
I've never liked the directory.d/* infrastructure. In so many cases, even with a properly configured sshd_config, the resulting configuration file is not so large that it benefits from being split up.
You have to deal with ordering issues, symlink management in some cases, and unless the "namespace" of sorting number prefixes is strictly defined, it's never something that's convenient or durable to "patch" new files into. The proliferation of 99_* files shows the anti-utility this actually provides.
I much prefer configuration files with a basic "include" or "include directory" configuration item. Then I can scope and scale the configuration in ways that are useful to me and not some fragile distribution oriented mechanism. Aside from that with xz I don't think I want my configurations "patchable" in this way.
Config directories are there to solve change management problems like idempotency.
If you have one big file then different tools, or even the same tool but different points of that tools life cycle, can result in old config not correctly removed, new config applied multiple times, or even a corrupt file entirely.
This isnt an issue if you’re running a personal system which you hand edit those config files. But when you have fleets of servers, it becomes a big problem very quickly.
With config directories, you then only need to track the lifecycle of files themselves rather than the content of those files. Which solves all of the above problems.
The .d directories are important on Debian and Ubuntu where packaging needs to provide different snippets based on the set of installed packages, the VM environment, other configuration inputs like through cloud-init and so forth, and update them during upgrades, but also (as per policy) preserve user customisations on anything in /etc.
Since pretty much every file has different syntax, this is virtually impossible to do any other way.
The conf.d isn’t because the config file is large. It’s because it’s easier to disable or enable something with an “echo blah > conf.d/10-addin.conf” or an “rm conf.d/50-blah.conf” than it is to do sed -i or grep blah || echo blah >>
1. They add huge configuration files where 99% are commented out.
2. Sometimes they invent whole new systems of configuration management. For example debian with apache httpd does that.
I don't need all of that. I just need simple 5-line configuration file.
My wish: ship absolutely minimal (yet secure) configuration. Do not comment out anything. Ask user to read manuals instead. Ship your configuration management systems as a separate packages for those who need it. Keep it simple by default.
Yeah, first-wins is definitely surprising. Off the top of my head, it feels like one would have to go out of one’s way to write a parser that does that (by storing an extra bit of state for each configuration item, and then checking it before setting the configuration item and toggling the state, rather than just applying the configuration item each time it is encountered).
Is there a good reason for this design? I can’t think of one, again off the top of my head, but of course I could be missing something.
It probably makes a bit more sense when you think about the fact that SSH frequently does a "try to match this host against a list of configured host-patterns" operation. In that case, "first match" is the obvious thing to do.
The SSHD internally has a config structure. Initially, it's all initialized with -1 for flags/numbers/timeouts, and with NULL for strings. When an option is parsed, there is a check whether the config structure's corresponding field is -1/NULL (depending on the type) before the parsed value is saved.
Another program with first-wins I've seen used dict/map for its config so the check was even simpler: "if optname not in config: config[optname] = parsed_value".
It's actually the simplest scheme. Reparse from the top whenever you need to query a setting. When you see one, exit. No need to even bother to store an intermediate representation. No idea if this matches the actual ssh implementation, but that's the way many historical parsers worked. The idea of cooking your text file on disk (into precious RAM!) is fairly modern.
Nope, the actual ssh implementation parses all the config files once, at the startup, using buffered file I/O and getline(). That means that on systems with modern libc, the whole config file (if it's small enough, less than 4 KiB IIRC?) gets read into the RAM and then getline() results are served from that buffer.
The scheme you propose is insane and if it was ever used (can you actually back that up? The disk I/O would kill your performance for anything remotely loaded), it was rightfully abandoned for much faster and simpler scheme.
Loading up your parsing code and reopening the file every time a setting is queried sounds to me like it would increase the average memory use of most programs.
Yes. Run this as a validation step during base os image creation, if such image is intended to start system with sshd. That way you can verify that distro you use did not pull the carpet from under your feet by changing something with base sshd config that you implicitly rely on.
The only time I hear or see anything about cloudinit, it is always a problem.
Nobody ever said "we don't need worry about that, cloudinit takes care of it".
In this particular case cloudinit presence in the story is incidental, delivery mechanism of said config file could have been different.
It's useful for initializing state that could not have been initialized before booting in the target environment. Canonical example, I guess, being ssh server and client keys management, but the list of modules it implements is long.
I initially read "the order of files in /etc/ssh/sshd_config.d/" to mean the order of files in the underlying directory inode, i.e. as returned by `ls -f` — and thought, "oh god"... But the lexicographical order, that's not too surprising.
but i guess learning is better late than never type of thing.
also what confuses people more on this is that openssh is properly designed, so configs are first seen wins. exactly so that file 0_ wins from 99_... but most people see badly designed software where 99_ overrides 0_. openssh way is exactly so it works best with confir or ssh/options where it matches by hosts and you can place the more important stuff first without fear defaults will override it.
Readit News
SSHD (server/non-client) still support admin-defined by having system-wide settings done firstly. For those who have multi-file SSHD configurations, breakdown of the many config file locations and scopes here as it covers default user, system-wide, specific user:
https://egbert.net/blog/articles/ssh-openssh-options-ways.ht...
Also I broken out each and every SSHD and SSH options along with their ordering by execution by using file name and numbering as well as its various state machine, dispatch, CLI equivalence, network context, and function nesting, all in:
https://github.com/egberts/easy-admin/tree/main/490-net-ssh
https://github.com/egberts/easy-admin/blob/main/490-net-ssh/...
Disclaimer: I do regular code reviews of OpenSSH and my employer authorizes me to release them (per se contract and NDA)
Also this showed how to properly mix and match authentication types using OR and AND logic(s) in
https://serverfault.com/a/996992
It is my dump mess so wade 'em and enjoy.
"In placing configuration files higher than user-defined configuration but Only with SSH client, can want..."
Also the most dangerous but flexible way to authenticate a user.
https://jpmens.net/2019/03/02/sshd-and-authorizedkeyscommand...
https://egbert.net/blog/articles/openssh-file-authorized_key...
You have to deal with ordering issues, symlink management in some cases, and unless the "namespace" of sorting number prefixes is strictly defined, it's never something that's convenient or durable to "patch" new files into. The proliferation of 99_* files shows the anti-utility this actually provides.
I much prefer configuration files with a basic "include" or "include directory" configuration item. Then I can scope and scale the configuration in ways that are useful to me and not some fragile distribution oriented mechanism. Aside from that with xz I don't think I want my configurations "patchable" in this way.
If you have one big file then different tools, or even the same tool but different points of that tools life cycle, can result in old config not correctly removed, new config applied multiple times, or even a corrupt file entirely.
This isnt an issue if you’re running a personal system which you hand edit those config files. But when you have fleets of servers, it becomes a big problem very quickly.
With config directories, you then only need to track the lifecycle of files themselves rather than the content of those files. Which solves all of the above problems.
Since pretty much every file has different syntax, this is virtually impossible to do any other way.
You don't have weird file patching going on with the potential to mess things up in super creative ways if someone has applied a hand edit.
With .d directories you have a file, you drop in that file, you manage that file, if that file changes then you change it back.
1. They add huge configuration files where 99% are commented out.
2. Sometimes they invent whole new systems of configuration management. For example debian with apache httpd does that.
I don't need all of that. I just need simple 5-line configuration file.
My wish: ship absolutely minimal (yet secure) configuration. Do not comment out anything. Ask user to read manuals instead. Ship your configuration management systems as a separate packages for those who need it. Keep it simple by default.
Is there a good reason for this design? I can’t think of one, again off the top of my head, but of course I could be missing something.
Another program with first-wins I've seen used dict/map for its config so the check was even simpler: "if optname not in config: config[optname] = parsed_value".
Deleted Comment
https://serverfault.com/questions/367085/iptables-first-matc...
The scheme you propose is insane and if it was ever used (can you actually back that up? The disk I/O would kill your performance for anything remotely loaded), it was rightfully abandoned for much faster and simpler scheme.
sshd -T reads the configuration file and prints information. It doesn't print what the server's currently-running configuration is: https://joshua.hu/sshd-backdoor-and-configuration-parsing
Every configuration change immediately applies to every new connection - no need to restart the service!
What good does cloudinit do really?
It's useful for initializing state that could not have been initialized before booting in the target environment. Canonical example, I guess, being ssh server and client keys management, but the list of modules it implements is long.
Well, why would it come up? You don't need to worry about things you don't need to worry about.
but i guess learning is better late than never type of thing.
also what confuses people more on this is that openssh is properly designed, so configs are first seen wins. exactly so that file 0_ wins from 99_... but most people see badly designed software where 99_ overrides 0_. openssh way is exactly so it works best with confir or ssh/options where it matches by hosts and you can place the more important stuff first without fear defaults will override it.
Deleted Comment
They've updated the documentation on /etc/ssh/sshd_config https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/20...