I disagree with this post so strongly - having spent most of my career installing, configuring, and managing other people’s software.
> The answer is that you, the end user, can not now. Every program is free to do its own thing and most do. If you have ever spent ages wondering why the exact same commands work when run from one terminal but not the other, this is probably why.
If the same program is behaving differently between two systems - it can -only- be the environment that’s different.
> Instead of coming up with a syntax that is as good as possible for the given problem, instead the goal is to produce syntax that is easy to use
Oh to be a developer. The “best possible syntax” is a universe of possibilities - environment variables are, thankfully, limited to strings. If all programs had to be configured with a Turing complete config language - that would just be a programming language! Limitations can set you free.
Sorry for the harsh tone. Please, and I believe I speak for most sysadmins, please continue to use environment variables.
One of the projects I currently work on, the configuration system/model/table is a monster.. I wish it was just strings. It basically contains boolean flags, strings, numbers etc but the most insidious one is, it can contain groups of related configs - meaning people started dumping stuff into that should be a normal table (ex, ShippingType:, Road, Rail, Air), so we get no foreign key constraints for reference data. This caused them to implement soft-deletes for it, so now you have some config values that float around forever because they were referenced somewhere (aka pseudo-foreignkey). It's so utterly dumb I want delete the whole thing but everyone thinks it works great (non-tech people).
I'm a developer, not a system-admin, but this is too much. Doesn't help that 5 different people has added 'features' to it over the years. The same can be accomplished with something waaaay simpler/cleaner. We also keep having config-related issues where people blame the system/servers/devops etc... every single time it is due to misconfig the project, so the system admins cannot do anything about it anyways (meaning we keep wasting their time, thinking the problem was with the servers). Let me pause here, need to take a blood pressure pill.
> The “best possible syntax” is a universe of possibilities - environment variables are, thankfully, limited to strings.
The same can be said for command line arguments. I usually prefer those instead of environment variables, because they must be explicitly specified instead of being implicitly passed by the parent process. I think of env vars as being more useful when repetitively calling commands interactively, to save some typing, or when you really do want processes to inherit config from parents (like PATH and its variants).
100% this. The developer behind Prometheus was a huge dick to people about env vars a while back, in similar fashion. Just the other day, they held another closed-doors vote after a year or so and finally decided they were OK.
Doesn't surprise me the creator of Meson of all people made the same dogmatic assertion.
I came here for this comment. I'm a developer and I have been using env vars for quite some time after I got burned numerous times by other options. I just can't see why I would use anything else.
> If the same program is behaving differently between two systems - it can -only- be the environment that’s different.
Yes, but the entire problem is that "the environment" is massive as it includes all hardware and software running on the device in question (and quite possible other devices as the network can easily be considered part of "the environment") which makes it difficult to track down differing behaviour.
"The environment" is not just environment variables. I've run into a spreadsheet bug where I got wrong results because of a CPU bug. Just because some global mutable state exists, that doesn't mean it's a good software design to have program behaviour depend on it.
>If the same program is behaving differently between two systems - it can -only- be the environment that’s different.
As a developer, I've generally used configuration files for changing the operation of my software and environment variables for information my code needs to know about WHERE the code is running. For example, the same code doing the exact same thing on 10 machines would have the same configuration file (or command line parameters in simpler cases) but the environment variables may change from machine to machine.
> Oh to be a developer. The “best possible syntax” is a universe of possibilities - environment variables are, thankfully, limited to strings. If all programs had to be configured with a Turing complete config language - that would just be a programming language! Limitations can set you free.
I was thinking about this while reading the post. What's the best syntax language or syntax for configuration, if environment variables are too simplistic and full-fledged programming languages are too powerful? That's why I'm interested in Dhall, which is a configuration language that's limited to compile-time only–i.e. it can't do anything at runtime.
This attitude is how we ended up with Active Directory. One assumes that the author doesn't spend much time in a command-line environment. The option to set defaults for one's normal working environment is obviously of value, and environment variables are hardly exclusive of configuration files (~/.bashrc, makefiles, ...). The theory that complex syntax and statelessness are inherently good and cost-free is naïve to the point of parody.
The author is correct that "this is the way we have always done it" isn't a good argument in and of itself to persist in a practice. However, they might be rewarded by a few minutes pondering a related idea: "if generations of people---many quite capable of modifying the system to use something else---persist in using something, it's possible they have a reason for doing so other than a deficit in competence or imagination."
On the facade environment variables may seem like they're orthogonal to global variables but they're not.
Environment variables are scoped to the current process. This could be your shell, but it could also be a web server. This doesn't make them leak proof, but unlike global variables, environment variables have a scope.
Environment variables are also used more widely as an API. Many CLIs have a command that when self-executed can act as a persistence layer in your shell. This functionality would be impossible without environment variables.
You mentioned the other motherload, which is that environment variables are quite often, again, used to communicate in Makefiles. You can see the natural scoping if you start to kick off an ad-hoc shell process inside a Make target.
I don't get this. From the same perspective you can argue global variables have scope too since they are "scoped to the current process". It's not like other processes can access your global variable, that's a very low bar for a scope.
Interestingly, reading this blog post, this doesn't seem to be common knowledge. The first comparison of global variables inside a process and environment variables left me wondering. It just felt wrong.
A gripe I have with environment variables is when they are used to modify a programs behavior deep inside it's belly and aren't treated like configuration input similar to program arguments.
Otherwise they are a universal way of configuring applications. Universal is good, universal is nice.
> Environment variables are scoped to the current process.
Obviously this is true in a sense. But for practical purposes, it depends on how the environment variables are set. For example, if I set them in my ~/.bash_profile, they're scoped to all bash processes that my current user runs. If I put them in /etc/profile, they're effectively global.
@ajarmst is that last related idea you quoted your own? That’s a powerful expression of how I see human culture: question everything, but remember to respect the ideas of the people who came before you. There might be a baby in the bath water you’re discarding.
It's a restatement of the lesson of Chesterton's Fence.
There is a fence somewhere you wish to get rid of. Nobody around knows why it was there in the first place. No one will let you tear it down until somebody figures out why it was put there in the first place.
The important take away, is that before changing something, one should understand the history of a thing, or one runs the risk of being a victim of the same problem the unknown thing was put in place to address.
Life is way more complicated than the umwelt of any one individual, so it is not safe to just change something without doing the footwork to understand what led it ro be in the first place.
Demonstrate that work has been done, and generally, no one will get in your way.
Strongly disagree. Environment variables are, IMHO, best tool for some simple configuration in unix. They match perfectly with behavior of the ecosystem and other tools in it (like unix shell).
Yes, if your OS is some unversal JS machine, then JSON would be better, if it is Lisp machine, then you would use S-expressions, but on Unix machine, environment/args are way to go.
There are two realistic alternatives - config files and arguments. They have each their own niche, where environment is somehwere between them.
Arguments are better for one-shot setting, not for some setting used always. You can use 'alias' to define shortcuts that always add some argument, but that is definitely more cumbersome.
Config files are good for always/default setting, but are too rigid. Changing config files is equivalent of changing global variable in code, it has system/user-wide effect. While i can just change environment in this one shell and it will affect just commands executed from that shell. Also, config files are much harded to be manipulated from scripts, and use different syntax for each tool.
Perhaps the ideal tool would allow every option to be set/changed from config file, environment and argument.
1. I worry about programs dumping all their environment variables to log files - credentials are now on disk, ingested into log storage...
2. Environment variables are inherited by child processes by default. This is undoubtable useful. But it can also cause problems.
I wish the ghosts of unix past had forseen the need for a way to mark particular variables as, say, 'sensitive' and 'noexport', allowing them to opt out of the default behaviour.
It would have been so say to say "variables starting with _ are not inherited and should be censored when output", but we're about 40 years too late for that to catch on...
I used to agree with (2), but now I think Meh, it's an implementation detail whether the program uses my environment variable 'directly' or with a child process, it's not meaningful to make that distinction.
When it is meaningful (and this is supported today) is to set them just for specific programs/invocations, rather than exporting for a long-running interactive shell (and everything within it) willy-nilly.
More innovation around making that easier would be interesting, env vars that should be set specified by program, isolated from others, for example. So `foobar` would actually get executed like `FOO_SECRET=hunter2 foobar` without specifying it every time or having it exported in the shell, and in a generic way not specific to each program's config.
It's not really related but for some reason 'summon' is on my mind as a tool to mention. I haven't used it in anger yet, but it is interesting. It's not quite this though, or at least, it solves only the 'storage' part of the implementation of what I described, not the 'orchestration' or mapping of programs to vars/summon invocations.
My first thought on the headline are specialized concerns of the above: environment variables are an attack surface. If you use them for configuration, it's all too easy for an attacker to modify them without the victim knowing. Just look at issues with LD_PRELOAD: https://attack.mitre.org/techniques/T1574/006/
That said, I agree with GP that environment variables are super useful and super simple. But I've also been burned more than a couple of times by setting something in the past and then having it caused unexpected bugs that are hard to trace down as they aren't in my working memory. They're a double-edged sword, to be sure.
1. If your program is chatty, it can be chatty in the same way regardless of where the improperly logged secrets come from; it's still your fault for being coarse and lazy. There's little difference between logging all environment variables(and/or all command line parameters) and logging the whole configuration object.
2. If your child processes shouldn't inherit environment variables, set them properly. The "ghosts of Unix past" have "foreseen the need" for execve(2) and execveat(2), which don't pass anything by "default".
> I wish the ghosts of unix past had forseen the need for a way to mark particular variables as, say, 'sensitive' and 'noexport', allowing them to opt out of the default behaviour.
The default behavior is a non-exported variable. If you want child processes to see it, you must export it.
I was taught many moons ago that configuration, like ogres and onions, is best considered in layers:
1. default values: What will most users in most places find most useful/least infuriating?
2. configuration files (system-wide, then user): What will most users on this system want most of the time? What will this particular user want most of the time?
3. environment variables: How should this session (i.e., a potentially large series of related executions) be tailored?
4. command line options: What is most useful for this particular run?
I was also taught that:
- figuring out how to go from an option to the name of a corresponding environment variable to a line in a config file should be both straightforward and well documented; and
- sometimes you need a more complex configuration than is cleanly supportable through any other method than a file. In such a case, the location of that file can itself be passed through options and the environment.
This is precisely how I setup my utilities. I have found following practices useful:
1. Print the path of the system and user level configurations that the utility honours in the help text (-h/-?)
2. For an option that can be set interactively or via environment variable, specify the environment variable name in the help text itself to provide maximum choice to the user.
3. Provide a -viewconfig option that prints out the final resolved configuration state so that the user can see the actual configuration that is in effect. Combined with a -dryrun option, this can provide a lot of confidence to the user to try out things without breaking anything.
My guess for this is that some people have not had the good fortune of seeing software that followed this pattern and how nice it is.
I thought it was common knowlegse that if really wanted to do configuration right on a given project, you do all 4 (with some library support) and you write your code to gracefully handle the right piece of configuration from the appropriate "override level" (again usually with the support of a good library).
See also: Domain Driven Design[0] which (if you ignore the consultant-fodder and jargon that comes with it) is probably one of the best written guides of how you should abstract systems, just like the gang of four book is a good introduction to structures in program/algorithm implementation you're likely to see in real life.
> Envvars have some legitimate usages (such as enabling debug logging) but they should never, ever be used for configuring core functionality of programs.
As though logging weren't core functionality!
The actual thing that is bad is grabbing an environment variable in the middle of your program. You should grab all the configuration in one place and use it to configure local state that is transparently passed around. Furthermore, flags, env vars, and config files are all just maps from strings to configuration, so you should use some system that can transparently layer them on top of one another. All of my new CLIs use flags first and fall back to ENV vars if the flag wasn't set.
I like layered config as well. It really should be the default way of thinking about config. Our custom application framework handles that exact sort of layering and it's wonderful. I give it an annotated class representing the config that I need and it handles populating the fields from the config and generating the help message if something is missing.
(for default values, make sure you consider safety too! for example a debug option that might show PII is likely to be most useful to most people using the program, but shouldn't default to on because if it were on in prod the consequences would be serious)
yeah, i don't get too worked up about "how" config values enter the application as long as i can easily see "where" they are initialized/validated.
an immutable config object/class created on startup that reads files/env vars/whatever and has appropriate assertions to ensure good values were used and crashes the app for missing/bad values usually keeps things sane.
an app where each subcomponent has its own config that it gets in its own way usually leads to confusion and delay
I have a pet peeve about this attitude. These methods are being used for decades and well understood with all their advantages and disadvantages.
One day, someone comes and tells that it's bad and considered harmful, and happily tells the only right way to do it. A flame war ensues then.
I'm all for moving things forward and evolution, but can't we take a milder stance and move forward in a more peaceful way? Attacking something so well established because of personal reasons feels so wrong from my PoV.
That thing wouldn't be a de-facto standard if it was too bad, right? I think we shouldn't play with the foundation that much.
> That thing wouldn't be a de-facto standard if it was too bad, right?
I disagree. Very often, the 'easiest' option wins, not the one that necessarily is the 'best', especially in the long term. Environment variables for configurations are fast and easy, and work well on the happy path - but break down quite fast when dealing with more complex cases (eg. more complex types than strings). They have a treacherous way of seeming the simplest and most pragmatic solution at first, but becoming a an untyped, underdocumented hairball after some time in more complex software.
Furthermore, as long as environments have existed in UNIX and UNIX-like derivatives, their usage to configure the bulk of the behaviour of most services/programs are relatively new. The more old-school the service you deploy somewhere, the more likely it has a file-driven config. Indeed, sometimes it seems like 90% of the Docker code out there is converting environment variables into configuration files.
> Indeed, sometimes it seems like 90% of the Docker code out there is converting environment variables into configuration files.
This is a consequence of Docker's choice of the "image" as an abstraction layer. It's not trivial to say "run this image but with this config file added" (yes you could bind mount one in, or create a new derived image, but those are both harder and come with more pitfalls).
In most common docker usage, there are exactly two ways to influence the operation of the program contained within the image: Environment variables, and command line arguments.
> I disagree. Very often, the 'easiest' option wins, not the one that necessarily is the 'best', especially in the long term.
Thank you for your disagreement and discussion, honestly. Actually, I think using environment variables are a burden. Needs more documentation, more explicit warnings, a lot of handling, etc.
So, environment variables are not the easiest way out there. Especially when almost any programming language has nice config file libraries out of the box. Instead these variables are added as a convenience feature for some frequent scenarios where tool needs to adapt itself to the environment it needs to run in, just before starting or needs to be run repeatedly with small, transient changes to the config.
> Furthermore, as long as environments have existed in UNIX and UNIX-like derivatives, their usage to configure the bulk of the behaviour of a service/program are relatively new.
This is not what I see in my career. Bulk of the applications we installed and ran used some forms of environment variables for runtime configuration of the tool/application.
The reason for that the variable had a great deal of effect in the behavior of the program (which was generally scientific) and making multiple runs without modifying a file very effective. You need these runs to conduct research BTW, and you're on a cluster and jobs run long and whatnot.
TBH, most of these applications also had configuration files or "sensible defaults" and they either created their default files if there was none. And if there was a file, the environment variable was acting as an override.
So I had experimental software, fixed most of the parameters in the file and tried some other things by overriding some parameter(s) with an environment variable. Nothing was abused or misused.
> Indeed, sometimes it seems like 90% of the Docker code out there is converting environment variables into configuration files.
I've never seen it TBH, and if that's not documented well, it would be a big bag of fun for the users of that code.
Sure, but everyone understands that option and knows the pitfalls. This is kind of the core of the “worse is better” mindset, which somehow seems to have disappeared from the collective consciousness even though Unix is bigger than ever.
> Attacking something so well established because of personal reasons feels so wrong from my PoV.
That is a mischaracterization of the post. The author is making a clear technical point about how environmental variables are global mutable state. Labeling that as an "attack because of personal reasons" is just plain misleading.
> That thing wouldn't be a de-facto standard if it was too bad, right?
How much of the post did you read? Your point is almost exactly the same as the 3rd listed in the post:
> It's the same old trifecta of why things are bad and broken:
> 1. Envvars are easy to add
> 2. There are existing processes that only work via envvars
> 3. "This is the way we have always done it so it must be correct!"
I don't think so. First of all, as I detailed in [0] and [1], my central point of disagreement is the tone and attitude of the post, not the usage of environment variables itself.
There are a lot of scenarios where environment variables makes a lot of sense, and scenarios where using them is absolute madness as we discussed with q3k in [1].
> How much of the post did you read?
All of it. BTW, please remember asking this question is directly against guidelines [2] (sec: In comments, guideline 8).
> Your point is almost exactly the same as the 3rd listed in the post: 3. "This is the way we have always done it so it must be correct!"
As I said in my other comments, I do not directly support the exact opposite of the author's stance. My disagreement is in the tone and rigidity of viewpoint. To quote myself:
I'm not calling this is good with the persistence of the original author. I tell that it's one of the realities that we have, and instead of burning it with torches, why not build better conventions around it with better attitude and language?
> These methods are being used for decades and well understood with all their advantages and disadvantages.
Just because something has been used for decades does not make it good - for example, avoidable mutable state. And it certainly does not make it well-understood - as a consultant the number of brain frying environment variable configurations I've had to deal with which no permies could tell me anything about defies belief.
> someone comes and tells that it's bad and considered harmful,
Yes, some things are bad and are actively harmful. Famously, unstructured programming using gotos. Would you like to go back to that? Believe me, you would not. But perhaps you are not a programmer?
> That thing wouldn't be a de-facto standard if it was too bad, right?
> Just because something has been used for decades does not make it good.
I'm not calling this is good with the persistence of the original author. I tell that it's one of the realities that we have, and instead of burning it with torches, why not build better conventions around it with better attitude and language?
Maybe we can try: "Instead of burying all config under environment variables, why not try doing it like this?", and slowly build something better, step by step. Nothing is inherently good or bad, but can be abused. So the abuse of environment variables as a shortcut needs to stop, one may say and I'd agree, and may also volunteer to help to build a better thing.
But, shunning it with anger and shouting "I'm the one who knows all right things!" sure creates backlash, like here.
All in all, I'm against the attitude, not the idea of improving a situation.
> Yes, some things are bad and are actively harmful.
It might be, but even your solution might not be right. Why the attitude?
> unstructured programming using gotos. Would you like to go back to that?
Did that on some older, limited hardware, and it was fun. It was not OK by today's standards, but I had to. I'll do it again if it's the only thing I can do to work on that particular hardware again.
> But perhaps you are not a programmer?
I just design algorithms and develop scientific applications which run on HPC clusters, nothing fancy.
> It's not a "de-facto standard", it's simply bad.
I didn't say it's good. I say it's a fact. I'm not disagreeing on its bad sides. I'm not OK with the attitude.
Agree. Also I don't see any proposal put forth.
Did I miss it? The gist I got was "env vars are bad and if we don't use something else (what?) we're hung up in the past".
Easy fix (if it's the shared, mutable state which bugs you):
* Create one class responsible for ingesting env vars at startup.
* Call it from main, and abort early with nice messages if it fails to read something.
* Now you have a nice (preferably immutable) class which guarantees the config is in a 'good state', and is self-documenting because it lists all the keys it uses to lookup env vars with.
This is essentially what I do in Rails apps. The only reference to an env var is in an initializer that sets an option in the global rails config structure.
The mutable state can be helpful. It is sometimes helpful to be able to change an app’s config without having to restart it. Ingesting the envs on startup into a class removes this ability.
It's ok for interactive applications, but if you are writing a CLI command (what is different from an interactive CLI application), a system library or a deamon, don't ever let the same application that uses a configuration also change it.
When your non-interactive programs do that and anything at all goes wrong, it's basically impossible to determine the source of the problem. Also, it is common that bugs that one could just avoid triggering by configuration now become unavoidable.
(But if you mean reload the config after getting a SIGHUP or something like that, yeah, this is ok, and the best way to do that is by restarting everything on your program, even if you keep the same process, so your read-once class won't be a problem.)
If The same pattern works well in python at the module level, if your application is setup as a package. A module config.py sets a bunch of python variables like
import os
ENV_VAR=os.environ.get('ENV_VAR', default_value)
then the rest of the application can grab configuration with
from .config import ENV_VAR
Since the assignment code executes on import, all config is read in when any piece of it is first used, consistency checks and logging can be written into the config.py module as normal python statements, config values can be cast to appropriate types (raising exceptions if they fail), etc.
In node.js there are quite a few packages that do exactly this - it's a great pattern and forces you to define which env values you will rely on in one place, rather than dripping them all over your codebase.
Oh this guy again. This guy created Meson and has a pattern of being 1) Quite toxic and 2) Entirely dogmatic when it comes to software design. He shows no interest in discussing design problems with Meson and asserts his viewpoints as truth and fact, resorting to snide comebacks instead of having thoughtful conversation.
Doesn't surprise me he wrote an article like this. Completely misguided and isn't rooted in reality.
Okay, so when you write that this blog author, who made a post arguing how environmental variables are global mutable state, is quite toxic and resorting to snide comebacks instead of having thoughtful conversation, then that is just you engaging in thoughtful conversation about the issue (which is envvars), and not you making a toxic ad-hominem attack at all, right?
It surprises me that people put more attention to the author than to the content. I have no idea who the author is, but his post doesn't seem to me toxic at all nor dogmatic.
On the other hand, your comment sounds a bit toxic, to be honest: "because the author is X it must be that all of his articles are X as well".
The best argument, perhaps only valid argument, is lack of an array type. Easy to work around. The rest seems misguided or ridiculous.
>There is no way to know which one of these is the correct form
What? Of course there is.
The author is also calling it mutable global state, and seems to reference an application becoming confused when ENV isn't set. This reads to me, perhaps incorrectly, that the author doesn't understand that envvars aren't and don't behave like shared global variables. That is, changing one won't affect running applications.
Readit News
> The answer is that you, the end user, can not now. Every program is free to do its own thing and most do. If you have ever spent ages wondering why the exact same commands work when run from one terminal but not the other, this is probably why.
If the same program is behaving differently between two systems - it can -only- be the environment that’s different.
> Instead of coming up with a syntax that is as good as possible for the given problem, instead the goal is to produce syntax that is easy to use
Oh to be a developer. The “best possible syntax” is a universe of possibilities - environment variables are, thankfully, limited to strings. If all programs had to be configured with a Turing complete config language - that would just be a programming language! Limitations can set you free.
Sorry for the harsh tone. Please, and I believe I speak for most sysadmins, please continue to use environment variables.
One of the projects I currently work on, the configuration system/model/table is a monster.. I wish it was just strings. It basically contains boolean flags, strings, numbers etc but the most insidious one is, it can contain groups of related configs - meaning people started dumping stuff into that should be a normal table (ex, ShippingType:, Road, Rail, Air), so we get no foreign key constraints for reference data. This caused them to implement soft-deletes for it, so now you have some config values that float around forever because they were referenced somewhere (aka pseudo-foreignkey). It's so utterly dumb I want delete the whole thing but everyone thinks it works great (non-tech people). I'm a developer, not a system-admin, but this is too much. Doesn't help that 5 different people has added 'features' to it over the years. The same can be accomplished with something waaaay simpler/cleaner. We also keep having config-related issues where people blame the system/servers/devops etc... every single time it is due to misconfig the project, so the system admins cannot do anything about it anyways (meaning we keep wasting their time, thinking the problem was with the servers). Let me pause here, need to take a blood pressure pill.
The same can be said for command line arguments. I usually prefer those instead of environment variables, because they must be explicitly specified instead of being implicitly passed by the parent process. I think of env vars as being more useful when repetitively calling commands interactively, to save some typing, or when you really do want processes to inherit config from parents (like PATH and its variants).
But overall I still agree with your sentiment.
Doesn't surprise me the creator of Meson of all people made the same dogmatic assertion.
What a circus this industry has become.
What kind of ways did you get burned by other things than env vars in a way in which env vars would not?
Yes, but the entire problem is that "the environment" is massive as it includes all hardware and software running on the device in question (and quite possible other devices as the network can easily be considered part of "the environment") which makes it difficult to track down differing behaviour.
"The environment" is not just environment variables. I've run into a spreadsheet bug where I got wrong results because of a CPU bug. Just because some global mutable state exists, that doesn't mean it's a good software design to have program behaviour depend on it.
As a developer, I've generally used configuration files for changing the operation of my software and environment variables for information my code needs to know about WHERE the code is running. For example, the same code doing the exact same thing on 10 machines would have the same configuration file (or command line parameters in simpler cases) but the environment variables may change from machine to machine.
The article's point stands if we're being lame and treating internal code details as external.
I was thinking about this while reading the post. What's the best syntax language or syntax for configuration, if environment variables are too simplistic and full-fledged programming languages are too powerful? That's why I'm interested in Dhall, which is a configuration language that's limited to compile-time only–i.e. it can't do anything at runtime.
Dead Comment
The author is correct that "this is the way we have always done it" isn't a good argument in and of itself to persist in a practice. However, they might be rewarded by a few minutes pondering a related idea: "if generations of people---many quite capable of modifying the system to use something else---persist in using something, it's possible they have a reason for doing so other than a deficit in competence or imagination."
Environment variables are scoped to the current process. This could be your shell, but it could also be a web server. This doesn't make them leak proof, but unlike global variables, environment variables have a scope.
Environment variables are also used more widely as an API. Many CLIs have a command that when self-executed can act as a persistence layer in your shell. This functionality would be impossible without environment variables.
You mentioned the other motherload, which is that environment variables are quite often, again, used to communicate in Makefiles. You can see the natural scoping if you start to kick off an ad-hoc shell process inside a Make target.
Deleted Comment
A gripe I have with environment variables is when they are used to modify a programs behavior deep inside it's belly and aren't treated like configuration input similar to program arguments.
Otherwise they are a universal way of configuring applications. Universal is good, universal is nice.
Obviously this is true in a sense. But for practical purposes, it depends on how the environment variables are set. For example, if I set them in my ~/.bash_profile, they're scoped to all bash processes that my current user runs. If I put them in /etc/profile, they're effectively global.
There is a fence somewhere you wish to get rid of. Nobody around knows why it was there in the first place. No one will let you tear it down until somebody figures out why it was put there in the first place.
The important take away, is that before changing something, one should understand the history of a thing, or one runs the risk of being a victim of the same problem the unknown thing was put in place to address.
Life is way more complicated than the umwelt of any one individual, so it is not safe to just change something without doing the footwork to understand what led it ro be in the first place.
Demonstrate that work has been done, and generally, no one will get in your way.
Yes, if your OS is some unversal JS machine, then JSON would be better, if it is Lisp machine, then you would use S-expressions, but on Unix machine, environment/args are way to go.
There are two realistic alternatives - config files and arguments. They have each their own niche, where environment is somehwere between them.
Arguments are better for one-shot setting, not for some setting used always. You can use 'alias' to define shortcuts that always add some argument, but that is definitely more cumbersome.
Config files are good for always/default setting, but are too rigid. Changing config files is equivalent of changing global variable in code, it has system/user-wide effect. While i can just change environment in this one shell and it will affect just commands executed from that shell. Also, config files are much harded to be manipulated from scripts, and use different syntax for each tool.
Perhaps the ideal tool would allow every option to be set/changed from config file, environment and argument.
1. I worry about programs dumping all their environment variables to log files - credentials are now on disk, ingested into log storage...
2. Environment variables are inherited by child processes by default. This is undoubtable useful. But it can also cause problems.
I wish the ghosts of unix past had forseen the need for a way to mark particular variables as, say, 'sensitive' and 'noexport', allowing them to opt out of the default behaviour.
It would have been so say to say "variables starting with _ are not inherited and should be censored when output", but we're about 40 years too late for that to catch on...
When it is meaningful (and this is supported today) is to set them just for specific programs/invocations, rather than exporting for a long-running interactive shell (and everything within it) willy-nilly.
More innovation around making that easier would be interesting, env vars that should be set specified by program, isolated from others, for example. So `foobar` would actually get executed like `FOO_SECRET=hunter2 foobar` without specifying it every time or having it exported in the shell, and in a generic way not specific to each program's config.
It's not really related but for some reason 'summon' is on my mind as a tool to mention. I haven't used it in anger yet, but it is interesting. It's not quite this though, or at least, it solves only the 'storage' part of the implementation of what I described, not the 'orchestration' or mapping of programs to vars/summon invocations.
That said, I agree with GP that environment variables are super useful and super simple. But I've also been burned more than a couple of times by setting something in the past and then having it caused unexpected bugs that are hard to trace down as they aren't in my working memory. They're a double-edged sword, to be sure.
2. If your child processes shouldn't inherit environment variables, set them properly. The "ghosts of Unix past" have "foreseen the need" for execve(2) and execveat(2), which don't pass anything by "default".
The default behavior is a non-exported variable. If you want child processes to see it, you must export it.
This is exactly what the most widely used golang configuration library does: https://github.com/spf13/viper
https://pypi.org/project/tconf/
1. default values: What will most users in most places find most useful/least infuriating?
2. configuration files (system-wide, then user): What will most users on this system want most of the time? What will this particular user want most of the time?
3. environment variables: How should this session (i.e., a potentially large series of related executions) be tailored?
4. command line options: What is most useful for this particular run?
I was also taught that:
- figuring out how to go from an option to the name of a corresponding environment variable to a line in a config file should be both straightforward and well documented; and
- sometimes you need a more complex configuration than is cleanly supportable through any other method than a file. In such a case, the location of that file can itself be passed through options and the environment.
1. Print the path of the system and user level configurations that the utility honours in the help text (-h/-?)
2. For an option that can be set interactively or via environment variable, specify the environment variable name in the help text itself to provide maximum choice to the user.
3. Provide a -viewconfig option that prints out the final resolved configuration state so that the user can see the actual configuration that is in effect. Combined with a -dryrun option, this can provide a lot of confidence to the user to try out things without breaking anything.
I thought it was common knowlegse that if really wanted to do configuration right on a given project, you do all 4 (with some library support) and you write your code to gracefully handle the right piece of configuration from the appropriate "override level" (again usually with the support of a good library).
See also: Domain Driven Design[0] which (if you ignore the consultant-fodder and jargon that comes with it) is probably one of the best written guides of how you should abstract systems, just like the gang of four book is a good introduction to structures in program/algorithm implementation you're likely to see in real life.
[0]: https://en.wikipedia.org/wiki/Domain-driven_design
> Envvars have some legitimate usages (such as enabling debug logging) but they should never, ever be used for configuring core functionality of programs.
As though logging weren't core functionality!
The actual thing that is bad is grabbing an environment variable in the middle of your program. You should grab all the configuration in one place and use it to configure local state that is transparently passed around. Furthermore, flags, env vars, and config files are all just maps from strings to configuration, so you should use some system that can transparently layer them on top of one another. All of my new CLIs use flags first and fall back to ENV vars if the flag wasn't set.
an immutable config object/class created on startup that reads files/env vars/whatever and has appropriate assertions to ensure good values were used and crashes the app for missing/bad values usually keeps things sane.
an app where each subcomponent has its own config that it gets in its own way usually leads to confusion and delay
One day, someone comes and tells that it's bad and considered harmful, and happily tells the only right way to do it. A flame war ensues then.
I'm all for moving things forward and evolution, but can't we take a milder stance and move forward in a more peaceful way? Attacking something so well established because of personal reasons feels so wrong from my PoV.
That thing wouldn't be a de-facto standard if it was too bad, right? I think we shouldn't play with the foundation that much.
I disagree. Very often, the 'easiest' option wins, not the one that necessarily is the 'best', especially in the long term. Environment variables for configurations are fast and easy, and work well on the happy path - but break down quite fast when dealing with more complex cases (eg. more complex types than strings). They have a treacherous way of seeming the simplest and most pragmatic solution at first, but becoming a an untyped, underdocumented hairball after some time in more complex software.
Furthermore, as long as environments have existed in UNIX and UNIX-like derivatives, their usage to configure the bulk of the behaviour of most services/programs are relatively new. The more old-school the service you deploy somewhere, the more likely it has a file-driven config. Indeed, sometimes it seems like 90% of the Docker code out there is converting environment variables into configuration files.
This is a consequence of Docker's choice of the "image" as an abstraction layer. It's not trivial to say "run this image but with this config file added" (yes you could bind mount one in, or create a new derived image, but those are both harder and come with more pitfalls).
In most common docker usage, there are exactly two ways to influence the operation of the program contained within the image: Environment variables, and command line arguments.
Thank you for your disagreement and discussion, honestly. Actually, I think using environment variables are a burden. Needs more documentation, more explicit warnings, a lot of handling, etc.
So, environment variables are not the easiest way out there. Especially when almost any programming language has nice config file libraries out of the box. Instead these variables are added as a convenience feature for some frequent scenarios where tool needs to adapt itself to the environment it needs to run in, just before starting or needs to be run repeatedly with small, transient changes to the config.
> Furthermore, as long as environments have existed in UNIX and UNIX-like derivatives, their usage to configure the bulk of the behaviour of a service/program are relatively new.
This is not what I see in my career. Bulk of the applications we installed and ran used some forms of environment variables for runtime configuration of the tool/application.
The reason for that the variable had a great deal of effect in the behavior of the program (which was generally scientific) and making multiple runs without modifying a file very effective. You need these runs to conduct research BTW, and you're on a cluster and jobs run long and whatnot.
TBH, most of these applications also had configuration files or "sensible defaults" and they either created their default files if there was none. And if there was a file, the environment variable was acting as an override.
So I had experimental software, fixed most of the parameters in the file and tried some other things by overriding some parameter(s) with an environment variable. Nothing was abused or misused.
> Indeed, sometimes it seems like 90% of the Docker code out there is converting environment variables into configuration files.
I've never seen it TBH, and if that's not documented well, it would be a big bag of fun for the users of that code.
That is a mischaracterization of the post. The author is making a clear technical point about how environmental variables are global mutable state. Labeling that as an "attack because of personal reasons" is just plain misleading.
> That thing wouldn't be a de-facto standard if it was too bad, right?
How much of the post did you read? Your point is almost exactly the same as the 3rd listed in the post:
> It's the same old trifecta of why things are bad and broken:
> 1. Envvars are easy to add
> 2. There are existing processes that only work via envvars
> 3. "This is the way we have always done it so it must be correct!"
I don't think so. First of all, as I detailed in [0] and [1], my central point of disagreement is the tone and attitude of the post, not the usage of environment variables itself.
There are a lot of scenarios where environment variables makes a lot of sense, and scenarios where using them is absolute madness as we discussed with q3k in [1].
> How much of the post did you read?
All of it. BTW, please remember asking this question is directly against guidelines [2] (sec: In comments, guideline 8).
> Your point is almost exactly the same as the 3rd listed in the post: 3. "This is the way we have always done it so it must be correct!"
As I said in my other comments, I do not directly support the exact opposite of the author's stance. My disagreement is in the tone and rigidity of viewpoint. To quote myself:
I'm not calling this is good with the persistence of the original author. I tell that it's one of the realities that we have, and instead of burning it with torches, why not build better conventions around it with better attitude and language?
Please see [0] and [1] for further clarification.
[0]: https://news.ycombinator.com/item?id=26660409
[1]: https://news.ycombinator.com/item?id=26660553
[2]: https://news.ycombinator.com/newsguidelines.html
Just because something has been used for decades does not make it good - for example, avoidable mutable state. And it certainly does not make it well-understood - as a consultant the number of brain frying environment variable configurations I've had to deal with which no permies could tell me anything about defies belief.
> someone comes and tells that it's bad and considered harmful,
Yes, some things are bad and are actively harmful. Famously, unstructured programming using gotos. Would you like to go back to that? Believe me, you would not. But perhaps you are not a programmer?
> That thing wouldn't be a de-facto standard if it was too bad, right?
It's not a "de-facto standard", it's simply bad.
I'm not calling this is good with the persistence of the original author. I tell that it's one of the realities that we have, and instead of burning it with torches, why not build better conventions around it with better attitude and language?
Maybe we can try: "Instead of burying all config under environment variables, why not try doing it like this?", and slowly build something better, step by step. Nothing is inherently good or bad, but can be abused. So the abuse of environment variables as a shortcut needs to stop, one may say and I'd agree, and may also volunteer to help to build a better thing.
But, shunning it with anger and shouting "I'm the one who knows all right things!" sure creates backlash, like here.
All in all, I'm against the attitude, not the idea of improving a situation.
> Yes, some things are bad and are actively harmful.
It might be, but even your solution might not be right. Why the attitude?
> unstructured programming using gotos. Would you like to go back to that?
Did that on some older, limited hardware, and it was fun. It was not OK by today's standards, but I had to. I'll do it again if it's the only thing I can do to work on that particular hardware again.
> But perhaps you are not a programmer?
I just design algorithms and develop scientific applications which run on HPC clusters, nothing fancy.
> It's not a "de-facto standard", it's simply bad.
I didn't say it's good. I say it's a fact. I'm not disagreeing on its bad sides. I'm not OK with the attitude.
* Create one class responsible for ingesting env vars at startup.
* Call it from main, and abort early with nice messages if it fails to read something.
* Now you have a nice (preferably immutable) class which guarantees the config is in a 'good state', and is self-documenting because it lists all the keys it uses to lookup env vars with.
It's ok for interactive applications, but if you are writing a CLI command (what is different from an interactive CLI application), a system library or a deamon, don't ever let the same application that uses a configuration also change it.
When your non-interactive programs do that and anything at all goes wrong, it's basically impossible to determine the source of the problem. Also, it is common that bugs that one could just avoid triggering by configuration now become unavoidable.
(But if you mean reload the config after getting a SIGHUP or something like that, yeah, this is ok, and the best way to do that is by restarting everything on your program, even if you keep the same process, so your read-once class won't be a problem.)
Immutable config via a config class that can exit early (prefereably startup) if there is a misconfiguration
Doesn't surprise me he wrote an article like this. Completely misguided and isn't rooted in reality.
On the other hand, your comment sounds a bit toxic, to be honest: "because the author is X it must be that all of his articles are X as well".
The best argument, perhaps only valid argument, is lack of an array type. Easy to work around. The rest seems misguided or ridiculous.
>There is no way to know which one of these is the correct form
What? Of course there is.
The author is also calling it mutable global state, and seems to reference an application becoming confused when ENV isn't set. This reads to me, perhaps incorrectly, that the author doesn't understand that envvars aren't and don't behave like shared global variables. That is, changing one won't affect running applications.