Hey, original author here. Thanks for sharing this and making it rise to the front page! :) By the way, the title probably deserves a (2020) and it would be nice if "test" wasn't capitalized, because it actually refers to the command.
[[ is not really a builtin, it's fundamentally syntactical (but presumably uses a mostly-inaccessible builtin internally). Fun fact, `]]` is also a reserved word despite never being allowed in a context where reserved words matter.
In some non-bash shells, the `function` keyword is needed to declare certain types of function.
For make `$(shell)`, if you're building a lot of targets the performance difference can be measurable. Still, it loses in the nop case, so you should actually usually do `include` to trigger re-making.
GNU is completely right to ignore POSIX, since POSIX is not useful for solving most real problems.
A lot of the GNUisms described in https://jmmv.dev/2021/08/useless-use-of-gnu.html are very useful in interactive use. Search the current directory without an explcit . - useful. Append an option to a command you just typed - heck yeah, always annoyed when commands don't support that.
For scripts though, sticking to POSIX sh often makes sense, yeah. You should at least be aware if you use of Bash-isms.
This article complains that using the extended GNU features (--ignore-case, set -o pipefail etc) makes scripts less portable. Fair enough.
What it doesn't explain is why a Linux user should much care about portability. OpenBSD and FreeBSD are alive and well, but the number of users seems so small that they aren't a particular concern. Maybe you could argue that we "should" consider these OSes out of a sense of fairness, but where does that stop? Do I also need to consider something obscure like vxWorks?
BusyBox (Alpine) is more interesting, but the changes there are so significant that a port will almost always be needed anyway.
Are there other compelling reasons to care about the non-GNU ecosystem?
I find the obsession with shell-script portability in contexts where it doesn't matter to be bizarre, but this particular argument is amusing:
> the number of users seems so small that they aren't a particular concern. Maybe you could argue that we "should" consider these OSes out of a sense of fairness, but where does that stop?
This is the same argument that many companies and products have made over the years (and still do at times) for ignoring Linux. To have a Linux user use that same argument against OSes with even smaller userbases is kind of amusing to see.
What I'm trying to say is that open standards (and the portability that comes with them) is not something that just happens on its own. It takes active maintenance, and part of that maintenance is opting to adhere to the standard even when it would be more convenient to use extensions available in the most popular systems.
Will you personally suffer from liberally using Bashisms? Not in the first order. But if we encourage that sort of thinking as a rule, the standards become meaningless. I believe that would be a net negative change for the world, but there are many intelligent people who would disagree.
You will notice that the parent article mentions "dash."
The dash shell is small and fast, but it does not allow any bash/korn language extensions beyond what was recorded in the POSIX.2 standard in the early 1990s.
Linux users should care because the Debian/Ubuntu family use dash as the system shell, so this problem is very real as many have learned.
`-o pipefail` isn’t a GNU extension or even a difference in user space programs, but specifically a bash option. The portability is between shells, not operating systems. It will work on bash, regardless of the underlying OS, but may or may not work on csh, ksh, fish, etc.
Mac's command-line tools are the BSD versions. I always get confused about the command-line arguments for `date` because they differ so much betwen platforms.
if [ a = b ] || grep -q ^hello$ /usr/share/dict/words; then
echo "test failed and grep succeeded"
fi
> “You pick whether to be amused or horrified. I don’t know how exactly my coworker reacted when I hinted at this during a recent code review I did for them.”
Tangential, but your blog software seems to have mangled the headings; it's serving eg:
make $(shell …) expansion
instead of
make $(shell ...) expansion
(note that (as is correctly written in the body) that's three pediods, not one elipsis, so mldr wouldn't be correct even on it's own, meaning there's two probably-unrelated bugs affecting it).
Hmm… I’ll have to check later but right now I’m seeing the right thing in iOS. The blog is built by Hugo, so it’s all static files. But maybe something changed with the latest update. Thanks.
> [ a = b ] && echo "Oops!" || echo "Expected; phew!"
Not to be taken as a general rule though. I might be mistaken but I think that bash would parse the line as:
([ a = b ] && echo "Oops!") || echo "Expected; phew!"
so if the command sequence after `&&` fails, then the code sequence after `||` is executed anyway:
illo@joe:~ $ [ "a" == "a" ] && >/dev/full echo "strings match" || echo "strings don't match"
-bash: echo: write error: No space left on device
strings don't match
illo@joe:~ $
This is different from the semantics of the `if` block:
illo@joe:~ $ if [ "a" == "a" ]; then >/dev/full echo "strings match"; else echo "strings don't match"; fi
-bash: echo: write error: No space left on device
illo@joe:~ $
No it won't. set -e is implicit disabled for the first command with && and ||. Same for a command after if/while/until and after !.
It should only matter if you implicit return immediately after.
I stopped using [ a few years ago because ‘test’ reinforces the idea that this is just a command like any other, not syntax. Also, “man test” is much more pleasant that sifting through “man bash”.
I think GP's point was that `[` feels like syntax, but - importantly - isn't.
Yes, `[` is a command, and has a man page and everything, but in a script it doesn't look like a command. It looks like something "special" is going on.
Whereas using `test` emphasises the point that you're just running another program. That all `if` does is check the output status of whatever program is being run, whether that's `test`/`[`, or `grep`, or anything else at all.
(Personally, I don't think that emphasis is necessary. But I've been using shell scripts long enough that I understand these nuances fairly well without having to think about them much any more. So I think that GP's point is a reasonable perspective to have, and worth considering.)
I'm with you, the [ (and [[ bashism) introduces lots of confusions about what is really happening, I could never manage any real confidence.
That said, [[ being guaranteed to be built-in certainly had its purpose at ages where shell script performance had any kind of relevance, and that was no so long ago.
[[ has more to do with trying to build an intuitive shell scripting environment than performance. [[ makes conditionals behave much more like you'd expect from other programming languages. I think it's a great idea, but then again, if I don't have to care for POSIX portability, I'd rather use something that's not a shell language for scripting.
I’ll probably just use test after reading this, I write bash scripts infrequently enough that if statements always get me (whitespace). Now that i see the why it’s super obvious, and using test helps communicate that it’s just args
The biggest footgun in `[` and `test` is the single argument behavior. For example, you might attempt to check if a variable is nonempty like so:
[ -n $FOO ]
but if FOO is unset, it expands to nothing (as opposed to the empty string), so this is equvalent to:
[ -n ]
and POSIX requires that the one-argument form of `[` succeed if that argument (here, "-n") is non-empty. So this will falsely report that $FOO is non-empty.
I think your last sentence needs to go first. Quote your variables! There's no actual footgun in the specification of the test builtin -- the footgun is shell itself. The behaviour that you mention makes sense because
[ "$FOO" ]
is always the non-empty check regardless what it contains (could be "-n").
chubot has written an interesting document¹ exploring more of the nuance with test/[/[[, and many of the other entries in that blog have intriguing explanations of the oddities of our shells(a random example²).
"bash only" typically refers to "bashisms", that is, bash features not present in the plain Bourne shell (or Bourne-compatible interpreters such as dash). The fact that other shells (such as zsh) may include those features ... is beside the point of writing universally compatible shell scripts.
Confirming my facts for this comment, #TIL that "dash" is the "Debian Alquist Shell", that is, Debian's "ash" shell:
While zsh, bash, mksh, ksh93, probably others have it, sure. But many don't -- and not totally irrelevant ones either. Debian's default, dash, for example, does not support `[[`.
It's usually pretty trivial to avoid them, especially if you're willing to call other mandated commands like awk, etc. But often, with a bit of creative thinking, most non-standard features can be replicated with some combination of `set`, separate functions and/or subshells.
Shell scripts, in general, have dozens of footguns, are pretty much impossible to statically analyze, difficult to make truly robust, and many of the shells themselves -- e.g., bash -- have huge, borderline inauditable codebases.
I can think of a dozen reasons not to write shell scripts. Yet still, there is incredible value in the fact that some form of POSIX-compliant/compliant-enough shell can usually be found on most systems.
All of that value goes out the window, though, the moment you start relying on non-standard features.
That is, test and [ are specified by POSIX and usually are physical binaries (but might also be masked by shell builtins). Whereas [[ is not specified by POSIX and usually only exists as a shell builtin.
Not installed by default on BSDs. Sometimes not installed in minimal environments, e.g. where BusyBox is all you have available. Maybe not installed by default on Unixes? Not sure
Writing Bourne-compliant scripts ensures maximum portability.
As many here have noted, bash isn't universally available, with another possible issue being OpenWRT devices. Stock/base images tend to use a Bourne-compatible shell, not full Bash. Though the latter's installable through opkg, for sufficiently small devices (typical of consumer kit), you simply won't have the space to install them.
There's also the slight PITA that Apple's OSX ships with a very old, pre-GPLv2 Bash, out of licensing concerns. (Apple is phenomenally averse to GPL-based code, much as some *BSDs are, such as OpenBSD.)
And if you're dealing with legacy systems (which tend to be extraordinarily and stubbornly persistently legacy), you'll often find that either bash isn't present or is quite dated.
I freely confess that I tend to write fairly recent-feature bash scripts myself by default, and appreciate many of the newer features. But if and when I am writing portable code, I'll go back to Bourne-compatibility.
But when writing system level code, an appreciation for standards and the very long tail of legacy standards and the limitations they impose is in fact a large component of professional maturity.
I've worked on several modern projects that needed to be able to run on a wide variety of Unix platforms, several of which didn't have bash. Writing for the common shell denominator was important, not archaic.
it’s contextual as most things. need to actually use and work on the machine? use whatever shell you want to make your life easier.
need a script to run on many different systems and/or need to write a script to be managed automatically by a service account? probably you want a shell with syntax that is guaranteed to be the same on all your systems.
I really didn't understand why the last if statement is confusing. Is it because when starting out with shell scripting one would usually assume that the [ is a part of the bash scripting language not just another program? If it's then I think I get it now. Otherwise please mention why it's surprising. Also, @author thanks for a nice article. was a good reed.
Readit News
Here is something related from 2021 that also touches on bash's [[ operator and that I think you might enjoy in this context: https://jmmv.dev/2021/08/useless-use-of-gnu.html
In some non-bash shells, the `function` keyword is needed to declare certain types of function.
For make `$(shell)`, if you're building a lot of targets the performance difference can be measurable. Still, it loses in the nop case, so you should actually usually do `include` to trigger re-making.
GNU is completely right to ignore POSIX, since POSIX is not useful for solving most real problems.
POSIX, being a standard that can be tested, is at least a specification or agreement that can be met by multiple products to enable them to interact
In shells that aren't POSIX-compliant [0], maybe. In which case: yes, there are many wildly different scripting languages with REPLs.
[0] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
For scripts though, sticking to POSIX sh often makes sense, yeah. You should at least be aware if you use of Bash-isms.
What it doesn't explain is why a Linux user should much care about portability. OpenBSD and FreeBSD are alive and well, but the number of users seems so small that they aren't a particular concern. Maybe you could argue that we "should" consider these OSes out of a sense of fairness, but where does that stop? Do I also need to consider something obscure like vxWorks?
BusyBox (Alpine) is more interesting, but the changes there are so significant that a port will almost always be needed anyway.
Are there other compelling reasons to care about the non-GNU ecosystem?
> the number of users seems so small that they aren't a particular concern. Maybe you could argue that we "should" consider these OSes out of a sense of fairness, but where does that stop?
This is the same argument that many companies and products have made over the years (and still do at times) for ignoring Linux. To have a Linux user use that same argument against OSes with even smaller userbases is kind of amusing to see.
----
What I'm trying to say is that open standards (and the portability that comes with them) is not something that just happens on its own. It takes active maintenance, and part of that maintenance is opting to adhere to the standard even when it would be more convenient to use extensions available in the most popular systems.
Will you personally suffer from liberally using Bashisms? Not in the first order. But if we encourage that sort of thinking as a rule, the standards become meaningless. I believe that would be a net negative change for the world, but there are many intelligent people who would disagree.
The dash shell is small and fast, but it does not allow any bash/korn language extensions beyond what was recorded in the POSIX.2 standard in the early 1990s.
Linux users should care because the Debian/Ubuntu family use dash as the system shell, so this problem is very real as many have learned.
Isn't that normal everyday shell use?
(And I might quote '^hello$' personally just on principle for having special chars, especially dollar, and to help syntax highlighters.)
Tangential, but your blog software seems to have mangled the headings; it's serving eg:
instead of (note that (as is correctly written in the body) that's three pediods, not one elipsis, so mldr wouldn't be correct even on it's own, meaning there's two probably-unrelated bugs affecting it).> Special cases aren't special enough to break the rules.
> Although practicality beats purity.
:)
Deleted Comment
----
And the fact that the if block tests a regular command means we can also do things like
----Something I have not yet bothered to figure out is whether I should write
or use the built in logical and of test: As long as performance is not a concern, I can see roughly equal reasons in favour of either.Not to be taken as a general rule though. I might be mistaken but I think that bash would parse the line as:
so if the command sequence after `&&` fails, then the code sequence after `||` is executed anyway: This is different from the semantics of the `if` block:According to POSIX, the -a and -o binary primaries and the '(' and ')' operators have been marked obsolescent. See https://pubs.opengroup.org/onlinepubs/9699919799/utilities/t... under "Application Usage".
Deleted Comment
There’s no need for test(1) / [(1) or conditional expressions ([[…]]) if you’re doing arithmetic:
if ((1+1 == 2)); then …; fi
Bash has "help test" for a quick cheatsheet.
The [ command is very old; it was already present in Version 7 Unix in 1979.
Yes, `[` is a command, and has a man page and everything, but in a script it doesn't look like a command. It looks like something "special" is going on.
Whereas using `test` emphasises the point that you're just running another program. That all `if` does is check the output status of whatever program is being run, whether that's `test`/`[`, or `grep`, or anything else at all.
(Personally, I don't think that emphasis is necessary. But I've been using shell scripts long enough that I understand these nuances fairly well without having to think about them much any more. So I think that GP's point is a reasonable perspective to have, and worth considering.)
That said, [[ being guaranteed to be built-in certainly had its purpose at ages where shell script performance had any kind of relevance, and that was no so long ago.
Remember to quote your variables!
Or use [ -n ${FOO-} ] which will replace the unset variable with an empty string.
https://news.ycombinator.com/item?id=26776956
Or whatever the magic string is. Enable all the errors at the start of the script
Deleted Comment
¹ https://www.oilshell.org/blog/2017/08/31.html
² https://www.oilshell.org/blog/2016/11/18.html
But at least it explains why you need spaces on both sides of the brackets.
https://zsh.sourceforge.io/Doc/Release/Conditional-Expressio...
Confirming my facts for this comment, #TIL that "dash" is the "Debian Alquist Shell", that is, Debian's "ash" shell:
<https://en.wikibooks.org/wiki/Guide_to_Unix/Explanations/Cho...>
zsh and ksh have it; in fact I'm pretty sure it originated with ksh in 1988 or earlier.
While zsh, bash, mksh, ksh93, probably others have it, sure. But many don't -- and not totally irrelevant ones either. Debian's default, dash, for example, does not support `[[`.
IMO, unless you're writing something like shell-specific dotfiles, avoid non-POSIX features.
It's usually pretty trivial to avoid them, especially if you're willing to call other mandated commands like awk, etc. But often, with a bit of creative thinking, most non-standard features can be replicated with some combination of `set`, separate functions and/or subshells.
Shell scripts, in general, have dozens of footguns, are pretty much impossible to statically analyze, difficult to make truly robust, and many of the shells themselves -- e.g., bash -- have huge, borderline inauditable codebases.
I can think of a dozen reasons not to write shell scripts. Yet still, there is incredible value in the fact that some form of POSIX-compliant/compliant-enough shell can usually be found on most systems.
All of that value goes out the window, though, the moment you start relying on non-standard features.
It feels like some completely archaic concern to target the minimal common shell denominator.
I write my shell scripts in sh, because it comes with every unix-like os by default.
I know that bash has more features, but I've never really missed them. I switch from sh to perl for more complicated tasks.
To each their own, right?
As many here have noted, bash isn't universally available, with another possible issue being OpenWRT devices. Stock/base images tend to use a Bourne-compatible shell, not full Bash. Though the latter's installable through opkg, for sufficiently small devices (typical of consumer kit), you simply won't have the space to install them.
There's also the slight PITA that Apple's OSX ships with a very old, pre-GPLv2 Bash, out of licensing concerns. (Apple is phenomenally averse to GPL-based code, much as some *BSDs are, such as OpenBSD.)
And if you're dealing with legacy systems (which tend to be extraordinarily and stubbornly persistently legacy), you'll often find that either bash isn't present or is quite dated.
I freely confess that I tend to write fairly recent-feature bash scripts myself by default, and appreciate many of the newer features. But if and when I am writing portable code, I'll go back to Bourne-compatibility.
But when writing system level code, an appreciation for standards and the very long tail of legacy standards and the limitations they impose is in fact a large component of professional maturity.
For the rest of us, we have learned the hard way, sometime repeatedly, portability and adherence to published standards matters.
need a script to run on many different systems and/or need to write a script to be managed automatically by a service account? probably you want a shell with syntax that is guaranteed to be the same on all your systems.