Advice at a higher level: don’t try too hard to handle error cases in your scripts.
The best shell scripts are a list of imperative instructions to get something done. If an error occurs, bail as early as possible and with a useful message. Don’t try any harder than that.
For example:
if ! complex_function
then
handle_error
fi
…doesn’t work as you’d expect because the “if” context changes the rules of set -e in complex_function. It’s much better to have complex_function crash your script with a helpful error message that the operator can ameliorate.
You may have a clever way of solving this problem, but the cleverer your solution gets, the further you diverge from the core language of shell scripting. You will suffer the lisp/Ruby problem of every script being in its own unique language that’s based on but not identical to the language with which your colleagues are familiar.
One shouldn't dismiss shell scripts completely – the command line is the primary interface to Unix and with more and more idempotent commands showing up every year ("ip route replace") it only gets easier to write simple, imperative lists of commands.
This was one of the annoyances that I aimed to solve with my own shell, murex. It has predictable if blocks but also try blocks.
I used to be an advocate for `set -e` etc but these days I've come to the conclusion that the POSIX standard just needs to be retired for simple shell scripts, even if it is just for personal use. And it doesn't have to be my shell language that replaces it, but it shouldn't be something aiming for Bash/sh compatibility. This is probably going to be an unpopular opinion though but I base it on years of Bash/sh use and then getting fed up with it enough to write me own shell + scripting language.
As for anything that needs to be used and maintained by a team, that's probably around the time you have to question if shell script is even the right solution and whether you need to break out into Python (though Python has it's own issues too so one has to make an informed decision based on your own business requirements).
For most uses, I consider all of these unwanted at best and potential disasters at worst. If you use them, make sure you understand exactly what each of these do. If you just follow a dictum you found on HN blindly and just copy-paste them, you're in for a world of hurt.
This may be controversial, but sometimes I feel the biggest mistake is using a shell script in the first place. For example, on servers where you have PHP installed anyway, a PHP CLI script is often an alternative. Say what you will about PHP, but PHP code is much more readable than shell scripts - especially if everyone working on the server is fluent in PHP anyway. Shell scripts do have their niche, but often you find that the scope of a script keeps growing, then soon it becomes cumbersome and makes you wish you would have chosen another alternative from the start...
Once I get a script to a point of defining functions or anything but the simplest if or for loop, I rewrite in a proper language (Perl, Python, even PHP) and bundle it as a deb onto our internal repo.
Too many times I didn’t move from bash until too late.
I still write bash, I’ve got one which runs up a 20 line ffmpeg command with a couple of variables, that’s not a problem. On the other hand I changed the complexity of a file analysis tool from grep/sort/uniq/sed to Perl a couple of weeks ago before it became too large.
Most linux distributions come with python installed. Anything more than invoking a binary and redirecting the output? Just write it in (pure) python I say!
Edit: by pure Python, I mean don’t require any `pip install`s
Python is terrible at juggling files, setting up pipelines and working with processes. The resulting code is more complicated than the equivalent bash, not to mention longer, and hidden gotchas abound (subprocess deadlocks anyone?). I'll take 50 lines of bash instead of 200+ lines of Python.
This is really huge. If a script is going to be used by anyone other than yourself, it needs to be readily accessible (e.g. understood) for the group that will be using it.
Shell script is a kind of DSL for the OS operations and PHP a kind of DSL for the web programming. Similarly, it does not make sense to advocate the usage of R for web programming but people do it anyway.
Shell script have their deficiencies but it's the best for their intended environment. Alternatively now there's Oil shell that looks promising for modern approach to shell programming [1]. Expect to see exponential usages of shell programming now that we have new popular OS related technology including containers, isolates, and functional package management like Nix and Guix.
The advantage I find with Python is that it's commonly part of the Linux systems I come across (e.g. Debian-based and RHEL-based Linux systems). I'd have to install the Node.js runtime. So despite preferring Node, I find myself writing more and more Python.
It felt nice when i discovered shellcheck. I made the mistake of trying to get my scripts 100% compliant... and was then astounded to find i broke several of them.
If you don't know what you are doing, and shellcheck knows more, it might be useful. But if you know what you are doing, shellcheck becomes annoying quickly.
Example: echo "$(command)" is marked as SC2005, useless echo. But command does not always print a newline, so "fixing" this would garble your output under some conditions.
> Example: echo "$(command)" is marked as SC2005, useless echo. But command does not always print a newline, so "fixing" this would garble your output under some conditions.
It should still be `printf '%s\n' "$(command)"` for portability unless the intention is for the output of your command to possible be interpreted as a flag for echo.
Oh, yeah. I've broken multiple historic scripts in obscure ways "fixing" them under shellcheck's direction.
It's a general lesson for linters: if it points out code where the results may be surprising, then be aware that the "surprising" result may be necessary for the code's function.
The minute I need anything other than 4-5 commands executed sequentially, I switch to python. It's not worth dealing with the hassle of string substitution, error handling, list handling, filtering, etc in bash.
Nice tips, although doesn't look comprehensive. It's certainly hard learning all the gotchas of shell scripting.
In Deployment from Scratch I teach minimum amount of Bash to get servers up and running. I avoid teaching anything I don't have to. For example, most people are fine with just "set -euo pipefail" and understanding simple functions and pipes.
If your script is small enough, you will be fine.
If your script is getting big, it might be a time to switch to smth else.
Comparing shamelessly to my own Next Generation Shell, puting aside the completely insane syntax matters that nobody today would do or accept (such as $x expanding to zero or more arguments).
cleaning up temp files
Use TmpFile. It is automatically removed when the script exits.
stopping automatically on error
As in many other modern languages, NGS has exceptions and prints stack trace when they are not handled. Nothing special is required from the programmer.
echoing errors
Use built in error("my message") function. Alternatively, use exit("my message") to print the error and exit, the exit code defaults to 1 but can provided. Tired of seeing bash scripts starting with definition of warn(), debug(), error(), etc functions. These are part of standard library in NGS.
You are welcome to check out the Next Generation Shell project.
A very common one I've hit once or twice is people assuming bash is the system shell (which is common with Linux, but far from ubiquitous as Debian for example uses dash) and using bashisms while leaving the hash-bang as #!/bin/sh.
I always go with #!/bin/bash as I do often use bashisms, and only #!/bin/sh when I know I've been careful (because at the start I know I'm writing something intended to be as portable as practical, or I've gone through with a fine-tooth comb to test compatibility at a later time).
Or its superior(?) friend "#!/usr/bin/env bash" which will select the bash that the user has on their PATH, which is especially considerate to Homebrew/Linuxbrew folks who have a modern version in $HOMEBREW_PREFIX/bin
I believe there are some esoteric systems that relocate "env" as "/bin/env" or such, but the good ole "90/10" rule applies here
I used to use `#!/usr/bin/env bash` for this reason (Linux / MacOS + Homebrew user) but have been bitten a few times by bash version differences when I write a script that is later called by a system utility like launchd (which doesn't inherit my PATH) that goes with /bin/bash instead.
Because the backwards compatibility tends to be good, if I'm writing a script that will be run non-interactively, I will usually write a fixed `#!/bin/bash` shebang just so I can be sure that it run with the expected bash version on Linux or Mac.
Readit News
The best shell scripts are a list of imperative instructions to get something done. If an error occurs, bail as early as possible and with a useful message. Don’t try any harder than that.
For example:
…doesn’t work as you’d expect because the “if” context changes the rules of set -e in complex_function. It’s much better to have complex_function crash your script with a helpful error message that the operator can ameliorate.You may have a clever way of solving this problem, but the cleverer your solution gets, the further you diverge from the core language of shell scripting. You will suffer the lisp/Ruby problem of every script being in its own unique language that’s based on but not identical to the language with which your colleagues are familiar.
One shouldn't dismiss shell scripts completely – the command line is the primary interface to Unix and with more and more idempotent commands showing up every year ("ip route replace") it only gets easier to write simple, imperative lists of commands.
I used to be an advocate for `set -e` etc but these days I've come to the conclusion that the POSIX standard just needs to be retired for simple shell scripts, even if it is just for personal use. And it doesn't have to be my shell language that replaces it, but it shouldn't be something aiming for Bash/sh compatibility. This is probably going to be an unpopular opinion though but I base it on years of Bash/sh use and then getting fed up with it enough to write me own shell + scripting language.
As for anything that needs to be used and maintained by a team, that's probably around the time you have to question if shell script is even the right solution and whether you need to break out into Python (though Python has it's own issues too so one has to make an informed decision based on your own business requirements).
My best shell advice is to put these commands at the beginning of every script:
Use them to have a saner life.Example for set -e: https://mywiki.wooledge.org/BashFAQ/105
In C we always write `if (func()) handle_error` for the standard 0=success convention, and reading shell IFs is mentally straining
Too many times I didn’t move from bash until too late.
I still write bash, I’ve got one which runs up a 20 line ffmpeg command with a couple of variables, that’s not a problem. On the other hand I changed the complexity of a file analysis tool from grep/sort/uniq/sed to Perl a couple of weeks ago before it became too large.
Most linux distributions come with python installed. Anything more than invoking a binary and redirecting the output? Just write it in (pure) python I say!
Edit: by pure Python, I mean don’t require any `pip install`s
It made sense: everyone could read R, not many could read bash/shell as most had learned R on a windows PC in grad school of statistics.
So while sometimes a bit clumsy and not very portable, these scripts could be read by anyone.
Couple decades ago even wrote some init scripts in Perl because they needed to be complicated.
Shell script have their deficiencies but it's the best for their intended environment. Alternatively now there's Oil shell that looks promising for modern approach to shell programming [1]. Expect to see exponential usages of shell programming now that we have new popular OS related technology including containers, isolates, and functional package management like Nix and Guix.
[1]https://www.oilshell.org/
Dead Comment
If you don't know what you are doing, and shellcheck knows more, it might be useful. But if you know what you are doing, shellcheck becomes annoying quickly.
Example: echo "$(command)" is marked as SC2005, useless echo. But command does not always print a newline, so "fixing" this would garble your output under some conditions.
It should still be `printf '%s\n' "$(command)"` for portability unless the intention is for the output of your command to possible be interpreted as a flag for echo.
It's a general lesson for linters: if it points out code where the results may be surprising, then be aware that the "surprising" result may be necessary for the code's function.
I've begun piping such untrustworthy commands to xargs -n1 to guarantee good behavior.
Deleted Comment
In Deployment from Scratch I teach minimum amount of Bash to get servers up and running. I avoid teaching anything I don't have to. For example, most people are fine with just "set -euo pipefail" and understanding simple functions and pipes.
If your script is small enough, you will be fine. If your script is getting big, it might be a time to switch to smth else.
cleaning up temp files
Use TmpFile. It is automatically removed when the script exits.
stopping automatically on error
As in many other modern languages, NGS has exceptions and prints stack trace when they are not handled. Nothing special is required from the programmer.
echoing errors
Use built in error("my message") function. Alternatively, use exit("my message") to print the error and exit, the exit code defaults to 1 but can provided. Tired of seeing bash scripts starting with definition of warn(), debug(), error(), etc functions. These are part of standard library in NGS.
You are welcome to check out the Next Generation Shell project.
https://github.com/ngs-lang/ngs
Deleted Comment
I always go with #!/bin/bash as I do often use bashisms, and only #!/bin/sh when I know I've been careful (because at the start I know I'm writing something intended to be as portable as practical, or I've gone through with a fine-tooth comb to test compatibility at a later time).
I believe there are some esoteric systems that relocate "env" as "/bin/env" or such, but the good ole "90/10" rule applies here
Because the backwards compatibility tends to be good, if I'm writing a script that will be run non-interactively, I will usually write a fixed `#!/bin/bash` shebang just so I can be sure that it run with the expected bash version on Linux or Mac.