As silly as it sounds, when I was a new Unix SysAdmin, I read the entirety of "man 1 bash", which includes all bash builtins. I found that it improved by bash-foo 100x simply by knowing about so many of the utilities. I also took some cliff notes for things that seemed generally useful.
I did it for an hour or so a night for a week or so.
That being said, a few of my personal favorites to memorize:
* Keyboard shortcuts and how they are useful. A few example: CTRL-l (no need to ever use /usr/bin/clear), CTRL-k, CTRL-u, CTRL-e, CTRL-a, CTRL-w, CTRL-arrow left, CTRL-arrow right, CTRL-r (history reverse search with find as you type autocomplete)
The best way you can learn the shell is by using Linux as your primary desktop for at least a few months. You'll get very proficient very quickly by doing that.
One thing I have found that less people seem to know, is that the Unix metacharacters are expanded by the shell (bash etc.) not by individual commands. What this implies is that any command, whether built-in or written by you (in C, bash, Python or any other language), has metacharacter support automatically. That is, things like the file globbing/wildcard characters like *, ?, and [ ranges ].
This was not originally true on DOS (as a counterexample) and not sure whether it is true on Windows today (haven't checked), though I did notice that more commands seem to support wildcards in Windows nowadays.
Also, for some years now, Windows too has had redirections like:
command >file.txt 2>&1
(redirect stderr (2) to the same destination that stdout (1) is pointing to, i.e. file.txt), which Unix had from the start.
Yes, the man page is very good. The online gnu documentation is good, too. Some of the information will not be categorized as bash.[1] Also check out sed and awk.
Most of the responses here so far that do not include some sort of a guide are not the responses you're looking for (imho).
Mind your pipes and quotes. Guard your variables with braces. Do not export everything, test for and (try to) handle return codes and conditions and keep it simple (emphasis simple) but most of all just write it.
BASH (or Bourne) is ubiquitous when dealing with systems (vs programs). You don't need to be on the fashionable lang of the day by any measure. BASH, for most cases, will always be there, always ready and, in most cases, is the default human interface for deployed systems. As scripting languages go you don't need "better", you need dependability, zero dependencies with no requirement for modules or any other whizbangwoohoo plug-in. Language Fashionistas and personal preferences aside at least some level of fluency with BASH should be mandatory for anyone interfacing with a system.
You're going to get a lot of snark from people saying things like "don't", or "learn python instead".
This epitomizes "a little knowledge is a dangerous thing".
Bash has many cringeworthy aspects, but we have to deal with the world as it is, not the world as we would like it to be, and the reality is that bash is the default shell on 99.9% of unix boxes you encounter — even if you use an alt shell on your machine.
Coworkers machine? Bash. Default AWS AMI? Bash. init script to bootstrap $DAEMON? Bash. ssh to a server at your workplace? Bash. Random O'Reilly Linux tutorial? Assumes bash.
My advice?
Take some time.
Sit down.
and read "man bash"
cover-to-cover.
at least once.
A lot of the illogical things in bash make a lot more sense once you understand its parsing/expansion rules. And once you understand what is in the language vs an external terminal program in your PATH.
Since that sounds unappealing (and I scoffed at that very advice for many years), I've also found the wooledge bash guide to be very helpful.
Bash is unbeatable as a functional concept for chaining command-line tools together. Once you start getting functions or even a lot of if/while constructs, it's usually time to switch to Python/Perl.
I've been using linux since 1992 and if there's one thing I can't stress enough is to use full directories and not anything abbreviated. After 25 years, I still find myself slipping up, overlooking some minute detail, causing data loss.
I have written a simple tool called mann (https://github.com/soheilpro/mann) to help me remember little things that I learn when working in Bash/Zsh.
Basically, every time I learn something useful about a command, I add it to its mann page and then whenever I need it in the future, I simply run 'mann <command>' to find it.
Here's the current output of my 'mann sed', for example:
# Add char to beginning of each line
sed 's/^/#/'
# Replace with newline
sed 's/<oldvalue>/\'$'\n''/g'
# Replace newline
sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/<newvalue>/g'
# Plus sign
sed -E 's/foo+/bar'
# Digit
sed -E 's/[[:digit:]]/bar'
# Inplace
sed -i'.bak' -e <pattern> <file>
This looks great. I've always kept those types of little snippets in the README of my dotfiles repo - and always keep a printed out copy on my desk. But this seems way more practical.
If you're not already familiar with it, I would suggest learning about the basic Unix process model -- fork, execve, wait, open, pipe, dup2 and friends.
Bash is essentially a DSL for these. A lot of the weirdness you see in the language is due to these abstractions leaking through. For example:
* Quoting is building execve's argv parameter. It's hard to quote correctly if you don't know what exactly you're working towards.
* Redirections are opening and copying file descriptors. It explains their scope, order and nesting behavior.
* Variables are modifying and passing the environment, and their weird scope is due to forks imposed by the process model.
Once you know how you can do whatever you want in C through the basic syscalls, Bash is an extremely efficient and far less surprising shortcut to do it.
can you expand on this? does this elucidate, e.g., variable substitution and the difference between single- and double-quotes? or does it just help demonstrate when you need quotes for an argument that may contain whitespace?
The behavior of quotes can and should be described in terms of how they affect the words they expand to (i.e. the argv you build), so yes, it clarifies all this.
For example, all the various wrong ways of quoting var="My File.txt" or otherwise incorrectly using such a name will result in variations on a wrong argument list:
Besides the obvious answers of just reading the manual, looking up howtos, and stackoverflow I can recommend some habits that might increase your uptake of bash.
1. If you are not running a unix as your default OS switch to one (ie Linux or Mac).
2. Create a bin (~/bin) directory in your home directory of all your shells scripts and source control it. Any script you ever write put in that directory. Even if its not bash (ie python, perl). I find that it is critical to look at how you did things previously to help you learn as well as it saves time.
3. Any command that is complicated one liner that you create or see on the internet... create script and put in the bin directory mentioned above.
4. Optimize your personal bin directory and review frequently.
5. If you run Linux build your system from scratch (ie read Linux from scratch).
6. Bonus to the above: Automate the creation of your personal system through Packer and Bash!
7. Find where things are not automated.
8. Bash is more than just the shell. A knowledge of GNU coreutils as well as tmux/screen is worthwhile and highly recommended.
9. Learn the readline shortcuts. Particularly "ctrl-r".
Bash as a scripting language is actually pretty amazing. It gives you everything you need to perform some quick-and-dirty tasks with minimal overhead. If you need only work on sequential data, files and processes, it's a perfect match.
It's not a full-fledged programming language by any stretch of the imagination (lacking structures more complex than associative arrays), but it's damn good for scripts of all sorts.
As an example, I've reimplemented a subset of Ansible (a command able to send "modules" on multiple machines via SSH and capturing+caching their output for subsequent queries) in ~150 lines of Bash. Considering that the size of Ansible, written in the more proper Python, is ~15000 LOC, I'd say Python is the much lesser scripting language.
Edit: to answer the OP's question, the documentation I've found most helpful to learn Bash is the one present on the Linux Documentation Project, with the page for arrays deserving special mention : http://tldp.org/LDP/abs/html/arrays.html. I spent a lot of time reading the manual before stumbling upon that documentation, and none of it really clicked until I had a few examples before my eyes.
Typically a program like Ansible will have the majority of its use cases implemented in a minority of its code, while the rest of the code will be there to support special cases, edge cases, rare use cases, etc. So that contributes to the disparity in size.
Ansible has to take into account numerous edge cases, operating systems, backwards compatibility, etc. etc. Of course it's much, much bigger than your 150 lines bash script.
OTOH, if you can structure your problem as a composition of pipelines, it can be quite a bit faster in bash than in a "proper" language. You get to choose optimized tools, and they run concurrently.
Writing efficient bash code forces you to think about your problem differently. It's a very similar process to thinking functionally; e.g. you don't want to deal with lines of a file one at a time in a loop, you want to do filters and maps in languages like grep, sed and awk to deal with data in a streaming fashion with a minimum of forked processes.
Is there a "proper scripting language" somewhere that supports the same first-class access to Unix programs and the same piping | syntax that shells in general do?
Indeed. While you can get the rules regarding file name escaping right, the language does nothing to help you get it right every time, so unless you're careful someone will leave a file with a space (or, God help you, a newline) in the name in exactly the wrong place and blow it up.
Also, even if you manage to become better in Bash, you are bound to lose your skills at some point when you have been programming in other languages for a while.
I always have to look up how to do even basic things in Bash. I just don't use it often enough for these things to "stick".
How about OP does anyway? Bash as a scripting language is fucking great!
Since when is simplicity an argument against writing programs? Whether scripts or frameworks? "Hard to read" is not neccessarily an inherent trait[1] of the language, and more likely wrong on some PEBKAC level.
I have a customised environment at near 10k lines of bash in 5 projects, all of it in the correct tool for the job, aka a proper scripting language, so I can suggest another use for your thumb :-)
If you have really limited resources, then bash or one of the other shells are the only tools at hand. Embedded devices might not give you enough disk space to get perl, ruby or python.
Readit News
I did it for an hour or so a night for a week or so.
That being said, a few of my personal favorites to memorize:
* Parameter expansion: https://www.gnu.org/software/bash/manual/html_node/Shell-Par...
* All of test(1) as you can use them in any if statement (/usr/bin/[ is a real command!): https://linux.die.net/man/1/test
* Knowing most of the bash internal variables: http://tldp.org/LDP/abs/html/internalvariables.html
* Keyboard shortcuts and how they are useful. A few example: CTRL-l (no need to ever use /usr/bin/clear), CTRL-k, CTRL-u, CTRL-e, CTRL-a, CTRL-w, CTRL-arrow left, CTRL-arrow right, CTRL-r (history reverse search with find as you type autocomplete)
The best way you can learn the shell is by using Linux as your primary desktop for at least a few months. You'll get very proficient very quickly by doing that.
One thing I have found that less people seem to know, is that the Unix metacharacters are expanded by the shell (bash etc.) not by individual commands. What this implies is that any command, whether built-in or written by you (in C, bash, Python or any other language), has metacharacter support automatically. That is, things like the file globbing/wildcard characters like *, ?, and [ ranges ].
This was not originally true on DOS (as a counterexample) and not sure whether it is true on Windows today (haven't checked), though I did notice that more commands seem to support wildcards in Windows nowadays.
Also, for some years now, Windows too has had redirections like:
command >file.txt 2>&1
(redirect stderr (2) to the same destination that stdout (1) is pointing to, i.e. file.txt), which Unix had from the start.
[1] http://www.gnu.org/software/coreutils/manual/html_node/index...
Yes, and not only in an if statement. You can also use test or [ command in commands of the form:
test condition && commmand2
or
test condition || command2
which will only work if the condition is true or false, respectively (IIRC, need to check this).
Also one should take a look at rlwrap after becoming comfortable with the keyboard shortcuts.
Mind your pipes and quotes. Guard your variables with braces. Do not export everything, test for and (try to) handle return codes and conditions and keep it simple (emphasis simple) but most of all just write it.
BASH (or Bourne) is ubiquitous when dealing with systems (vs programs). You don't need to be on the fashionable lang of the day by any measure. BASH, for most cases, will always be there, always ready and, in most cases, is the default human interface for deployed systems. As scripting languages go you don't need "better", you need dependability, zero dependencies with no requirement for modules or any other whizbangwoohoo plug-in. Language Fashionistas and personal preferences aside at least some level of fluency with BASH should be mandatory for anyone interfacing with a system.
You're going to get a lot of snark from people saying things like "don't", or "learn python instead".
This epitomizes "a little knowledge is a dangerous thing".
Bash has many cringeworthy aspects, but we have to deal with the world as it is, not the world as we would like it to be, and the reality is that bash is the default shell on 99.9% of unix boxes you encounter — even if you use an alt shell on your machine.
Coworkers machine? Bash. Default AWS AMI? Bash. init script to bootstrap $DAEMON? Bash. ssh to a server at your workplace? Bash. Random O'Reilly Linux tutorial? Assumes bash.
My advice?
Take some time.
Sit down.
and read "man bash"
cover-to-cover.
at least once.
A lot of the illogical things in bash make a lot more sense once you understand its parsing/expansion rules. And once you understand what is in the language vs an external terminal program in your PATH.
Since that sounds unappealing (and I scoffed at that very advice for many years), I've also found the wooledge bash guide to be very helpful.
Don't do this. $PATH exists for a reason.
Basically, every time I learn something useful about a command, I add it to its mann page and then whenever I need it in the future, I simply run 'mann <command>' to find it.
Here's the current output of my 'mann sed', for example:
https://github.com/ben174/dotfiles/blob/master/README.md
Bash is essentially a DSL for these. A lot of the weirdness you see in the language is due to these abstractions leaking through. For example:
* Quoting is building execve's argv parameter. It's hard to quote correctly if you don't know what exactly you're working towards.
* Redirections are opening and copying file descriptors. It explains their scope, order and nesting behavior.
* Variables are modifying and passing the environment, and their weird scope is due to forks imposed by the process model.
Once you know how you can do whatever you want in C through the basic syscalls, Bash is an extremely efficient and far less surprising shortcut to do it.
can you expand on this? does this elucidate, e.g., variable substitution and the difference between single- and double-quotes? or does it just help demonstrate when you need quotes for an argument that may contain whitespace?
For example, all the various wrong ways of quoting var="My File.txt" or otherwise incorrectly using such a name will result in variations on a wrong argument list:
Meanwhile, all the correct ones result in the same, correct argv: If you don't know which argument list you're aiming for, you basically have to go by guesswork and superstitions.Do you recommend a book or something like that? And preferably for a beginner?
Love - Linux System Programming
The Unix-Haters Handbook
(Unfortunately, the best resource ever for this kind of stuff, from which I learned, does not have an English translation.)
[1]: https://github.com/koalaman/shellcheck
It will highlight common mistakes, and their wiki explains each one detail and how you should use an alternate, better implementation.
1. If you are not running a unix as your default OS switch to one (ie Linux or Mac).
2. Create a bin (~/bin) directory in your home directory of all your shells scripts and source control it. Any script you ever write put in that directory. Even if its not bash (ie python, perl). I find that it is critical to look at how you did things previously to help you learn as well as it saves time.
3. Any command that is complicated one liner that you create or see on the internet... create script and put in the bin directory mentioned above.
4. Optimize your personal bin directory and review frequently.
5. If you run Linux build your system from scratch (ie read Linux from scratch).
6. Bonus to the above: Automate the creation of your personal system through Packer and Bash!
7. Find where things are not automated.
8. Bash is more than just the shell. A knowledge of GNU coreutils as well as tmux/screen is worthwhile and highly recommended.
9. Learn the readline shortcuts. Particularly "ctrl-r".
I've not been able to find anything good on setting up a dev machine image with packer.
Anything that is not simple in bash gets hard to read and debug and probably is wrong on some subtle levels.
I have a rule of thumb that any shell script that grows beyond a screenful of lines gets redone in a proper scripting language.
It's not a full-fledged programming language by any stretch of the imagination (lacking structures more complex than associative arrays), but it's damn good for scripts of all sorts.
As an example, I've reimplemented a subset of Ansible (a command able to send "modules" on multiple machines via SSH and capturing+caching their output for subsequent queries) in ~150 lines of Bash. Considering that the size of Ansible, written in the more proper Python, is ~15000 LOC, I'd say Python is the much lesser scripting language.
Edit: to answer the OP's question, the documentation I've found most helpful to learn Bash is the one present on the Linux Documentation Project, with the page for arrays deserving special mention : http://tldp.org/LDP/abs/html/arrays.html. I spent a lot of time reading the manual before stumbling upon that documentation, and none of it really clicked until I had a few examples before my eyes.
Writing efficient bash code forces you to think about your problem differently. It's a very similar process to thinking functionally; e.g. you don't want to deal with lines of a file one at a time in a loop, you want to do filters and maps in languages like grep, sed and awk to deal with data in a streaming fashion with a minimum of forked processes.
I would love something like that.
Also, even if you manage to become better in Bash, you are bound to lose your skills at some point when you have been programming in other languages for a while.
I always have to look up how to do even basic things in Bash. I just don't use it often enough for these things to "stick".
Since when is simplicity an argument against writing programs? Whether scripts or frameworks? "Hard to read" is not neccessarily an inherent trait[1] of the language, and more likely wrong on some PEBKAC level.
I have a customised environment at near 10k lines of bash in 5 projects, all of it in the correct tool for the job, aka a proper scripting language, so I can suggest another use for your thumb :-)
1: https://www.reddit.com/r/commandline/comments/2kq8oa/the_mos...
https://github.com/EtiennePerot/parcimonie.sh/blob/48044f913...