This is going to confuse the frankfurter out of a newbie. First it presents the idea that Windows has a shell, and even multiple shell programs. Screenshots 'n' everything.
But then, nope:
> Windows is not anything like Linux under the hood. So to get a shell working, we have three options:
> Use a tool which provides common Linux tools which have been written to work with Windows
> Use a "virtual machine" running Linux
> Use the Windows Subsystem for Linux
What "common Linux tools which have been written to work with Windows" is referring to is installing Cygwin.
This a fairly hamfisted presentation of a subject matter that could be handled in a way that is simultaneously easier to grasp and factually correct.
First of all, your goal is to teach about a specific command interpreter language, you really want to clarify how there are multiple ones which have different syntax, and brush that whole topic aside as fast as possible.
> You can see what the version is of your shell by running:
> bash --version
Yikes. If we know we are running Bash, the way to know what version it is is:
echo $BASH_VERSION
and not to run another instance of Bash.
Can this book teach something simple, like comments?
> The shell ignores any text which follows a # hash symbol.
Wrong:
$ echo 'all of # this is seen'
all of # this is seen
"this is seen" looks like an example of "any text" to me.
A hash comment marker is ignored if it is not quoted and escaped so as to be part of a word. The hash is also part of certain bits of syntax:
$# (dollar hash) number of positional parameters
${variable#pat} expand the content of variable, chopping off
a leading portion which matches the pat pattern.
Example:
$ echo $TERM
xterm-256color
$ echo ${TERM#xte}
rm-256color
The classic Windows shell has many flaws in its design, appears to have come from OS/2, and it's amazing that it works at all for how it has been used:
Hello, this is dwmkerr the author - your comments are all spot on - the earlier in the book you go the more work it needs, I've learnt a lot along the way TBH probably the entire first part needs a rewrite, that's actually going on at the moment as I'm working with a publisher.
I've linked to this post/comments on the repo (issue #210), they should all get addressed but it might be a while before the online version gets sorted, I'm closing out the final chapters online, once I've done this I'll likely go back to the beginning and rewrite/edit/put in a more consistent style (more code snippets, fewer screenshots etc).
I agree with all your comments from a technical point of view.
I disagree with all of them from an educational point of view.
Having said that, it helps to clarify whether the audience is technical or education oriented. The title implies a technical book, so I'm with you on the comments.
For the education oriented audience you need to explain everything, in some good order of revelation. Ideally not all at once. What is it that we are calling "shell"? Why is it called that? Where does it come from? Why are there different kinds of these things? Why do we go out of our way to install some other one intended for some other system? What's Linux?
This author doesn't have a chance in a room full of newbies. You have to have your technical dope straight, and it's not enough.
This is indeed the thing that Windows has called "shell" before adopting that word for command interpreters (PowerShell).
explorer.exe uses the "ShellApi" to do some of its things, which applications can also use, like launching a file via its associated application (ShellExecute/ShellExecuteEx).
I don't want to discourage you, but if you know the subject matter and are inclined to work on a book, you should just write your own from scratch as the sole author: the normal way most books are written.
(I personally wouldn't write a book about the shell. I would be too paralyzed by the thought that at any moment, Stéphane Chazelas might crack it open.)
"The best way to get the right answer on the Internet is not to ask a question, it's to post the wrong answer." - Ward Cunningham (possibly)
Thanks to those who've commented with corrections/suggestions, I'll go through them try and get the content fixed up where possible. Will need to have a lie down and possibly an epidural before going through the comments again to pull out the changes needed.
For people who think that they're the intended audience of these kinds of guides: do you find them effective?
I learned how to use a POSIX shell years before I learned how to program (outside of a shell, that is), and I've often wondered how to help people (particularly younger colleagues) who learned things the other way around.
My experience so far is that a lot of people get stuck in an "experience trough," for lack of a better phrase: they're comfortable navigating around the filesystem and running basic commands, but struggle with some of the more powerful building blocks that can make someone a real shell power user (`xargs`, nontrivial uses of `find`, here-strings, subshells and blocks, process substitution, etc.).
The single biggest stumbling block I've seen in my co-workers is quotes. Most of them think in terms of python/javascript/etc where quotes mean string type, and have no concept of how args and splitting on spaces works, or how it's all stringly-typed.
That "experience trough" you describe I think exists because they don't actually understand the fundamentals, so any time they've attempted something new it's based on assumptions from a totally different paradigm, does something strange and unexpected, and reinforces sticking to the same safe and simple commands.
Yup. I came the same path as you, and realized about the same.
It's just a different mindset. Instead of a std library you have tons of prebuilt binaries ready to perform actions and talk to one another. And that is really, really, powerful once you learn what's all available and how to use/abuse it.
Software engineering in general is so different. Tons of mantras to follow, TDD, OOP, etc.
Each has its place, but generally if I can do it in a few lines of shell, why spend 20 minutes writing a hundred lines of something more pure? On the other hand, shells do not seem well suited for anything large and complex, admittedly. So watching people waste an hour writing something that could be a Bash one liner is about as painful as watching someone try to debug a 500 line bash script.
Do you mind elaborating? What do you recommend for progressing out of that? I'm pushing for Red Hat learning from my employer. Any favorite learning resources?
This is the problem: I don't really know. It doesn't help that POSIX sh (and GNU Bash) are objectively terrible programming languages, and that it takes a decent amount of masochism to "enjoy" them in the way that (I think) engenders competence.
For me, it was a lot of scripting and then iterating on those scripts when I realized that they conceptually broke down in places (even if they never broke in practice). Things like spaces where you don't want them, needing to pass around data that might be too large for `argv` (because of stack limits), etc.
One needs to sit down and read through in-depth documentation or a book like this one. Many people don’t seem to have the patience or interest to do so.
The author of the musl C library has a very poor opinion of the POSIX shell, and has published his "tricks" to force expected behavior.
This is very useful for those who are stumped by a failing script.
"I am a strong believer that Bourne-derived languages are extremely bad, on the same order of badness as Perl, for programming, and consider programming sh for any purpose other than as a super-portable, lowest-common-denominator platform for build or bootstrap scripts and the like, as an extremely misguided endeavor. As such you won’t see me spending many words on extensions particular to ksh, Bash, or whatever other shells may be popular."
For me, it helped to stop thinking of the shell as an efficient way to accomplish a task (it certainly is), but as a way to express my intent (the way I would in a normal programming language). That, combined with some automated tooling (shellcheck) and a handful of best practices (arrays over strings, avoiding the "cat pipe" pattern[1], etc.) has worked well for me. But I don't know how well that generalizes!
I did it by blocking out time to read the manual pages. Yes, it is slow and tedious. But the manuals make a lot more sense once you already know how the tools work at a basic level.
> but struggle with some of the more powerful building blocks that can make someone a real shell power user (`xargs`, nontrivial uses of `find`, here-strings, subshells and blocks, process substitution, etc.).
I am one of those... as soon as I don't know how to do something in the shell, though, I just fire up a REPL or a "scratch" in the IDE and write what I want in a "proper" language. Sorry, but shel scripting is not always the best way to go. I eventually did learn me some AWK and that was useful, but most of the time I don't use even that, it's just so much easier for a programmer to use a programming language to do stuff.
Google shell style guide [0] was also a good read. I thought that the "When to use Shell" section is a section that is good for any kind of guide, not just for bash / shell.
Also, maybe not so much a pitfall / bug, but something I had to deal with recently was that bash does not handle the EINTR when calling write() in the printf and echo builtins [1][2][3], etc.
If you are writing a script that is more than 100 lines long, or that uses non-straightforward control flow logic, you should rewrite it in a more structured language now. Bear in mind that scripts grow. Rewrite your script early to avoid a more time-consuming rewrite at a later date.
Q: what's up with this '/bin/echo' ?
A: bash's builtin 'echo' command does not check calls to write() against
errors. If you use it in the cgroup file system, you won't be
able to tell whether a command succeeded or failed.
I will use the word `bash` for convenience. `bash` is powerful because it is convenient. `bash` is also powerful because it doesn't tell users how to behave or what to do. `bash` is all about empowering the user. That is what makes it great. `bash` is a meta language for the convenience and the empowerment of programmers. Over time, impossible tasks become trivial. You start off by using `bash` to create simple models of larger/more complex problems, then you adopt pipes and redirection, and eventually you don't need much handholding and you are 10x better at modeling problems, mocking designs, and general problem solving. Will there be 8 hour days to understand some simple concepts? Sure. Do they pay off? Yes. Absolutely yes. Undergraduate computer science background makes bash even more useful and helpful. Outside of college learning, you might be able to grok some things using man pages, tutorials, and various resources (like books).
I've actively used bash and vi every day for decades now, but somehow never tried this mode.
It's fascinating. I wonder if it will be worth it for the change of command-line editing habits. It certainly feels faster for most command-line edits.
I am a vi user and tried vi mode in zsh but never liked it. For some reason the modal aspect doesn’t fit for me in the context of the shell. I’ve learned enough emacs bindings to work the shell.
Heavy shell and vim user here, but I simply cannot imagine how this might be convenient. A bunch of readline-style shortcuts will do the job just fine. And they are everywhere. SSH, database prompts, even login screens support them.
Preemptively justifying something more efficient that you haven't used, because what you have is "fine," is a behavior I see from heavy Vim users when they justify not using IDEs, which are much more efficient than Vim for writing code. (I'm also a heavy Vim user, and IDEs are so much nicer for coding, especially efficiency). So, I'd say your story checks out.
Programs that use readline can be configured to use vi mode via inputrc. For the few that aren't you can always use rlwrap. Overall there's no downside.
With Ubuntu using dash as its /bin/sh and macOS defaulting to zsh for interactive shells (I think their /bin/sh is still bash, but haven’t checked), this is probably the least true now that it’s been for a couple decades.
When called as /bin/sh or if the POSIXLY_CORRECT environment variable is set, it will act as a POSIX shell.
If not, it will behave very differently. The one major difference that I know is that aliases will be ignored in the execution of a script, so "alias p=printf" will fail (for example). This is not POSIX-compliant.
Thanks for taking the time to do this and share it. I think it's a great resource for someone just starting out (this guy). Then more advanced materials can take it from there.
But then, nope:
> Windows is not anything like Linux under the hood. So to get a shell working, we have three options:
> Use a tool which provides common Linux tools which have been written to work with Windows
> Use a "virtual machine" running Linux
> Use the Windows Subsystem for Linux
What "common Linux tools which have been written to work with Windows" is referring to is installing Cygwin.
This a fairly hamfisted presentation of a subject matter that could be handled in a way that is simultaneously easier to grasp and factually correct.
First of all, your goal is to teach about a specific command interpreter language, you really want to clarify how there are multiple ones which have different syntax, and brush that whole topic aside as fast as possible.
> You can see what the version is of your shell by running:
> bash --version
Yikes. If we know we are running Bash, the way to know what version it is is:
and not to run another instance of Bash.Can this book teach something simple, like comments?
> The shell ignores any text which follows a # hash symbol.
Wrong:
"this is seen" looks like an example of "any text" to me.A hash comment marker is ignored if it is not quoted and escaped so as to be part of a word. The hash is also part of certain bits of syntax:
https://blog.nullspace.io/batch.html
Powershell is a new arrival, limited outside of Windows. I admit that I don't know it.
The POSIX shell also has severe problems with ambiguity, and it is not an LR-parsed language:
https://archive.fosdem.org/2018/schedule/event/code_parsing_...
Use the right tool for the job, imperfect as they may be.
It is unfortunate that awk didn't grow and supplant the shell, as for many years the BWK "One True Awk" did use straightforward lex and yacc grammars.
It’s actually based on the DOS command shell [1], which in turn was modeled after CP/M [2].
[1] https://en.m.wikipedia.org/wiki/COMMAND.COM
[2] https://en.m.wikipedia.org/wiki/CP/M
I've linked to this post/comments on the repo (issue #210), they should all get addressed but it might be a while before the online version gets sorted, I'm closing out the final chapters online, once I've done this I'll likely go back to the beginning and rewrite/edit/put in a more consistent style (more code snippets, fewer screenshots etc).
I disagree with all of them from an educational point of view.
Having said that, it helps to clarify whether the audience is technical or education oriented. The title implies a technical book, so I'm with you on the comments.
This author doesn't have a chance in a room full of newbies. You have to have your technical dope straight, and it's not enough.
explorer.exe uses the "ShellApi" to do some of its things, which applications can also use, like launching a file via its associated application (ShellExecute/ShellExecuteEx).
(I personally wouldn't write a book about the shell. I would be too paralyzed by the thought that at any moment, Stéphane Chazelas might crack it open.)
Thanks to those who've commented with corrections/suggestions, I'll go through them try and get the content fixed up where possible. Will need to have a lie down and possibly an epidural before going through the comments again to pull out the changes needed.
I learned how to use a POSIX shell years before I learned how to program (outside of a shell, that is), and I've often wondered how to help people (particularly younger colleagues) who learned things the other way around.
My experience so far is that a lot of people get stuck in an "experience trough," for lack of a better phrase: they're comfortable navigating around the filesystem and running basic commands, but struggle with some of the more powerful building blocks that can make someone a real shell power user (`xargs`, nontrivial uses of `find`, here-strings, subshells and blocks, process substitution, etc.).
That "experience trough" you describe I think exists because they don't actually understand the fundamentals, so any time they've attempted something new it's based on assumptions from a totally different paradigm, does something strange and unexpected, and reinforces sticking to the same safe and simple commands.
It's just a different mindset. Instead of a std library you have tons of prebuilt binaries ready to perform actions and talk to one another. And that is really, really, powerful once you learn what's all available and how to use/abuse it.
Software engineering in general is so different. Tons of mantras to follow, TDD, OOP, etc.
Each has its place, but generally if I can do it in a few lines of shell, why spend 20 minutes writing a hundred lines of something more pure? On the other hand, shells do not seem well suited for anything large and complex, admittedly. So watching people waste an hour writing something that could be a Bash one liner is about as painful as watching someone try to debug a 500 line bash script.
I’m not really sure it’s fair to assume that you can use advanced topics after reading any book aimed at beginners.
Also, no one is going to learn how to use any of your advanced topics unless they have a need. And people rarely have a need for those things.
For me, it was a lot of scripting and then iterating on those scripts when I realized that they conceptually broke down in places (even if they never broke in practice). Things like spaces where you don't want them, needing to pass around data that might be too large for `argv` (because of stack limits), etc.
This is very useful for those who are stumped by a failing script.
"I am a strong believer that Bourne-derived languages are extremely bad, on the same order of badness as Perl, for programming, and consider programming sh for any purpose other than as a super-portable, lowest-common-denominator platform for build or bootstrap scripts and the like, as an extremely misguided endeavor. As such you won’t see me spending many words on extensions particular to ksh, Bash, or whatever other shells may be popular."
https://www.etalabs.net/sh_tricks.html
[1]: https://www.linuxjournal.com/content/put-down-pipe
I am one of those... as soon as I don't know how to do something in the shell, though, I just fire up a REPL or a "scratch" in the IDE and write what I want in a "proper" language. Sorry, but shel scripting is not always the best way to go. I eventually did learn me some AWK and that was useful, but most of the time I don't use even that, it's just so much easier for a programmer to use a programming language to do stuff.
Google shell style guide [0] was also a good read. I thought that the "When to use Shell" section is a section that is good for any kind of guide, not just for bash / shell.
Also, maybe not so much a pitfall / bug, but something I had to deal with recently was that bash does not handle the EINTR when calling write() in the printf and echo builtins [1][2][3], etc.
[0] https://google.github.io/styleguide/shellguide.html#s1.2-whe...
[1] https://unix.stackexchange.com/a/487260 [2] https://github.com/torvalds/linux/blob/ca1fdab7fd27eb069df13... [3] https://lists.gnu.org/archive/html/bug-bash/2018-01/msg00031...It's fascinating. I wonder if it will be worth it for the change of command-line editing habits. It certainly feels faster for most command-line edits.
And neither bash nor sh are that great, so no reason to put down the idea that they shouldn't be the "default"
When called as /bin/sh or if the POSIXLY_CORRECT environment variable is set, it will act as a POSIX shell.
If not, it will behave very differently. The one major difference that I know is that aliases will be ignored in the execution of a script, so "alias p=printf" will fail (for example). This is not POSIX-compliant.