Okay, so when I worked at Sony about 25 years ago, I got assigned this project to fix our order management system, which was extremely slow, and kept crashing.
I jumped in and started digging around, and to my horror, the OMS was a giant set of shell scripts running on an AIX server, which evolved over a decade and was abandoned. It was over 50,000 lines of code! It was horrendous and shit kept timing out everywhere -- orders, payments, and other information were moved from server to server over FTP, parsed with complicated sed/awk, and inventory was tracked in text files (also FTPd around.)
At the time, perl seemed like the most practical way for me to migrate the mess -- I rewrote all of the shell piece by piece, starting with the simplest peices and replaced them with small perl modules as part of a larger perl application, refactoring along the way. It took me 3 months and I moved the whole thing to about 5000 lines of perl, and it ran 10-100x faster with almost none of the failures in the original system.
As terrible as it was, it's one of the most satisfying things I've ever done. :-)
That's deleting 800 lines a day, each day. Did you need to read through the original code, get a deep understanding and match its behavior exactly or did you throw away huge chunks and write new code as you thought it should behave?
Was there a lot of boilerplate that could be replaced quickly?
There was tons of duplicate code, unnecessary code, dead code, etc. There was also a lot of code for which CPAN modules could entirely replace. (Also, I'm a workaholic who obsesses about a problem until it's fully solved.)
OSH is the most bash-compatible shell in the world, and YSH is a new language
ls | sort | uniq | wc -l # this is both OSH and YSH
var mydict = {foo: 42, bar: ['a', 'b']} # this is new YSH stuff you can start using
json write (mydict)
The difference between OSH and YSH is exactly a set of "shopt" options [1], although YSH feels like a brand new language too! There is a smooth blend.
I think it's worth it for 2 things alone
- YSH checks all errors - you never lose an exit code
- YSH has real arrays and doesn't mangle your variables with word splitting
There's a lot more: modules with namespaces (use mymodule.ysh), buffered I/O that's not slow, etc.
The first really big one I wrote was the ~7000 line installer for the Enrust CA and directory, which ran on, well, all Unixes at that time. It didn't initially, of course, but it grew with customer demand.
The installation itself wasn't especially complicated, but upgrades were, a little, and this was back when every utility on every Unix had slight variations.
Much of the script was figuring out and managing those differences, much was error detection and recovery and rollback, some was a very primitive form of package and dependency management....
DEC's Unix (the other one, not Ultrix) was the most baffling. It took me days to realize that all command line utilities truncated their output at column width. Every single one. Over 30 years later and that one still stands out.
Every release of HP-UX had breaking changes, and we covered 6.5 to 11, IIRC. I barely remember Ultrix or the Novell one or Next, or Sequent. I do remember AIX as being weird but I don't remember why. And of course even Sun's three/four OS's had their differences (SunOS pre 4.1.3; 4.1.3; Solaris pre 2; and 2+) but they had great FMs. The best.
I think he was meaning that they truncated the lines even when called from a script, with their output going somewhere other than a terminal, not just when run interactively
> DEC's Unix (the other one, not Ultrix) was the most baffling. It took me days to realize that all command line utilities truncated their output at column width. Every single one. Over 30 years later and that one still stands out.
OK, so JOOC I ran wc against the main binary and supporting libraries for a project I did last year: It's a script to manage linear assured pipelines implemented as a series of containers (an input protocol adapter, one or more filters, an output protocol adapter). The whole thing was intended to be useful for people who aren't necessarily experts in either containers or protocols, but who have an idea of how they want to filter/transform files as they transit the pipeline.
It's 6224 lines, so far.
There is a top-level binary with sub-functions, sort of like how
There is a sub command to add a new sub command, which creates the necessary libraries and pre-populates function definitions from a template; the template includes short and long usage functions, so that
cbap -h
or
cbap pipeline -h
give useful and reasonable advice.
There are subcommands for manipulating base images, components (which are images with specific properties for use as containers in the pipelines), and pipelines themselves. A LOT of code is for testing, to make sure that the component and pipeline definitions are correctly formatted. (Pipelines are specified in something-almost-TOML, so there is code to parse toml, convert sections to arrays, etc., while components are specified as simple key=value files, so there is code to parse those, extract LHS and RHS, perform schema validation, etc.).
Since pipeline components can share properties, there is code to find common properties in var and etc files, specify component properties, etc.
There are a lot user and group and directory and FIFO manipulation functions tailored to the security requirements: When a pipeline is setup, users and groups and SEL types and MCS categories are generated and applied, then mapped into to the service files that start the components (so there is a lot of systemd manipulation as well).
Probably the single biggest set of calls are the functions that get/set component properties (which are really container properties) and allow us to use data-driven container definitions, with each property having a get function, a validation function, and an inline (in a pipeline) version, for maximum flexibility.
Finally, there is code that uses a lot of bash references to set variables either from files, the environment, or the command line, so that we can test rapidly.
It also support four levels of user, from maintainer (people who work on the code itself), developer (people who develop component definitions), integrators (people who build pipelines from components), and operators (people who install pipelines), with the ability to copy and package itself for export to users at any of those levels (there is a lot of data-driven, limited recursive stuff happening therein).
Since target systems can be any Linux, it uses makeself to package and extract itself.
For example, an integrator can create a pipeline definition, which will produce a makeself file that, when run on the target system, will create all users, groups, directories, FIFOs (the inter-component IPC), apply DAC and MAC, create systemd files, copy images to each user, and launch the pipeline - with a delete option to undo all of that.
There is some seccomp in there as well, but we've paused that as we need to find the right balance between allow- and deny- listing.
At one point I considered writing an interpreter for my scripting language Lil in bash to maximize portability, but quickly realized that floating-point arithmetic would be extremely painful (can't even necessarily depend on bc/dc being available in every environment) and some of the machines in my arsenal have older versions of bash with very limited support for associative arrays. My compromise was to instead target AWK, which is a much more pleasant general-purpose language than most shells, and available in any POSIX environment: https://beyondloom.com/blog/lila.html
> can't even necessarily depend on bc/dc being available in every environment
Just discovered this myself, also trying to make a language target shell. Was really surprised bc/dc was not present I think in Ubuntu install in WSL2. Also using awk for floating point math, but just shelling out to it.
Yep! I considered shelling out to AWK for the same reason, as a bc/dc alternative, but rapidly found that nearly everything else bash could do was easier and less error-prone (and workable on much older systems) if I moved the whole script into pure AWK.
As someone who has written and maintained large Perl programs at various points in my career. There is a reason why people do this- Java and Python like languages work fine when interfaces and formats are defined, and you often have 0 OS interaction. That is, you use JSON/XML/YAML or interact with a database or other programs via http(s). This creates an ideal situation where these languages can shine.
When people do large quantity text and OS interaction work, languages like Java and Python are a giant pain. And you will begin to notice how Shell/Perl become a breeze to do this kind of work.
This means nearly every automation task, chaotic non-standard interfaces, working with text/log files, or other data formats that are not structured(or at least well enough). Add to this Perl's commitment towards backwards compatibility, a large install base and performance. You have 0 alternatives apart from Perl if you are working to these kind of tasks.
I have long believed that a big reason for so much manual drudgery these days, with large companies hiring thousands of people to do trivially easy to automate tasks is because Perl usage dropped. People attempt to use Python or Java to do some big automation tasks and quit soon enough when they are faced with the magnitude of verbosity and overall size of code they have to churn and maintain to get it done.
Strong disagree that it's because "Omg, no more Perl" but just complexity cranked up and that Perl person stitching scripts together became their full job and obviously Perl only got you so far. So now you have additional FTE who is probably expensive.
Also, if end user is on Windows, there is already Perl like option on their desktop, it's called Powershell and will perform similar to Perl.
I did a big automation task in native code, because efficiency is desirable in such cases, while bash+grep favor running a new process for every text line. In order to be efficient, you need to minimize work, and thus batch and deduplicate it, which means you need to handle data in a stateful manner while tracking deduplication context, which is easier in a proper programming language, while bash+grep favor stateless text processing and thus result in much work duplication. Another strategy for minimization of work is accurate filtering, which is easier to express imperatively with nice formatting in a proper programming language, grep and regex are completely unsuitable for this. Then if you use line separated format, git awards you with escaping to accommodate for whatever, which is inconsistently supported and can be disabled by asking null terminated string format with -z option, I don't think bash has any way to handle it, while in a sufficiently low level language it's natural, and it also allows for incremental streaming so you don't have to start a new process for every text line.
As a bonus you can use single code base for everything no matter if there's http or something else in the line.
Yes I agree - my favorite language is Python, but it can be annoying/inefficient for certain low-level OS things. This is why I created https://www.oilshell.org (and the linked wiki page)
I've been seriously considering learning some Perl 5-fu ever since I realized it's installed by default on so many Linux and BSD systems. I think even OpenBSD comes with perl installed.
That may not seem like a big advantage until you're working in an environment where you don't actually have the advantage of just installing things from the open Internet (or reaching the Internet at all).
I think the main problem with writing large programs as bash scripts is that shell scripting languages were never really designed for complexity. They excel at orchestrating small commands and gluing together existing tools in a quick, exploratory way. But when you start pushing beyond a few hundred lines of Bash, you run into a series of limitations that make long-term maintenance and scalability a headache.
First, there’s the issue of readability. Bash's syntax can become downright cryptic as it grows. Variable scoping rules are subtle, error handling is primitive, and string handling quickly becomes messy. These factors translate into code that’s harder to maintain and reason about. As a result, future maintainers are likely to waste time deciphering what’s going on, and they’ll also have a harder time confidently making changes.
Next, there’s the lack of robust tooling. With more mature languages, you get static analysis tools, linters, and debuggers that help you spot common mistakes early on. For bash, most of these are either missing or extremely limited. Without these guardrails, large bash programs are more prone to silent errors, regressions, and subtle bugs.
Then there’s testing. While you can test bash scripts, the process is often more cumbersome. Complex logic or data structures make it even trickier. Plus, handling edge cases—like whitespace in filenames or unexpected environment conditions—means you end up writing a ton of defensive code that’s painful to verify thoroughly.
Finally, the ecosystem just isn’t built for large-scale Bash development. You lose out on modularity, package management, standardized dependency handling, and all the other modern development patterns that languages like Python or Go provide. Over time, these deficits accumulate and slow you down.
I think using Bash for one-off tasks or simple automation is fine — it's what it’s good at. But when you start thinking of building something substantial, you’re usually better off reaching for a language designed for building and maintaining complex applications. It saves time in the long run, even if the initial learning curve or setup might be slightly higher.
Using ShellCheck as a linter can catch a lot of the common footguns and there are a LOT of footguns and/or unexpected behaviour that can catch out even experienced Bash writers. However, Bash/shell occupies a unique place in the hierarchy of languages in that it's available almost everywhere and will still be around in 30 years. If you want a program that will run almost everywhere and still run in 30 years time, then shell/Bash is a good choice.
I'd almost always prefer c99 to shell for anything more than 100 lines of code or so. There is even a project I saw here recently that can bootstrap tcc in pure shell (which can then be used to bootstrap gcc). I'm somewhat skeptical that bash will still be used for anything but legacy scripts in 30 years, despite it's impressive longevity to this point, but I could sadly be proven wrong.
Dealing with this at work right now. Digging through thousands of lines of Bash. This script wasn’t written a long time ago, so no clue why they went with Bash.
The script works but it always feels like something is going to break if I look at the code the wrong way.
If you have thousands of lines of bash, don't like maintaining it, but don't necessarily want to rewrite the whole thing at once, that's what https://www.oilshell.org/ is for!
(I created the project and the wiki page. Right now the best bet is to join https://oilshell.zulipchat.com/ if it interests you. People who want to test it out should be comfortable with compiling source tarballs, which is generally trivial because shells have almost no dependencies.)
The first step is:
shopt --set strict:all # at the top of the file
Or to run under bash
shopt -s strict:all 2>/dev/null || true
And then run with "osh myscript.bash"
OSH should run your script exactly the same as bash, but with better error messages, and precise source locations.
And you will get some strictness errors, which can help catch coding bugs. It's a little like ShellCheck, except it can detect things at runtime, whereas ShellCheck can't.
I'm pretty sure the largest handwritten shell program I used back in the day on a regular basis was abcde (A Better CD Encoder)[1] which clocks in at ~5500 LOC.[2]
Not that I'd know anything about it, but this was one of the tools recommended on What.CD back in the day. Along with Max (my friends tell me) https://github.com/sbooth/Max
Many of these programs are true gems; the rkhunter script, for instance is both nice code (can be improved) and a treasure trove of information*.
Note that much of the code size of these scripts is dedicated to ensuring that the right utilities exist across the various platforms and perform as expected with their various command line options. This is the worst pain point of any serious shell script author, even worse than signals and subprocesses (unless one enjoys the pain).
*Information that, I would argue, would be less transparent if rkhunter had been written in a "proper" programming language. It might be shoved off in some records in data structures to be retrieved; actions might be complex combinations of various functions---or, woe, methods and classes---on nested data structures; logging could be JSON-Bourned into pieces and compressed in some database to be accessed via other methods and so on.
Shell scripts, precisely due to the lack of such complex tools, tend to "spill the beans" on what is happening. This makes rkhunter, for instance, a decent documentation of various exploits and rootkits without having to dig into file upon file, structure upon structure, DB upon DB.
The FreeBSD Update client is about 3600 lines of sh code. Not huge compared to some of the other programs mentioned here, but I'm inclined to say that "tool for updating an entire operating system" is a pretty hefty amount of functionality.
The code which builds the updates probably adds up to more lines, but that's split across many files.
Readit News
I jumped in and started digging around, and to my horror, the OMS was a giant set of shell scripts running on an AIX server, which evolved over a decade and was abandoned. It was over 50,000 lines of code! It was horrendous and shit kept timing out everywhere -- orders, payments, and other information were moved from server to server over FTP, parsed with complicated sed/awk, and inventory was tracked in text files (also FTPd around.)
At the time, perl seemed like the most practical way for me to migrate the mess -- I rewrote all of the shell piece by piece, starting with the simplest peices and replaced them with small perl modules as part of a larger perl application, refactoring along the way. It took me 3 months and I moved the whole thing to about 5000 lines of perl, and it ran 10-100x faster with almost none of the failures in the original system.
As terrible as it was, it's one of the most satisfying things I've ever done. :-)
That's deleting 800 lines a day, each day. Did you need to read through the original code, get a deep understanding and match its behavior exactly or did you throw away huge chunks and write new code as you thought it should behave?
Was there a lot of boilerplate that could be replaced quickly?
50,000/90 = 555 ???
https://www.oilshell.org/
OSH is the most bash-compatible shell in the world, and YSH is a new language
The difference between OSH and YSH is exactly a set of "shopt" options [1], although YSH feels like a brand new language too! There is a smooth blend.I think it's worth it for 2 things alone
- YSH checks all errors - you never lose an exit code
- YSH has real arrays and doesn't mangle your variables with word splitting
There's a lot more: modules with namespaces (use mymodule.ysh), buffered I/O that's not slow, etc.
Gradually upgrading -https://github.com/oils-for-unix/oils/wiki/Gradually-Upgradi... (people are writing new YSH, but not many people have gradually upgraded, so I'd definitely appreciate feedback from people with a big "shell script problem")
---
There is a FAQ here about Perl:
Are you reinventing Perl? - https://www.oilshell.org/blog/2021/01/why-a-new-shell.html#a...
Not to say that migrating to Perl is worse in any way, i.e. if you already know Perl or your team knows it.
But objectively YSH is also a shell, so I think more of the code carries over, and there is a more direct upgrade path.
---
[1] Unix Shell Should Evolve like Perl 5 - https://www.oilshell.org/blog/2020/07/blog-roadmap.html#the-... - i.e. with compatible upgrade options
Haskell (e.g. shh) and Clojure (Babashka) are also a nice for this usecase, but more niche options.
The first really big one I wrote was the ~7000 line installer for the Enrust CA and directory, which ran on, well, all Unixes at that time. It didn't initially, of course, but it grew with customer demand.
The installation itself wasn't especially complicated, but upgrades were, a little, and this was back when every utility on every Unix had slight variations.
Much of the script was figuring out and managing those differences, much was error detection and recovery and rollback, some was a very primitive form of package and dependency management....
DEC's Unix (the other one, not Ultrix) was the most baffling. It took me days to realize that all command line utilities truncated their output at column width. Every single one. Over 30 years later and that one still stands out.
Every release of HP-UX had breaking changes, and we covered 6.5 to 11, IIRC. I barely remember Ultrix or the Novell one or Next, or Sequent. I do remember AIX as being weird but I don't remember why. And of course even Sun's three/four OS's had their differences (SunOS pre 4.1.3; 4.1.3; Solaris pre 2; and 2+) but they had great FMs. The best.
Do you mean OSF1/, Digital Unix, or Tru64 Unix?
It's 6224 lines, so far.
There is a top-level binary with sub-functions, sort of like how
or work.There is a sub command to add a new sub command, which creates the necessary libraries and pre-populates function definitions from a template; the template includes short and long usage functions, so that
or give useful and reasonable advice.There are subcommands for manipulating base images, components (which are images with specific properties for use as containers in the pipelines), and pipelines themselves. A LOT of code is for testing, to make sure that the component and pipeline definitions are correctly formatted. (Pipelines are specified in something-almost-TOML, so there is code to parse toml, convert sections to arrays, etc., while components are specified as simple key=value files, so there is code to parse those, extract LHS and RHS, perform schema validation, etc.).
Since pipeline components can share properties, there is code to find common properties in var and etc files, specify component properties, etc.
There are a lot user and group and directory and FIFO manipulation functions tailored to the security requirements: When a pipeline is setup, users and groups and SEL types and MCS categories are generated and applied, then mapped into to the service files that start the components (so there is a lot of systemd manipulation as well).
Probably the single biggest set of calls are the functions that get/set component properties (which are really container properties) and allow us to use data-driven container definitions, with each property having a get function, a validation function, and an inline (in a pipeline) version, for maximum flexibility.
Finally, there is code that uses a lot of bash references to set variables either from files, the environment, or the command line, so that we can test rapidly.
It also support four levels of user, from maintainer (people who work on the code itself), developer (people who develop component definitions), integrators (people who build pipelines from components), and operators (people who install pipelines), with the ability to copy and package itself for export to users at any of those levels (there is a lot of data-driven, limited recursive stuff happening therein).
Since target systems can be any Linux, it uses makeself to package and extract itself.
For example, an integrator can create a pipeline definition, which will produce a makeself file that, when run on the target system, will create all users, groups, directories, FIFOs (the inter-component IPC), apply DAC and MAC, create systemd files, copy images to each user, and launch the pipeline - with a delete option to undo all of that.
There is some seccomp in there as well, but we've paused that as we need to find the right balance between allow- and deny- listing.
(Yes, I use shellcheck. Religiously. :->)
Just discovered this myself, also trying to make a language target shell. Was really surprised bc/dc was not present I think in Ubuntu install in WSL2. Also using awk for floating point math, but just shelling out to it.
https://pubs.opengroup.org/onlinepubs/9699919799.2008edition...
When people do large quantity text and OS interaction work, languages like Java and Python are a giant pain. And you will begin to notice how Shell/Perl become a breeze to do this kind of work.
This means nearly every automation task, chaotic non-standard interfaces, working with text/log files, or other data formats that are not structured(or at least well enough). Add to this Perl's commitment towards backwards compatibility, a large install base and performance. You have 0 alternatives apart from Perl if you are working to these kind of tasks.
I have long believed that a big reason for so much manual drudgery these days, with large companies hiring thousands of people to do trivially easy to automate tasks is because Perl usage dropped. People attempt to use Python or Java to do some big automation tasks and quit soon enough when they are faced with the magnitude of verbosity and overall size of code they have to churn and maintain to get it done.
Also, if end user is on Windows, there is already Perl like option on their desktop, it's called Powershell and will perform similar to Perl.
As a bonus you can use single code base for everything no matter if there's http or something else in the line.
A few links for context:
Are you reinventing Perl?
https://www.oilshell.org/blog/2021/01/why-a-new-shell.html#a...
The Unix Shell Should Evolve Like Perl 5 (with compatible upgrade options, rather than a big bang like Perl 6/Raku)
https://www.oilshell.org/blog/2020/07/blog-roadmap.html#the-...
A Tour of YSH - https://www.oilshell.org/release/latest/doc/ysh-tour.html
That may not seem like a big advantage until you're working in an environment where you don't actually have the advantage of just installing things from the open Internet (or reaching the Internet at all).
First, there’s the issue of readability. Bash's syntax can become downright cryptic as it grows. Variable scoping rules are subtle, error handling is primitive, and string handling quickly becomes messy. These factors translate into code that’s harder to maintain and reason about. As a result, future maintainers are likely to waste time deciphering what’s going on, and they’ll also have a harder time confidently making changes.
Next, there’s the lack of robust tooling. With more mature languages, you get static analysis tools, linters, and debuggers that help you spot common mistakes early on. For bash, most of these are either missing or extremely limited. Without these guardrails, large bash programs are more prone to silent errors, regressions, and subtle bugs.
Then there’s testing. While you can test bash scripts, the process is often more cumbersome. Complex logic or data structures make it even trickier. Plus, handling edge cases—like whitespace in filenames or unexpected environment conditions—means you end up writing a ton of defensive code that’s painful to verify thoroughly.
Finally, the ecosystem just isn’t built for large-scale Bash development. You lose out on modularity, package management, standardized dependency handling, and all the other modern development patterns that languages like Python or Go provide. Over time, these deficits accumulate and slow you down.
I think using Bash for one-off tasks or simple automation is fine — it's what it’s good at. But when you start thinking of building something substantial, you’re usually better off reaching for a language designed for building and maintaining complex applications. It saves time in the long run, even if the initial learning curve or setup might be slightly higher.
The script works but it always feels like something is going to break if I look at the code the wrong way.
See my comment here, with some details: https://news.ycombinator.com/item?id=42354095
(I created the project and the wiki page. Right now the best bet is to join https://oilshell.zulipchat.com/ if it interests you. People who want to test it out should be comfortable with compiling source tarballs, which is generally trivial because shells have almost no dependencies.)
The first step is:
Or to run under bash And then run with "osh myscript.bash"OSH should run your script exactly the same as bash, but with better error messages, and precise source locations.
And you will get some strictness errors, which can help catch coding bugs. It's a little like ShellCheck, except it can detect things at runtime, whereas ShellCheck can't.
[1] https://abcde.einval.com
[2] https://git.einval.com/cgi-bin/gitweb.cgi?p=abcde.git;a=blob...
Allegedly.
Note that much of the code size of these scripts is dedicated to ensuring that the right utilities exist across the various platforms and perform as expected with their various command line options. This is the worst pain point of any serious shell script author, even worse than signals and subprocesses (unless one enjoys the pain).
*Information that, I would argue, would be less transparent if rkhunter had been written in a "proper" programming language. It might be shoved off in some records in data structures to be retrieved; actions might be complex combinations of various functions---or, woe, methods and classes---on nested data structures; logging could be JSON-Bourned into pieces and compressed in some database to be accessed via other methods and so on.
Shell scripts, precisely due to the lack of such complex tools, tend to "spill the beans" on what is happening. This makes rkhunter, for instance, a decent documentation of various exploits and rootkits without having to dig into file upon file, structure upon structure, DB upon DB.
The code which builds the updates probably adds up to more lines, but that's split across many files.