"If I type a shell command that's valuable - one that did something useful enough that I might want it again in future, and long and complicated enough that I'd be annoyed to have to figure it out a second time from scratch - then I can't rely on it just happening to be in my .bash_history. So instead I put it somewhere else: maybe a shell function in my .bashrc, or maybe a shell script in my directory of random useful scriptlets."
I have a folder marked executable with hundreds of small shell scripts, appended to $PATH. I never write long scripts. I sometimes write scripts that run other scripts but I try to avoid such dependencies. I do not use bash, I use NetBSD Almquist or modified Debian Almquist shell so all the scripts are highly portable. I can move a tarball of this folder from computer to computer, whether the OS is Linux or BSD. These scripts generally run faster than bash scripts.
It would be untenable to save all command line history since I spend every day in the shell in textmode (not x11) and only occasionally switch to a graphical environment. I sometimes use Almquist-based shells that have command line history removed. This forces me to write useful scripts.
A lot of cumulative shell knowledge is contained in that folder of scripts. These scripts have worked for decades unchanged. I think the Almquist shell may be my favourite interpreter. No matter what else I try I always come back to the shell language. There is something appealing about scripts that seem to work forever.
>It would be untenable to save all command line history since I spend every day in the shell in textmode (not x11) and only occasionally switch to a graphical environment
I don't understand this remark. What has x11 got to do with shell history? My bash appends all commands to .bash_history, immediately, regardless of whether it's run inside a terminal emulator or a VT.
Context switching between textmode and a graphics layer, like x11, is IME time-consuming and relatively janky. Whereas switching from a "terminal window" to some graphical program in a graphical environment like x11 is very quick and relatively smooth. Because I prefer textmode, I stay in textmode and I am not switching to a graphical environment very often, if at all. I am not using graphical programs. Consequently I accumulate an inordinate amount of command line history. Too much to save, IMHO.
As I mentioned, I do not use bash, I use a smaller, faster shell. I have no ".bash_history". And sometimes I use even smaller shells that have no command line history, in order to reduce program size.
I think they were implying that they generate a lot at the CLI, because they gots no GUI. Their whole day of computing would be pushing into a history file whereas most folk gotta lotta clickin’ happening too.
Might be wrong though.
I read this article in horror. Even after he said he kept /only/ the last 9800 commands.
⟩ history | wc -l
170919
I have (deduplicated) shell history with timestamps going back years. A little bit of CTRL+R and `history | grep …` I never have to remember how I did some arcane magic in the past, just that I did.
Maybe I'm a hoarder. My timestamped histories from four machines in four different locations added up to 898028 lines --- I kept most of my shell commands, including errors and iterative attempts to get a good command, since early 2009 in the servers that matter the most to me. I use named shells with different histories each (roughly one to four shells per project), so the largest single shell history in these logs is only 43424 lines (21712 commands). The average across all shells is a bit over 75 logged commands per day, which doesn't sound as bad as the total line count in these histories. Despite the advent of LLMs I still need a lot of internal esoteric hacks that are relatively easy to find in these histories. Perhaps I should switch to using a centralized db that keeps the outputs as well and use them to finetune LLMs, but the ease of simple unix/Emacs tools operating with standard/simple history files always felt attractive.
Same here. Also included the host, timestamp, user, and current directory. Unifying history across multiple machines is very pleasant when you know you did something, but don’t even remember where you did it.
How long does it seem to take to open a new terminal and reach a working shell?
Your hardware might be fast enough that it's negligible, but several years ago (maybe 2018-19ish?) I noticed my shell startup was getting sluggish and traced a large fraction of it back to the time bash took to load many lines into history.
I still like keeping it all, so I wrote something to dump it into a database as I go but then truncate working histories to something shorter (500, I think).
The only time I've ever had shell startup time be significant is when loading fancy shell rc stuff (oh-my-zsh is the most infamous but there are many others)
Unfortunately I've hit some obscure bug with exactly when `HISTFILESIZE` is applied, and had some shells truncate all my history undesirably. There are also problems with reading history if the timestamp format changes.
Just put them in files, like month per machine/mpount per shell (bash, csh, etc). Then the shell will only load the current month's history. You can use grep of the files and easily narrow down what to search in.
I use bash within tmux heavily, and got irked that a command I run in one shell session is not immediately available as a history item in other concurrent shell sessions. So I wrote a history plugin based on bash-preexec to track everything to two files: a per-directory history file, and a global history file.
I have a bash function which does history selection for me, by popping an fzf selector to look at the directory-specific history file. A keybinding within fzf allows me to switch to looking at the global history file instead.
Boring commands such as “cd”, “pushd”, and a few others don’t get logged. The log entries are in json format and include basic metadata; directory, timestamp, and pid.
Within the fzf history picker, another keybinding allows me to edit whichever file I’m actively using. So if I fumble a few times to construct a command, then when I get it right I just pop into the selector, edit the relevant file, and remove the lines I don’t want to misfire on again.
I’m sure this is basically what atuin does; now that I’m at the spot where directories are the unit of history relevant history, maybe I should give that tool another look.
One really interesting upside of all of this is that I now tend to make “activity-specific” directories in my repos. For example, I have a “.deploy” directory at the git root of most of my projects. There are no files within that directory; but my tool creates ~/.bash.d/history/home/belden/github/company/project/.deploy.json which contains the history of ~/github/company/project/.deploy/
Empty directories are invisible to git, but for me the directory “has” content: the log of how I need to deploy this service or that service.
It’s a weird way to use my shell, and just sprung out of the initial grief: I shouldn’t have to exit a shell session to have its history become available.
> command I run in one shell session is not immediately available as a history item in other concurrent shell sessions
It's hard to understand how any other way is usable. If you have a dozen concurrent shell sessions, some running a long running process for example, and you had to restart one of them, you wouldn't be able to just Ctrl-C and go to the previous command. For me it's way better shell sessions live their own life but the history is accumulated to a single pool.
I definitely want to keep every single shell command I ever executed, including mistakes. And I want to track across machines, projects, etc.
Lately I have been using atuin for this. I have some gripes about interface defaults, and how the interactive search for history wipes out most of what was on my terminal view and causes a big "wtf was I doing again?" moment when it changes the terminal so much, but the tradeoff of shared history is very very much worth it for me.
Mistakes often have side effects, and sometimes I don't notice it until later. It's good to come back to a project after a week and try to see where unintentional effects may have come from.
I've been doing this for years, and my shell history is usually only a few lines/commands long when I start a new session. I like it that way. My browser's roughly the same - I always start firefox in "private browsing mode" (my system's default browser application is a shell wrapper around firefox that takes care of that), and I very consciouly use a non-private instance to "soft-bookmark" stuff that gets committed to my browser history. Actual bookmarks are for all the stuff that I consider really important not to forget about. A few sites (such as HN) have the privilege of me visiting them in non-private browsing mode while authenticated via cookes.
People might find that weird, and it sure is a tad inconvenient when random, low-priority websites cosplay Internet Fort Knox with very short and annoying MFA login timeouts and methods, but for me, the benefits (I'm less susceptible to tracking and can click on stuff with less angst that I might somehow leak data from my nigh-eternal browser session to some site or another, and I get to start relatively fresh in each session) outweigh the costs.
>My unusual habit is: turn off the history file completely, by putting the command ‘unset HISTFILE’ in my .bashrc. [...] All the shell history I allow myself is localised and short-term.
I don't do this, but I do disable my browser history on my work laptop so that I'm forced to author things I, and presumably other people, can actually stumble their way to later on.
I've been following a similar approach for a while. In bash, I have `HISTCONTROL=IGNOREBOTH` activated (or `setopt hist_ignore_space hist_ignore_dups` in zsh - most shells have an equivalent setting), and I just prepend most commands with a space. So only the more or less important stuff ends up in .bash_history or .zsh_history. This has the added nice effect that I don't accidentally trigger something destructive when I browse through my history.
I also spend most of my online life in incognito windows, at least for sites that don't absolutely require a login. This keeps my browser history clean from all the disposable pages that I only visit once, and I take care to do only the more meaningful stuff in a regular browser window.
The history file has ALL the lines prefixed by space. I curate it a lot, with most frequent/recent commands near the end. And I add a lot of comments to the file so things are easily searched. And search for things, for example /rsync to find the rsync quick backup snapshot lines so first comes one with -n to check what it would do, then if it looks OK I press 'n' (vi mode) and get the same line without -n, or another 'n' and I get the same without -n and with --delete.
Obligatory plug for Atuin, which is a Sqlite-based shell history tool. It logs shell commands alongside timestamps, the working directory, and the return value.
You can optionally sync your history to a server in encrypted form to keep a shared history across hosts. The server is extremely easy to self host.
As discussed in TFA, it’s easy to filter or scrub junk like `cd ~/Desktop` so it doesn’t pollute your history. You can also fuzzy search and toggle between commands run in your current session, commands run on your current host, commands run in your CWD, and commands run on all hosts.
It’s my single favorite piece of dev tooling, and has made my job + life far smoother and easier. Highly recommend.
Ooooh, I'll have to take it for a test drive next week. This looks good for personal use as well as keeping a fleet of production maintenance/monitoring instances sync'd.
Readit News
I have a folder marked executable with hundreds of small shell scripts, appended to $PATH. I never write long scripts. I sometimes write scripts that run other scripts but I try to avoid such dependencies. I do not use bash, I use NetBSD Almquist or modified Debian Almquist shell so all the scripts are highly portable. I can move a tarball of this folder from computer to computer, whether the OS is Linux or BSD. These scripts generally run faster than bash scripts.
It would be untenable to save all command line history since I spend every day in the shell in textmode (not x11) and only occasionally switch to a graphical environment. I sometimes use Almquist-based shells that have command line history removed. This forces me to write useful scripts.
A lot of cumulative shell knowledge is contained in that folder of scripts. These scripts have worked for decades unchanged. I think the Almquist shell may be my favourite interpreter. No matter what else I try I always come back to the shell language. There is something appealing about scripts that seem to work forever.
I don't understand this remark. What has x11 got to do with shell history? My bash appends all commands to .bash_history, immediately, regardless of whether it's run inside a terminal emulator or a VT.
Context switching between textmode and a graphics layer, like x11, is IME time-consuming and relatively janky. Whereas switching from a "terminal window" to some graphical program in a graphical environment like x11 is very quick and relatively smooth. Because I prefer textmode, I stay in textmode and I am not switching to a graphical environment very often, if at all. I am not using graphical programs. Consequently I accumulate an inordinate amount of command line history. Too much to save, IMHO.
As I mentioned, I do not use bash, I use a smaller, faster shell. I have no ".bash_history". And sometimes I use even smaller shells that have no command line history, in order to reduce program size.
Your hardware might be fast enough that it's negligible, but several years ago (maybe 2018-19ish?) I noticed my shell startup was getting sluggish and traced a large fraction of it back to the time bash took to load many lines into history.
I still like keeping it all, so I wrote something to dump it into a database as I go but then truncate working histories to something shorter (500, I think).
Unfortunately I've hit some obscure bug with exactly when `HISTFILESIZE` is applied, and had some shells truncate all my history undesirably. There are also problems with reading history if the timestamp format changes.
I use bash within tmux heavily, and got irked that a command I run in one shell session is not immediately available as a history item in other concurrent shell sessions. So I wrote a history plugin based on bash-preexec to track everything to two files: a per-directory history file, and a global history file.
I have a bash function which does history selection for me, by popping an fzf selector to look at the directory-specific history file. A keybinding within fzf allows me to switch to looking at the global history file instead.
Boring commands such as “cd”, “pushd”, and a few others don’t get logged. The log entries are in json format and include basic metadata; directory, timestamp, and pid.
Within the fzf history picker, another keybinding allows me to edit whichever file I’m actively using. So if I fumble a few times to construct a command, then when I get it right I just pop into the selector, edit the relevant file, and remove the lines I don’t want to misfire on again.
I’m sure this is basically what atuin does; now that I’m at the spot where directories are the unit of history relevant history, maybe I should give that tool another look.
One really interesting upside of all of this is that I now tend to make “activity-specific” directories in my repos. For example, I have a “.deploy” directory at the git root of most of my projects. There are no files within that directory; but my tool creates ~/.bash.d/history/home/belden/github/company/project/.deploy.json which contains the history of ~/github/company/project/.deploy/
Empty directories are invisible to git, but for me the directory “has” content: the log of how I need to deploy this service or that service.
It’s a weird way to use my shell, and just sprung out of the initial grief: I shouldn’t have to exit a shell session to have its history become available.
It's hard to understand how any other way is usable. If you have a dozen concurrent shell sessions, some running a long running process for example, and you had to restart one of them, you wouldn't be able to just Ctrl-C and go to the previous command. For me it's way better shell sessions live their own life but the history is accumulated to a single pool.
this can be solved in vanilla bash by adding these lines to .bashrc (be sure to remove other HIST* configs options from it)
additionally, you need to restart or close all the bash sessions to be sure no copy of bash with old settings existsInfinite history instead of a large limit:
Also ignore commands starting with a space: Enable history substitution to verify commands before running them: Write commands to history files before running the commands instead of after the commands finish: Or if you already have a DEBUG trap:Lately I have been using atuin for this. I have some gripes about interface defaults, and how the interactive search for history wipes out most of what was on my terminal view and causes a big "wtf was I doing again?" moment when it changes the terminal so much, but the tradeoff of shared history is very very much worth it for me.
AI Agents using tools also benefit from seeing past usage of the tools (what worked, what didnt) to help them inform future usage
People might find that weird, and it sure is a tad inconvenient when random, low-priority websites cosplay Internet Fort Knox with very short and annoying MFA login timeouts and methods, but for me, the benefits (I'm less susceptible to tracking and can click on stuff with less angst that I might somehow leak data from my nigh-eternal browser session to some site or another, and I get to start relatively fresh in each session) outweigh the costs.
I don't do this, but I do disable my browser history on my work laptop so that I'm forced to author things I, and presumably other people, can actually stumble their way to later on.
https://hiandrewquinn.github.io/til-site/posts/disable-your-...
I also spend most of my online life in incognito windows, at least for sites that don't absolutely require a login. This keeps my browser history clean from all the disposable pages that I only visit once, and I take care to do only the more meaningful stuff in a regular browser window.
When trying to modify a command, I remove the space, and when done I search for saving history and editing with '/_hi':
and voila.I also keep the similar lines aligned with spacing to note the differences quickly. (e.g. the -n above)
When exiting a shell and there's nothing interesting, history -r; exit
Also always I go to the end of the file before quitting vi so next time I'm where it last ended and before the appended new commands.
It takes work upfront and discipline but the everyday use of shell becomes very fast and clean.
HISTIGNORE='m:??:info STARHERE:info:[bf]g:exit:[bf]g %[0-9]:help STARHERE:date:cal:cal ????:exec env ENV\STARHERE' (replace STARHERE with an asterisk)
generally works for me in bash(1).
The line above set up automatic history ignoring for the colon-delimited shell globs; eg
fg %3
won’t be recorded to shell history, since it’s matched by one of the globs.
You can optionally sync your history to a server in encrypted form to keep a shared history across hosts. The server is extremely easy to self host.
As discussed in TFA, it’s easy to filter or scrub junk like `cd ~/Desktop` so it doesn’t pollute your history. You can also fuzzy search and toggle between commands run in your current session, commands run on your current host, commands run in your CWD, and commands run on all hosts.
It’s my single favorite piece of dev tooling, and has made my job + life far smoother and easier. Highly recommend.
https://atuin.sh/
There's no way I'm going to manually curate every command I type, I've got actual work to do.
considering to ask where my json bash history is located and how that works - I noticed Ctrl-r looked different a while ago.