Rather than re-write your scripts to store temp files into /dev/shm, you can just mount /tmp using the tmpfs file system and get the same benefit for all your programs. Some distros do this by default.
The relevant line from fstab is:
tmpfs /tmp tmpfs noatime 0 2
Now any program that writes to /tmp will be writing to a RAM disk, thus sparing unnecessary wear on my SSD.
I do mention this offhand in the article: "The existence of /dev/shm is a boon for me mostly because it means I never have to worry about whether /tmp is really RAM-based again."
Now you have to worry about whether you can access /dev/shm. Please encourage people to use supported interfaces instead of random voodoo (anything under /dev that wasn't there in 1995) for day-to-day tasks.
I did this for a while, but writing files to ram can be dangerous, since most things assume unlimited disk space. I noticed that updates would fail on machines that had 16 gigs of ram unless I logged out of my window manager and did it from the TTY. Took quite a long time to realize it was because of all the compiles writing to /tmp. Much easier to just let the SSD get used.
This is an unnecessary optimization, particularly for the article's use case (small files that are read immediately after being written). Just use /tmp. The linux buffer cache is more than performant enough for casual usage and, indeed, most heavy usage too. It's far too easy to clog up memory with forgotten files by defaulting to /dev/shm, for instance, and you potentially also take memory away from the rest of the system until the next reboot.
For the author's purposes, any benefit is just placebo.
There absolutely are times where /dev/shm is what you want, but it requires understanding nuances and tradeoffs (e.g. you are already thinking a lot about the memory management going on, including potentially swap).
It's true that with small files, my primary interest is simply not to wear on my disk unnecessarily. However I do also often do work on large files, usually local data processing work.
"This optimization [of putting files directly into RAM instead of trusting the buffers] is unnecessary" was an interesting claim, so I decided to put it to the test with `time`.
$ # Drop any disk caches first.
$ sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
$
$ # Read a 3.5 GB JSON Lines file from disk.
$ time wc -l /home/andrew/Downloads/kaikki.org-dictionary-Finnish.jsonl
255111 /home/andrew/Downloads/kaikki.org-dictionary-Finnish.jsonl
real 0m2.249s
user 0m0.048s
sys 0m0.809s
$ # Now with caching.
$ time wc -l /dev/shm/kaikki.org-dictionary-Finnish.jsonl
255111 /dev/shm/kaikki.org-dictionary-Finnish.jsonl
real 0m0.528s
user 0m0.028s
sys 0m0.500s
$
$ # Drop caches again, just to be certain.
$ sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
$
$ # Read that same 3.5 GB LSON Lines file from /dev/shm.
$ time wc -l /dev/shm/kaikki.org-dictionary-Finnish.jsonl
255111 /dev/shm/kaikki.org-dictionary-Finnish.jsonl
real 0m0.453s
user 0m0.049s
sys 0m0.404s
Compared to the first read there is indeed a large speedup, from 2.2s down to under 0.5s. After the file had been loaded into cache from disk by the first `wc --lines`, however, the difference dropped to /dev/shm being about ~20% faster. Still significant, but not game-changingly so.
I'll probably come back to this and run more tests with some of the more complex `jq` query stuff I have to see if we stay at that 20% mark, or if it gets faster or slower.
A couple things to consider when benchmarking RAM file I/O verses disk-based file system I/O.
1 - Programs such as wc (or jq) do sequential reads, which benefit from file systems optimistically prefetching contents in order to reduce read delays.
2 - Check to see if file access time tracking is enabled for the disk-based file system (see mount(8)). This may explain some of the 20% difference.
Hard disagree. Disk buffer cache is too eager on writes (which makes sense for the usual case), so temporary data written to a filesystem are almost always written to the medium. With several GBs of temporary data it easily could fill up internal SSD write buffers and make whole system choppy.
My use case is to use yt-dlp to download videos to ramfs, watch them and then delete. Before i switched to ramfs, the final pass of yt-dlp (where audio and video tracks are merged to one file) ordinarily caused the issue with choppy system.
> It's far too easy to clog up memory with forgotten files by defaulting to /dev/shm, for instance, and you potentially also take memory away from the rest of the system until the next reboot.
Aren't both solved by swapping?
Although I suppose on Linux, neither having swap, nor it being backed by dynamically growing files, is guaranteed.
This only stands for modern storage devices with a controller. For SD cards that don't have wear leveling, writing to the same region will make it die faster.
For decades now I have been using RAM (mfs, tmpfs) as "disk", i.e., boot from USB, no internal HDD, no internal SSD.^1 Entire OS fits in memory. When compiling OS userland there is simply no comparison; disk is painfully slow. I am not a gamer. I use RAM as a filesystem and workspace. I just got another computer with 64GB RAM. I will get 96GB or more when price comes down.
1. I use removable, external drives for anything I want to save long-term. No "cloud" storage.
Back to the 90s, when Amiga OS had a dynamically sized RAM disk by default and showed an icon for it in the GUI like for any other device. It even offered a persistant RAM drive.
For those interested in a windows equivalent, with bells and whistles, ImDisk [0] has a nice ramdisk creation utility with options such as allocating memory dynamically, ntfs compression, and more.
For the more venturous there is GPURamDrive [1] , not as many options, as it was made as a more of an experiment, but with gpu's adding more and more vram, why not?
An assumption I’ve been revisiting is if I really do need to be writing to disk all the time. I can’t remember the last time I actually had a crash or other event where I would have abruptly lost my work.
I’m wondering if I can completely hide away the detail where I can work exclusively in memory (even when I habitually save my code) and “reconcile” as some task I do before shutdown.
In fact, that doesn’t even feel necessary… I git push my day’s work a number of times. None of that needs a local disk. And 64GB of memory was surprisingly affordable.
You might be interested in Tiny Core Linux [0], then, especially piCore. After the initial read from the persistent media, everything is in RAM, the entire filesystem. You are working exclusively in memory until and unless you run a specific command to save everything you care to save back to that media again.
I have it running on a Raspberry Pi so that my already sparingly-used SD card's lifespan gets extended to, hopefully, several years. I have never seen the green writing LED light blink on without me specifically triggering it.
I primarily use it as a cronslave [1]. It has ~50 separate cronjobs on it by now, all wheedling away at various things I want to make happen for free on a clock. But if you live out of a terminal and could spend your days happily inside tmux + vim or emacs -nw, there's nothing stopping you from just doing this. Feels a lot like driving stick shift.
Things will still get written to disk eventually, it's just that fsync() returns instantly without actually doing anything. It's sometimes used in CI and similarly-ephemeral systems, and can produce a noticeable reduction in i/o.
Be warned, though, that it has that name for a reason!
I've been running my daily development laptop on 64GB of RAM for 1,5 years. My anecdotal experience is that no, you don't need persistent storage for most things. In fact, often it's in your way -- it clutters the system over time by causing configuration errors and weird undefined program states. When you can just reboot and all works again it's great. Never going back.
Speaking of RAM and disks, does anyone know what happens if you structure LVM volume such that there is a RAM based tmpfs as a front cache? Consistency issues aside, could it increase performance? Suppose I have an application that behaves such that it has very IO heavy write buffer of around 100GB with 10x or so NVMe backed storage for more rarely used data. Would
you do something else? The main problem I have currently is that the NVMes overheat occasionally from high IOPS which adds a lot of latency variance.
I use jupyter notebooks for similar purpose, with the Python kernel's memory keeping the state I want for some random stuff, and notebook being a recipe how to reproduce the calculation/processing.
some of my kernels been running for weeks, as I come back and redo or rework some processing.
the neat thing about jupyter notebooks, is you can interleave python one-liners with bash one-liners and mix them as you wish.
Readit News
The relevant line from fstab is:
Now any program that writes to /tmp will be writing to a RAM disk, thus sparing unnecessary wear on my SSD.Swap on an SSD isn't even that slow.
For the author's purposes, any benefit is just placebo.
There absolutely are times where /dev/shm is what you want, but it requires understanding nuances and tradeoffs (e.g. you are already thinking a lot about the memory management going on, including potentially swap).
Don't use -funroll-loops either.
"This optimization [of putting files directly into RAM instead of trusting the buffers] is unnecessary" was an interesting claim, so I decided to put it to the test with `time`.
Compared to the first read there is indeed a large speedup, from 2.2s down to under 0.5s. After the file had been loaded into cache from disk by the first `wc --lines`, however, the difference dropped to /dev/shm being about ~20% faster. Still significant, but not game-changingly so.I'll probably come back to this and run more tests with some of the more complex `jq` query stuff I have to see if we stay at that 20% mark, or if it gets faster or slower.
1 - Programs such as wc (or jq) do sequential reads, which benefit from file systems optimistically prefetching contents in order to reduce read delays.
2 - Check to see if file access time tracking is enabled for the disk-based file system (see mount(8)). This may explain some of the 20% difference.
My use case is to use yt-dlp to download videos to ramfs, watch them and then delete. Before i switched to ramfs, the final pass of yt-dlp (where audio and video tracks are merged to one file) ordinarily caused the issue with choppy system.
I've used /dev/shm extensively for large datasets and it's consistently been a massive speed improvement. Not sure what you're talking about.
Well, complain to whoever's mounting it wrong to fix it.
Aren't both solved by swapping?
Although I suppose on Linux, neither having swap, nor it being backed by dynamically growing files, is guaranteed.
1. I use removable, external drives for anything I want to save long-term. No "cloud" storage.
https://grimore.org/amiga/rad
It seems the advantage of this has been mostly forgotten.
For the more venturous there is GPURamDrive [1] , not as many options, as it was made as a more of an experiment, but with gpu's adding more and more vram, why not?
[0] https://sourceforge.net/projects/imdisk-toolkit/
[1] https://github.com/abesada/GpuRamDrive
I’m wondering if I can completely hide away the detail where I can work exclusively in memory (even when I habitually save my code) and “reconcile” as some task I do before shutdown.
In fact, that doesn’t even feel necessary… I git push my day’s work a number of times. None of that needs a local disk. And 64GB of memory was surprisingly affordable.
I have it running on a Raspberry Pi so that my already sparingly-used SD card's lifespan gets extended to, hopefully, several years. I have never seen the green writing LED light blink on without me specifically triggering it.
I primarily use it as a cronslave [1]. It has ~50 separate cronjobs on it by now, all wheedling away at various things I want to make happen for free on a clock. But if you live out of a terminal and could spend your days happily inside tmux + vim or emacs -nw, there's nothing stopping you from just doing this. Feels a lot like driving stick shift.
[0]: http://tinycorelinux.net/
[1]: https://hiandrewquinn.github.io/til-site/posts/consider-the-...
Things will still get written to disk eventually, it's just that fsync() returns instantly without actually doing anything. It's sometimes used in CI and similarly-ephemeral systems, and can produce a noticeable reduction in i/o.
Be warned, though, that it has that name for a reason!
some of my kernels been running for weeks, as I come back and redo or rework some processing.
the neat thing about jupyter notebooks, is you can interleave python one-liners with bash one-liners and mix them as you wish.