Reading this article left me wondering just where the 200 million tags this guy needs are supposed to come from. Manual curation?! Automatically derived by file extension? file headers? what is the cost of opening a file, parsing its filetype, comparing against a reference, writing it to a database, etc. How is that cheaper than current indexers (which all seem to work fine btw)?
I rarely waste effort trying to remember filenames in the first place, much less needing some expensive tag curation to locate files. I simply use a bit of discipline organizing the directory structure(s). If I do ever need to actually search for something, it will be constrained to a narrow subset of directories and ignore the other 199.9 million files or whatever.
Moreover, I just don't have the problem of searching for filename fragments to begin with. Nor do I see a reasonable way to use a whole host of powerful unix techniques with a whackadoodle tiny tags filesystem. Or the need to produce a list of 20 million images in 2 seconds. What use would that be anyway? I'm not going to read a list like that - I'm going to operate on it.
Please correct me if I'm wrong, but the versatility of `find` is far more powerful if you actually need to handle/sort through that many files, and something like `fzf` probably curtails all these complaints in the first place.
If I had a penny for every time someone on HN responds with something like this - "just become more disciplined and you don't need X" - I'd be a millionaire. Doesn't matter what it is, type systems, memory safety, a better UI for Git… there's always someone ready to chime in with how their workflow means these problems don't happen, or, even better, asking the question why would anyone need this?
Yes, why would anyone need better search or a faster, easier to organise file system? I can't think why.
A better search and tagging can be valuable tools. But no matter how good search gets, it will not stop users from putting files untagged into one big junk folder.
Being able to think about how to order your files is a fundamental skill in this day and age and doing this on a big scale does indeed require discipline.
IMO it is just a false hope to think tags would help with the root cause of a lack of care about the data.
I’m all for optimising usability but there comes a point where one over optimises. While I love the app-orientated design of Android and iOS (ie you share data between apps rather than apps sharing a file system), they are effectively toy OSs for toy devices. Sure some people are hugely productive on them these days but they’re the exception rather than the norm. Whereas I depend on a file system to organise my data. In fact the file system is directly interpreted as name spacing in a number of different programming languages.
I get that most people are either too lazy or too technologically inept to own a computer but this race to the bottom to support everyone who doesn’t give a crap has to end somewhere. You see it with Windows UI in Win11 removing popular options so the designers can streamline the UI. You see it with this article too. Some people are always going to struggle simply because they are forced to use a computer day in and day out rather than them wanting to use it. But designing a unified system pandering for them but servicing everyone just makes the experience shit for those of us how genuinely know how to use a computer and depend on these features.
To use a car analogy (because for some reason people love comparing cars to computers…) I have no issue with track cars being sold without air con, a radio, etc because they’re a toy not a tool. So you optimise for that single purpose: racing on the track. But I sure as hell want the kitchen sink thrown into my family car.
I sometimes wonder if the problem isn’t computers but rather our assumption that everyone should be able to use a computer without training. If your job depends on using a file system correctly then you should be trained on that in exactly the same way that you’re taught how to use any of the specialist applications. In fact pre-computers, companies did exactly that with training their staff in how the filing system works!
By analogy: if you were to store all your paper copies of bank statements, bills, mortgage papers, etc... Would you just dump them in one big pile in the middle of your living room, or would you sort them into vaguely themed folders to impose some organisation? Hierarchical filesystems are valuable to those that want to organise data in ways that tags can only emulate.
Having every file pollute a global namespace seems to require more discipline than the current hierarchical system where you can easily copy a directory tree without having to worry about breaking something else.
That is the main problem with these so called "solutions", they usually take more effort and discipline than the problem they originally set out to solve. The right solution is just to learn the original system properly rather than trying to invent an even worse way to work around it.
The article proposes to replace the current file system approach (which works just fine for me, by the way, thank you very much) with something different to solve a problem that I (just like the post you reply to) have no interest in.
Better search? Sure! Improved speed of storage and retrieval? Great! But either show that it is not degrading current functionality or be ready for pushback from people suspicious that their current setups will break. My 2c.
don't you think a tag system would require even more discipline?
what do you think happens if you make a mistake with your tags and/or there are typos in the filename? With a directory structure, you can navigate to the location and see the list of items to quickly identify what you were looking for. It is far more forgiving when it comes to poor organisation or mistakes. With a pure tag system, a file with the wrong name/tags is pretty much forever lost.
Because such a system would entail its own drawbacks, such as a larger CPU load or a more fragile disk organization, whilst most people wouldn't really need it.
This is obviously hyperbole and you're well aware you haven't read 100 million Hacker News comments, let alone that many with exactly the same basic message.
But I was curious what this might be equivalent to in terms of time investment. A quick style guide check recommends 15-20 words per sentence for English language written communication. Assuming the low end of that, and minimal single-sentence comments, that is still equivalent to reading the entire 14-book Wheel of Time series, which tends to take most people several years, 369 times.
I guess the point is, the proposed system wouldn’t actually be easier to organize. The metadata that would make searching so easy is what’s missing. But a new data structure doesn’t solve for the missing metadata. And without that extra metadata, searching would not actually be improved.
i try to organize my stuff, but sometimes i forget where in the organization i put something. then a brute-force search helps. if i keep good directory and filenames, then locate will do the trick. once i found one item, any related other things are usually nearby.
Google photos is pretty amazing. I enter a search for "car" and immediately can see the photos os several of the cars I've owned over the years.
One day I needed to remember when I had travelled to a certain city, searched on my Google photos and it instantly showed the photos I took in the city, including the exact dates.
Yes, I know letting Google know all about my life like that through photos may not be the greatest idea... but wow, does the photo search work nicely?!
> It sounds somewhat like “gmail for files” which is … problematic because email search works well enough because it’s relatively rarely done.
Gmail does not work for me.
As someone in IT I get some number of automated messages (e.g., cron). With Gmail all I can do is tag them and have a "folder" / view of just those tagged messages. But they also pollute my Archive 'folder' as well.
But I do not want them there, because they are not a priority generally, and they pollute search results.
I want an actual separate folder to file these messages in that is out of the way so as not to pollute the rest of the namespace.
And that’s the issue. The status quo may be the best choice for the lowest common denominator. But some power users could get much more out of something with a different approach. You can’t force a one-size-fits-all ontology onto the masses.
People need to wake up and realize that not all software technologies need to be popular to be successful or useful. It seems people around here assume this without even thinking about it first.
My response to these types of proposals is “just imagine that folders are tags and each level of hierarchy is a tag, symlink for multiple tags.”
It’s funny because the author just proposed a different, I think worse due to novelty and minimal benefit, organizing hierarchy.
I think Apple has a decent approach where their spotlight indexes very well (I use hit command+space and the first letter or two instead of navigating finder), and they support tagging files.
When importing files into Didgets, the program automatically gathers information from the source file system and attaches specific tags to each file. For example, the file name is attached as a 'name' tag. Each folder name in its path is attached as a 'folder' tag. The file extension is attached as an 'extension' tag. In addition a SHA1 hash is created from the data stream and attached as a tag. You also imported them by dropping the files or folder onto a 'drop zone' on the create tab in the GUI. Any tags attached to that drop zone are also automatically attached to any file dropped on it. So dropping 100 photos on the 'My Wedding' drop zone might attach the tags 'Event = Wedding' and 'Year - 2022' to every photo. Searches for files that have a tag 'Folder = Microsoft' would find every file that had 'Microsoft' as a folder anywhere in its path.
> it will be constrained to a narrow subset of directories and ignore the other 199.9 million files or whatever.
I think this is a vastly underrated point. I am usually not interested in searching the majority of files on my filesystem. I can't remember the last time I needed to search through system files for normal computer use reasons.
I also think the author completely skips over how to handle related files. If my application needs to load a library, how does it find the file to use? If it's by name, how are name clashes handled? I suppose it could be by tag, with built-in tags, but then you won't be able to change the tags without having to change configs or the binary itself.
The core problem at keeping the files organized is that unless you are dealing with a stream of effectively pre-tagged files, tagging/categorizing/grouping emerges after sufficient number of files arrive. Therefore organizing is proactive
What this boils down to is that he thinks a flat namespace (tags) offers advantages over hierarchical namespaces (tree). They really don't. Once your tag space grows you will start to struggle with naming, and path-like structures (nested namespaces) start to creep back in. And you are right where you started: paths.
The treatment of immutability is too superficial to make any sense of so I don't know what the author is imagining. Ted Nelson has evolved some ideas on this for decades that might be worth knowing about. Some of which have kind of come to pass (if you squint and look at how non-destructive editing tools for video and audio work, for instance). However, very little of Ted's thinking has ever been burdened by usable implementation.
The concept of having multiple references to the same file already exists. So what he proposes can be realized with existing file systems just by introducing a different naming scheme and making extensive use of sym-/hard-linking.
Yes, a lot of file systems will have terrible lookup and traversal performance, but that problem exists in an orthogonal universe and can be solved. Is, indeed solved, in some fileystems if the marketing blurb doesn't lie.
If you think about how you would realize this using existing filesystems, by organizing them differently, the concept isn't as sexy anymore. Because it doesn't really involve a lot of new stuff and you start to see the inconvenience of having to cope with both novelty and problems you didn't have before.
The problems someone like me wants solved in filsystems are entirely different, and aren't so much about filesystems as it is about how you make the functionality useful to applications.
For instance, there are filesystems that offer snapshot semantics. Including COW-snapshots. This would be useful whenever applications need to do be able to roll back changes, switch between states, do backups while being live etc. Yet I know of no language which has snapshot as part of the standard OS interface. So people generally don't write application that take full advantage of what the underlying system offers.
Path based file systems take advantage of natural semantics we use for navigation. There is a wonderful overlap between how you navigate the real world, and how you navigate a hierarchical file system.
I have never (never ever ever) seen a tag based system actually work once you have large amounts of files and tags - Tags are manual, often duplicated with slight name changes or variations, hard to discover, and literally worse than a folder hierarchy for discoverability in almost every way.
Tags can be nice to have - but only if I also have a path. Otherwise they are utterly inferior.
Tags, being one of the most basic implementations of boolean retrieval, tend to suffer from feast-or-famine a lot, at least in my experience. Once you introduce hierarchical tagging, people will just use them like folders with each item having 1.0x tags on average.
The Didget system was designed to allow both a hierarchical folder tree as well as tags attached to individual files and folders. The tags do not replace the hierarchy unless you want them to.
If you have never ever seen a system that actually works, then maybe you should put Didgets to the test. I created 20 million files in it and attached an average of 100 tags to each one. Each tag had a value randomly picked among 1000 choices. Queries to find all files with a certain tag (e.g. Tag_134 = Value_875) each completed in less than a second.
>I have never (never ever ever) seen a tag based system actually work once you have large amounts of files and tags
If you've ever done online shopping, you probably have. For example, try going on Amazon or Newegg and searching for a GPU. You're shown a sidebar where you can easily filter results be certain tags such as: brand, price range, memory size, core count, in stock, energy star certified, free shipping, etc.
I think one of the problems is that there are many datasets where objects belong to multiple hierarchies and different hierarchies are more efficient for different tasks. For example I work with medical imaging data. Typically that gets organized around the DICOM object models which even defines multiple structures. Typically that's around the patient/encounter model and slight variations of that and data is stored in a database called a PACS, but working with PACS is extremely difficult because DICOM is optimized for clinical use cases. But there are other ways of organizing the data that are more efficient for other tasks for example for quality improvement or assurance or process monitoring. In fact different users of the data are likely to want views based on different hierarchies. Some software expects certain data layouts etc. There are some efforts to standardize file hierarchies and naming for certain tasks, but perhaps you're not doing that task. You can do things like symbolic links, but trees of symbolic links end up being super fragile in my experience and they're not particularly well supported on some operating systems.
I don't think the filesystem is the right place to solve these types of problems. I also think if you try to make the kind of filesystems that solve these problems you'll invariably end up with entirely new problems you really don't want (complexity, performance issues, unclear semantics etc).
You usually want more flexibility and control over how your data is projected to storage. (Say for instance you run out of storage and the scheme doesn't have any way to split the data across multiple filesystems). And you really want integrity constraints that stop you from pointing into thin air and help you clean up. Occasionally you also need to have the concept of identity (how do you refer to a given entity directly) without it being part of a projection that may have stopped existing - like a tag being deleted).
I think this is just a general lack of software for rare cases like clinical data?
I mean, it sounds like what you have described is solved for photos. I use digikam photo manager, and it automatically discovers all the photos on multiple the volumes, and supports showing files by date, location, tags, path -- whatever you like. And it is not very fragile at all -- it identifies photos by metadata-excluded hash, so moving the photos around does not break the links.
And back in Winamp days, it had an MP3 database which had basically the same properties.
I have no doubt that users could use more and better organization, but it seems like UX problem, not a filesystem one.
Couldn't that still be stored in a traditional folder hierarchy, but all in one folder and keep track of the tags related to file names in a sqlite database?
I'm not sure any of this really addresses the question - which is how do people really use files.
IME I have a number of live projects which can contain various numbers of source files, images, web links, PDFs and other documents, text files, and so on.
Then there are a number of files I access regularly which may not be associated with a project (like favourite music).
Then there's a mountain of data which is just there in case I ever need it. It includes backups of old projects, documents, music and art I keep because I think it's interesting but haven't read yet, web links that are filed and then (sadly...) forgotten, and so on.
I don't know how typical this is, and it doesn't matter. Because neither a tag based nor a tree based system address the real issue - which is designing a custom file workflow that collects related references of all kinds, doesn't confuse working data with long-term storage, allows off-site backups, allows collaboration, supports versioning on demand, and also makes it easy to find things.
I suppose all of that means some kind of process API which does a lot more than file.open() and file.close().
It could be built on tags, it could be built on trees, it could be built on some combination. Or on something else entirely.
The implementation matters a lot less than a set of available features which streamline common tasks in some fairly standardised and effective way.
> The concept of having multiple references to the same file already exists.
It does and it's really bad IMO. The author's suggestion of unique identifiers though would introduce all sorts of new problems, primarily it would make the transparency problems of existing systems worse.
Most applications rely on the location of a file, relative or otherwise to load data (e.g. configuration). That reliance is exploited by software engineers to implement configuration swaps, event processing, and many other features. Referencing files based on UIDs, or a series of tags that aren't guaranteed to be unique or not known to be off limits to regular users, would introduce all manner of complications.
I could also see it being terribly easy to introduce bugs loading files using filtered tags. Would applications need to have relative tags to mitigate these problems? Having unique paths works both as a filter for the user and an encapsulation for a system that allows you to localize your concern. Without that encapsulation by default, you will be spending a lot more time and concern dealing with files and tags.
There is a lot of stuff that should NEVER appear in a mixed view. (Google is full of examples of that.)
Tag coulds and other meta data can still be very useful. The challenge is creating useful tags/meta data automatically. For example a time stamp for every modification or a label for every application that created, modified or loaded the file. Perhaps even the applications you were using when the file was created/modified and the file names of the files loaded into the application. Train some AI to show you files you probably want given your current activity.
And yes, even with a layer on top of the FS to provide an abstraction (as an API, for instance) so you can build shells and applications, a tag + search based system would quite possibly be tedious to use.
I also don't think 64 bit ints provide a good way to definitively name things. Most people can't make sense of a list of 10 ints, but they will be able to remember at least where to look if you give them full paths.
> but it's tedious and you can't have multiple files hardlinked to the same path.
Little trick I learned to help sort images: Make a copy of the file in as many locations as you like, then run something like borg backup. One file, hardlinked in as many directories as you want.
The problem of hierarchical file systems and data location is a really old problem that has had many implementations (I even tried building one many years ago).
The pattern that has worked out really well for me, is to just organize my data into specialized collections, and not worry too much about the underlying filesystem.
I can't believe how much trouble I went through trying to find the filesystem that could do everything.
I mostly use git-annex, and various tagging systems, or just git repos. Now my data is much more portable and flexible. None of these tools are perfect, but I'm using tools that are mostly good at the job.
Whatever problem you are trying to solve, you probably don't need to solve it for your entire filesystem.
Systems that try to get rid off the "files & folders" abstraction of a file system tend to have much worse usability, in my opinion. I have an iPad Pro, and the lack of true file system abstractions is so painful. Every app has its own way to store and retrieve data, there's almost zero interoperability and it's super painful to copy, paste and move stuff around (I know it has gotten better but it's still so much worse than on any desktop OS).
I'm all for enriching the concept of a file system with additional meta-data (in fact many files do that) but I don't think that needs to happen in the file system itself. For example, software like Picasa leveraged meta-data contained in files to provide a new way of interacting with large number of photos. The author basically proposes to put such functionality directly into the file system, but I'm really not sure if that's a good idea. Right now it's easy to move files between different systems, e.g. from Mac to Windows or Linux. If file systems become meta-data management databases that will become much more difficult.
> Systems that try to get rid off the "files & folders" abstraction of a file system tend to have much worse usability,
IMHO it's because those systems just simplify, but don't move very deep in the space they opened up. If you don't offer power, then it's irrelevant which system you offer, they will all suck fast.
> Every app has its own way to store and retrieve data, there's almost zero interoperability and it's super painful to copy, paste and move stuff around (I know it has gotten better but it's still so much worse than on any desktop OS).
Which is kind of a surprise, I would think Apple would be interested to unify that space and offer a good user experience.
> Right now it's easy to move files between different systems, e.g. from Mac to Windows or Linux. If file systems become meta-data management databases that will become much more difficult.
Theoretically, it could be solved by using a meta-file-container. Something like a tar-container, which contains a file for meta-data and the actual content. We have this with specialized container-formats in media and office-filetypes. Making a universal format which would work equally well for any kind of file type could solve this problem of interoperability. This would even open up ways to improve files without changing them directly. Like adding subtitles or notes to a file, by just adding it to the container, not the file itself.
App sandboxing on iOS is supposed to be a security feature, I think? But it makes it impossible for apps like Obsidian to work with apps like Dropbox, which benefits Apple; they force people to use iCloud.
macOS has so much of the prep work for this kind of thing, but Apple has completely dropped the ball on the UI.
Spotlight parses and indexes all the existing metadata in your files (music ID3 tags, photo EXIF tags, etc - run `mdls` on a file in a terminal to see all the stuff it's extracted) and this could all be used to make some pretty powerful UIs, but all Apple has done is made one very handy universal search UI, and then a very poorly designed specific search UI, and then made stored searches (which are also useful, but limited in practicality by how bad the UI to create them is)
I have 394,175 photos and videos that I have personally taken since 1997. They are organized by a simple hierarchical system in 5,356 folders.
D:\masterarchive\source\YYYY\YYYYMMDD\photo file name
If I want to find a person, in a photo, I've used Google Picasa (when it was an offline product) and lately digiKam to do face matching, and tagging them with IPTC metadata tags in the photo files. Thus they survive moves across filesystems, etc.
I'm up for seeing alternatives, but there's a very high bar to clear here. People have been using directories and file storage since the middle ages.
You, as a data point (and I, and I imagine many people on HN), are strong evidence of the rule "sufficiently motivated and clever end users will always find a way to do what they want, regardless of the interface".
But that isn't really saying anything about if the interface sucks, or could be improved, just that you're motivated and clever enough to find a good and scalable solution for what you want to do given the limitations of the interface.
I do the same. This is the only way to organize things, by date.
I do the same with documents. I don't even want to think about categories, ontologies are always wrong. But today is 2022-02-24, no two ways about it. It's automatic, there's no need to think or decide anything, so it's not a big deal, you just do it. You can't make a mistake.
The thought of needing to properly tag every document I file is enough to make that a task I want to postpone. So it wont get done. That's a worse filesystem right there, because it doesn't exist.
But look at the path components: they're not organizing things just by date. The files are first organized by purpose (long-term storage), then by origin, and only then by date.
So they've already made the decision to commit the files to long-term storage, and to keep the original photos separate from subsequent edits, and to keep them separate from other image sources (e.g. downloads). That "tagging" required very little effort because they could just navigate to the existing tag in the filesystem, and put the new files there.
> This is the only way to organize things, by date.
I am terrible at dates - if I had to find photos by date I'd never find them. For anything older than a month that isn't on a known anniversary like a birthday, 90% of the time I find the photos using the map view in iCloud Photo Library. If I was limited to a filesystem view, my photo library would be far less useful.
What you've done here (and it's no bad thing given filesystems!) is define dates as an ad hoc index over a primary key which is the pair (date, filename).
What I'd like, personally, is a way to expose any sortable EXIF data as a 'filesystem', for example `~/Photos/Longitude/122-123/Latitude/36-37/`.
Most of the commentary on a tag system seems predicated on the idea that we can't derive a large volume of tags automatically from the circumstances and provenance of the data. For instance, instead of a Downloads folder (per se, it would be a view) we could have a "downloaded" tag, which could have "downloaded-by" = "Chrome" and "downloaded-from" = "https://example.com/a-url/".
That's a lot more useful to me than a Downloads folder, especially if those tags endure when I add further metadata of the "canonical folder" variety, also known as "moving" the file.
I actually started coding a way to automatically extract EXIF data from all imported jpeg files and automatically attach them as tags to the file. That way you could search for photos taken with a specific camera, or only photos where the flash was used, or location data (if your camera had GPS), etc.. I just haven't had the bandwidth to get back to that feature and finish it.
People have been using tables of contents and indices for books and other printed materials for a long time, but those are redundant in the era of CTRL-F. At best, they're useful only as an adjunct to searching digital content.
Likewise, imposing archaic methods of organisation on modern storage is sub-optimal. The argument is to move toward something that makes more sense given the capabilities of the medium.
What tags and searching do not provide is context. One affordance that putting files into a folder provides is reminding oneself that when I look at file A, I should probably remember about file B too. Later I may even forget about the existence of B, but when I go searching for A I'm going to see B as well.
Search and tagging are not contradictory to cataloguing. They're complementary.
It's also not true that old media didn't have any search facilities. Old technical books would each have an index of keywords at the end. That's search, just analog and requiring a bit more work from the publisher. This index didn't make the table of contents redundant.
There are two absolute genius inventions in computers so good and timeless that the sliced bread pales in comparison like a stupid troll comment on HN.
1. The keyboard
2. The hierarchical filesystem
Everything and anything else in input devices and data storage builds on these and the best solutions ever always are going to augment these, never replace them.
A good tag system will build on top of a filesystem and coexist with it, and offer value like stupidly fast search. Anything else will be lucky to survive a weekend of dubious fame on twitter, or up to a few months if you actively market it.
I've never seen a way to organize files that feels like it prioritizes files based on how much they mean to the user.
By that I mean pictures I take, papers I write, things I really wouldn't want to lose, vs 10,000 random system files.
For downloaded files sometimes the history of when it was downloaded, and from where, is almost as important as the contents.
Backups from old computers, and old phones start to pile up, and the chaos of trying to find that picture you took 3 phones ago, or the notes you took, or the recording you made, or the pdf you downloaded, or that code you wrote, or that map you made, is a real pain.
Digital clutter is one of my biggest problems.
I really need a good way to deduplicate and organize ALL my digital stuff. Tags might play a role, but I don't think they quite solve the problem.
It feels like the author didn't like tree based file structures? The provided software in the screencast remembers me of iTunes. Which I dismiss because it doesn't provide a logical *tree like structure*. And it is not a filesystem replacement either, it is a database which adds a lot complexity and hides actually data. Furthermore this assumes someone is maintaining the metadata (remember the MP3-Taggers?) instead of the files. Metadata itself is useful but the creator of the file should add it not the user. Regarding file manipulation the proven answer are file permissions but I think CGROUPs are the flexibel, modern approach.
Because I'm seeing "Windows Explorer" in background:
Windows Explorer has degraded in recent years, it is even hard to open your "home directory" and the UI is confusing. Look at the one from NT 4.0 which was much more close the fulfill the task.
And Apple:
I think the regret nowadays the howl iTunes? But instead the pushing hard on apps which contain the data. Now you have to look always into a single app and uses it facilities to retrieve a file. Android failed here, too. But using iOS is hard.
Windows clearly doesn't want people to GET to their home directory, for some reason. That seems goofy. If people don't understand that $user contains the rest of those folders (Documents, Downloads, Pictures, etc) they'll never be able to navigate on their own. That's bad.
In a sane tool that features an address bar, clicking any given directory would show, in the address bar, the path to that location. WinExp only rarely does this. If you click on, say, Desktop, it shows you This PC > Desktop, implying a relationship that is incorrect. Getting to your home folder without typing requires you to start with C: and drill down, which is objectively insane.
Even MORE bananas is that if you start at C: and drill down to Desktop, you DO get the correct path in the address bar. But if you then make a WinExp shortcut of that location, it goes back to the other behavior. WTF.
> Windows clearly doesn't want people to GET to their home directory, for some reason. That seems goofy. If people don't understand that $user contains the rest of those folders (Documents, Downloads, Pictures, etc) they'll never be able to navigate on their own. That's bad.
Yes. Windows prevents users nowadays from understanding a straightforward thing, file-systems. I mean it was always a bit clumsy with A:, C: and [D-Z]: and the weird desktop metaphor harmed as well.
Now I'm looking at the often criticized GNOME and the actually venerable Nautilus. They got it! All below / and in addition devices are directly usable (actually still somewhere below /run). The location bar reflects the current position. The desktop was removed because it never fit into a computer and the file system.
Some actions of Google within Chrome are also questionable. "There is not address entry field" because we don't want you to understand how the web is structured. What? File-Systems are a simple thing, hierarchical. And guess what, the web is similar. Compared to "right click", "double click" and it's new friends "long press", "hard press" and "swipe from somewhere" and "guess what the voice assistant can interpret".
Readit News
I rarely waste effort trying to remember filenames in the first place, much less needing some expensive tag curation to locate files. I simply use a bit of discipline organizing the directory structure(s). If I do ever need to actually search for something, it will be constrained to a narrow subset of directories and ignore the other 199.9 million files or whatever.
Moreover, I just don't have the problem of searching for filename fragments to begin with. Nor do I see a reasonable way to use a whole host of powerful unix techniques with a whackadoodle tiny tags filesystem. Or the need to produce a list of 20 million images in 2 seconds. What use would that be anyway? I'm not going to read a list like that - I'm going to operate on it.
Please correct me if I'm wrong, but the versatility of `find` is far more powerful if you actually need to handle/sort through that many files, and something like `fzf` probably curtails all these complaints in the first place.
Yes, why would anyone need better search or a faster, easier to organise file system? I can't think why.
Being able to think about how to order your files is a fundamental skill in this day and age and doing this on a big scale does indeed require discipline.
IMO it is just a false hope to think tags would help with the root cause of a lack of care about the data.
I get that most people are either too lazy or too technologically inept to own a computer but this race to the bottom to support everyone who doesn’t give a crap has to end somewhere. You see it with Windows UI in Win11 removing popular options so the designers can streamline the UI. You see it with this article too. Some people are always going to struggle simply because they are forced to use a computer day in and day out rather than them wanting to use it. But designing a unified system pandering for them but servicing everyone just makes the experience shit for those of us how genuinely know how to use a computer and depend on these features.
To use a car analogy (because for some reason people love comparing cars to computers…) I have no issue with track cars being sold without air con, a radio, etc because they’re a toy not a tool. So you optimise for that single purpose: racing on the track. But I sure as hell want the kitchen sink thrown into my family car.
I sometimes wonder if the problem isn’t computers but rather our assumption that everyone should be able to use a computer without training. If your job depends on using a file system correctly then you should be trained on that in exactly the same way that you’re taught how to use any of the specialist applications. In fact pre-computers, companies did exactly that with training their staff in how the filing system works!
That is the main problem with these so called "solutions", they usually take more effort and discipline than the problem they originally set out to solve. The right solution is just to learn the original system properly rather than trying to invent an even worse way to work around it.
Better search? Sure! Improved speed of storage and retrieval? Great! But either show that it is not degrading current functionality or be ready for pushback from people suspicious that their current setups will break. My 2c.
what do you think happens if you make a mistake with your tags and/or there are typos in the filename? With a directory structure, you can navigate to the location and see the list of items to quickly identify what you were looking for. It is far more forgiving when it comes to poor organisation or mistakes. With a pure tag system, a file with the wrong name/tags is pretty much forever lost.
But I was curious what this might be equivalent to in terms of time investment. A quick style guide check recommends 15-20 words per sentence for English language written communication. Assuming the low end of that, and minimal single-sentence comments, that is still equivalent to reading the entire 14-book Wheel of Time series, which tends to take most people several years, 369 times.
Sounds good, but unfortunately the article is not proposing any of that.
So, counter question: of course you need that, but how would the proposal of the article actual do anything about that?
I like the idea of having a queryable filesystem, but I wouldn't want that as a complete replacement of the directory structure.
One day I needed to remember when I had travelled to a certain city, searched on my Google photos and it instantly showed the photos I took in the city, including the exact dates.
Yes, I know letting Google know all about my life like that through photos may not be the greatest idea... but wow, does the photo search work nicely?!
I suspect a system like this would work, but the tags would eventually be used by many as a way to badly implement a hierarchy.
Gmail does not work for me.
As someone in IT I get some number of automated messages (e.g., cron). With Gmail all I can do is tag them and have a "folder" / view of just those tagged messages. But they also pollute my Archive 'folder' as well.
But I do not want them there, because they are not a priority generally, and they pollute search results.
I want an actual separate folder to file these messages in that is out of the way so as not to pollute the rest of the namespace.
And that’s the issue. The status quo may be the best choice for the lowest common denominator. But some power users could get much more out of something with a different approach. You can’t force a one-size-fits-all ontology onto the masses.
People need to wake up and realize that not all software technologies need to be popular to be successful or useful. It seems people around here assume this without even thinking about it first.
It’s funny because the author just proposed a different, I think worse due to novelty and minimal benefit, organizing hierarchy.
I think Apple has a decent approach where their spotlight indexes very well (I use hit command+space and the first letter or two instead of navigating finder), and they support tagging files.
I think this is a vastly underrated point. I am usually not interested in searching the majority of files on my filesystem. I can't remember the last time I needed to search through system files for normal computer use reasons.
I also think the author completely skips over how to handle related files. If my application needs to load a library, how does it find the file to use? If it's by name, how are name clashes handled? I suppose it could be by tag, with built-in tags, but then you won't be able to change the tags without having to change configs or the binary itself.
The treatment of immutability is too superficial to make any sense of so I don't know what the author is imagining. Ted Nelson has evolved some ideas on this for decades that might be worth knowing about. Some of which have kind of come to pass (if you squint and look at how non-destructive editing tools for video and audio work, for instance). However, very little of Ted's thinking has ever been burdened by usable implementation.
The concept of having multiple references to the same file already exists. So what he proposes can be realized with existing file systems just by introducing a different naming scheme and making extensive use of sym-/hard-linking.
Yes, a lot of file systems will have terrible lookup and traversal performance, but that problem exists in an orthogonal universe and can be solved. Is, indeed solved, in some fileystems if the marketing blurb doesn't lie.
If you think about how you would realize this using existing filesystems, by organizing them differently, the concept isn't as sexy anymore. Because it doesn't really involve a lot of new stuff and you start to see the inconvenience of having to cope with both novelty and problems you didn't have before.
The problems someone like me wants solved in filsystems are entirely different, and aren't so much about filesystems as it is about how you make the functionality useful to applications.
For instance, there are filesystems that offer snapshot semantics. Including COW-snapshots. This would be useful whenever applications need to do be able to roll back changes, switch between states, do backups while being live etc. Yet I know of no language which has snapshot as part of the standard OS interface. So people generally don't write application that take full advantage of what the underlying system offers.
I have never (never ever ever) seen a tag based system actually work once you have large amounts of files and tags - Tags are manual, often duplicated with slight name changes or variations, hard to discover, and literally worse than a folder hierarchy for discoverability in almost every way.
Tags can be nice to have - but only if I also have a path. Otherwise they are utterly inferior.
If you've ever done online shopping, you probably have. For example, try going on Amazon or Newegg and searching for a GPU. You're shown a sidebar where you can easily filter results be certain tags such as: brand, price range, memory size, core count, in stock, energy star certified, free shipping, etc.
You usually want more flexibility and control over how your data is projected to storage. (Say for instance you run out of storage and the scheme doesn't have any way to split the data across multiple filesystems). And you really want integrity constraints that stop you from pointing into thin air and help you clean up. Occasionally you also need to have the concept of identity (how do you refer to a given entity directly) without it being part of a projection that may have stopped existing - like a tag being deleted).
I mean, it sounds like what you have described is solved for photos. I use digikam photo manager, and it automatically discovers all the photos on multiple the volumes, and supports showing files by date, location, tags, path -- whatever you like. And it is not very fragile at all -- it identifies photos by metadata-excluded hash, so moving the photos around does not break the links.
And back in Winamp days, it had an MP3 database which had basically the same properties.
I have no doubt that users could use more and better organization, but it seems like UX problem, not a filesystem one.
IME I have a number of live projects which can contain various numbers of source files, images, web links, PDFs and other documents, text files, and so on.
Then there are a number of files I access regularly which may not be associated with a project (like favourite music).
Then there's a mountain of data which is just there in case I ever need it. It includes backups of old projects, documents, music and art I keep because I think it's interesting but haven't read yet, web links that are filed and then (sadly...) forgotten, and so on.
I don't know how typical this is, and it doesn't matter. Because neither a tag based nor a tree based system address the real issue - which is designing a custom file workflow that collects related references of all kinds, doesn't confuse working data with long-term storage, allows off-site backups, allows collaboration, supports versioning on demand, and also makes it easy to find things.
I suppose all of that means some kind of process API which does a lot more than file.open() and file.close().
It could be built on tags, it could be built on trees, it could be built on some combination. Or on something else entirely.
The implementation matters a lot less than a set of available features which streamline common tasks in some fairly standardised and effective way.
It does and it's really bad IMO. The author's suggestion of unique identifiers though would introduce all sorts of new problems, primarily it would make the transparency problems of existing systems worse.
Most applications rely on the location of a file, relative or otherwise to load data (e.g. configuration). That reliance is exploited by software engineers to implement configuration swaps, event processing, and many other features. Referencing files based on UIDs, or a series of tags that aren't guaranteed to be unique or not known to be off limits to regular users, would introduce all manner of complications.
I could also see it being terribly easy to introduce bugs loading files using filtered tags. Would applications need to have relative tags to mitigate these problems? Having unique paths works both as a filter for the user and an encapsulation for a system that allows you to localize your concern. Without that encapsulation by default, you will be spending a lot more time and concern dealing with files and tags.
Tag coulds and other meta data can still be very useful. The challenge is creating useful tags/meta data automatically. For example a time stamp for every modification or a label for every application that created, modified or loaded the file. Perhaps even the applications you were using when the file was created/modified and the file names of the files loaded into the application. Train some AI to show you files you probably want given your current activity.
A big difference is that one can naturally have multiple tags, and an entity could share tags with other entities.
Sure you can use hardlinking when it comes to files, but it's tedious and you can't have multiple files hardlinked to the same path.
And yes, even with a layer on top of the FS to provide an abstraction (as an API, for instance) so you can build shells and applications, a tag + search based system would quite possibly be tedious to use.
I also don't think 64 bit ints provide a good way to definitively name things. Most people can't make sense of a list of 10 ints, but they will be able to remember at least where to look if you give them full paths.
Little trick I learned to help sort images: Make a copy of the file in as many locations as you like, then run something like borg backup. One file, hardlinked in as many directories as you want.
Somewhat related:
Tagsistant https://news.ycombinator.com/item?id=14537650
TMSU https://news.ycombinator.com/item?id=11660492
BeOS File System https://news.ycombinator.com/item?id=17468920
TagSpaces https://news.ycombinator.com/item?id=12679597
git-annex https://news.ycombinator.com/item?id=29942796
Names should mean what, not where https://dl.acm.org/doi/10.1145/506378.506399
Unfortunaly it's not easy to get a real solution, and many people don't think that there is a problem at all (based on some comments in this thread).
Now adays I use git-annex, though it does have it's perks it seems a step in the right direction.
I can't believe how much trouble I went through trying to find the filesystem that could do everything.
I mostly use git-annex, and various tagging systems, or just git repos. Now my data is much more portable and flexible. None of these tools are perfect, but I'm using tools that are mostly good at the job.
Whatever problem you are trying to solve, you probably don't need to solve it for your entire filesystem.
SuperTag https://amoffat.github.io/supertag/
I'm all for enriching the concept of a file system with additional meta-data (in fact many files do that) but I don't think that needs to happen in the file system itself. For example, software like Picasa leveraged meta-data contained in files to provide a new way of interacting with large number of photos. The author basically proposes to put such functionality directly into the file system, but I'm really not sure if that's a good idea. Right now it's easy to move files between different systems, e.g. from Mac to Windows or Linux. If file systems become meta-data management databases that will become much more difficult.
IMHO it's because those systems just simplify, but don't move very deep in the space they opened up. If you don't offer power, then it's irrelevant which system you offer, they will all suck fast.
> Every app has its own way to store and retrieve data, there's almost zero interoperability and it's super painful to copy, paste and move stuff around (I know it has gotten better but it's still so much worse than on any desktop OS).
Which is kind of a surprise, I would think Apple would be interested to unify that space and offer a good user experience.
> Right now it's easy to move files between different systems, e.g. from Mac to Windows or Linux. If file systems become meta-data management databases that will become much more difficult.
Theoretically, it could be solved by using a meta-file-container. Something like a tar-container, which contains a file for meta-data and the actual content. We have this with specialized container-formats in media and office-filetypes. Making a universal format which would work equally well for any kind of file type could solve this problem of interoperability. This would even open up ways to improve files without changing them directly. Like adding subtitles or notes to a file, by just adding it to the container, not the file itself.
Spotlight parses and indexes all the existing metadata in your files (music ID3 tags, photo EXIF tags, etc - run `mdls` on a file in a terminal to see all the stuff it's extracted) and this could all be used to make some pretty powerful UIs, but all Apple has done is made one very handy universal search UI, and then a very poorly designed specific search UI, and then made stored searches (which are also useful, but limited in practicality by how bad the UI to create them is)
D:\masterarchive\source\YYYY\YYYYMMDD\photo file name
If I want to find a person, in a photo, I've used Google Picasa (when it was an offline product) and lately digiKam to do face matching, and tagging them with IPTC metadata tags in the photo files. Thus they survive moves across filesystems, etc.
I'm up for seeing alternatives, but there's a very high bar to clear here. People have been using directories and file storage since the middle ages.
But that isn't really saying anything about if the interface sucks, or could be improved, just that you're motivated and clever enough to find a good and scalable solution for what you want to do given the limitations of the interface.
I do the same with documents. I don't even want to think about categories, ontologies are always wrong. But today is 2022-02-24, no two ways about it. It's automatic, there's no need to think or decide anything, so it's not a big deal, you just do it. You can't make a mistake.
The thought of needing to properly tag every document I file is enough to make that a task I want to postpone. So it wont get done. That's a worse filesystem right there, because it doesn't exist.
It should be especially easy to have a full sub-tree view sorted by date, but it typically isn't.
So they've already made the decision to commit the files to long-term storage, and to keep the original photos separate from subsequent edits, and to keep them separate from other image sources (e.g. downloads). That "tagging" required very little effort because they could just navigate to the existing tag in the filesystem, and put the new files there.
I am terrible at dates - if I had to find photos by date I'd never find them. For anything older than a month that isn't on a known anniversary like a birthday, 90% of the time I find the photos using the map view in iCloud Photo Library. If I was limited to a filesystem view, my photo library would be far less useful.
What I'd like, personally, is a way to expose any sortable EXIF data as a 'filesystem', for example `~/Photos/Longitude/122-123/Latitude/36-37/`.
Most of the commentary on a tag system seems predicated on the idea that we can't derive a large volume of tags automatically from the circumstances and provenance of the data. For instance, instead of a Downloads folder (per se, it would be a view) we could have a "downloaded" tag, which could have "downloaded-by" = "Chrome" and "downloaded-from" = "https://example.com/a-url/".
That's a lot more useful to me than a Downloads folder, especially if those tags endure when I add further metadata of the "canonical folder" variety, also known as "moving" the file.
Likewise, imposing archaic methods of organisation on modern storage is sub-optimal. The argument is to move toward something that makes more sense given the capabilities of the medium.
Search and tagging are not contradictory to cataloguing. They're complementary.
It's also not true that old media didn't have any search facilities. Old technical books would each have an index of keywords at the end. That's search, just analog and requiring a bit more work from the publisher. This index didn't make the table of contents redundant.
There are two absolute genius inventions in computers so good and timeless that the sliced bread pales in comparison like a stupid troll comment on HN.
1. The keyboard
2. The hierarchical filesystem
Everything and anything else in input devices and data storage builds on these and the best solutions ever always are going to augment these, never replace them.
A good tag system will build on top of a filesystem and coexist with it, and offer value like stupidly fast search. Anything else will be lucky to survive a weekend of dubious fame on twitter, or up to a few months if you actively market it.
By that I mean pictures I take, papers I write, things I really wouldn't want to lose, vs 10,000 random system files.
For downloaded files sometimes the history of when it was downloaded, and from where, is almost as important as the contents.
Backups from old computers, and old phones start to pile up, and the chaos of trying to find that picture you took 3 phones ago, or the notes you took, or the recording you made, or the pdf you downloaded, or that code you wrote, or that map you made, is a real pain.
Digital clutter is one of my biggest problems.
I really need a good way to deduplicate and organize ALL my digital stuff. Tags might play a role, but I don't think they quite solve the problem.
Because I'm seeing "Windows Explorer" in background:
Windows Explorer has degraded in recent years, it is even hard to open your "home directory" and the UI is confusing. Look at the one from NT 4.0 which was much more close the fulfill the task.
And Apple:
I think the regret nowadays the howl iTunes? But instead the pushing hard on apps which contain the data. Now you have to look always into a single app and uses it facilities to retrieve a file. Android failed here, too. But using iOS is hard.
Windows clearly doesn't want people to GET to their home directory, for some reason. That seems goofy. If people don't understand that $user contains the rest of those folders (Documents, Downloads, Pictures, etc) they'll never be able to navigate on their own. That's bad.
In a sane tool that features an address bar, clicking any given directory would show, in the address bar, the path to that location. WinExp only rarely does this. If you click on, say, Desktop, it shows you This PC > Desktop, implying a relationship that is incorrect. Getting to your home folder without typing requires you to start with C: and drill down, which is objectively insane.
Even MORE bananas is that if you start at C: and drill down to Desktop, you DO get the correct path in the address bar. But if you then make a WinExp shortcut of that location, it goes back to the other behavior. WTF.
Yes. Windows prevents users nowadays from understanding a straightforward thing, file-systems. I mean it was always a bit clumsy with A:, C: and [D-Z]: and the weird desktop metaphor harmed as well.
Now I'm looking at the often criticized GNOME and the actually venerable Nautilus. They got it! All below / and in addition devices are directly usable (actually still somewhere below /run). The location bar reflects the current position. The desktop was removed because it never fit into a computer and the file system.
Some actions of Google within Chrome are also questionable. "There is not address entry field" because we don't want you to understand how the web is structured. What? File-Systems are a simple thing, hierarchical. And guess what, the web is similar. Compared to "right click", "double click" and it's new friends "long press", "hard press" and "swipe from somewhere" and "guess what the voice assistant can interpret".