This is brilliant! Being called a “file system access” api will confuse many people into thinking it’s about traditional file storage for “people” to use, like a file picker/save dialog. It’s not, this is about providing a block storage that can be used for other things.
The one I am most excited by is for persistent SQLite with proper acid transaction in the browser, not having to load the whole db into memory. Absurd SQL [0] currently does this by creating a VFS on top of IndexedDB. This would let it do it properly, and is likely to be upstreamed to SQL.JS which is the main SQLite WASM project.
I don’t see how this is fundamentally any different from IndexedDB: both are asynchronous transactional key-value databases. One is byte-oriented and the other object-oriented, but it looks to me like they’re essentially completely equivalent in what’s possible with them—except insofar as File System Access may allow you to avoid loading the entire thing into memory. (I say may because you can’t control the browser’s implementation; my feeling is that it’s not unlikely that the browser may keep the full thing in memory in FSA where I wouldn’t expect it to with IndexedDB, which if so would return them to basically the same position.)
Implementing SQLite on top of this would require that a commit write the changes, close the file, and then reopen it, since writes are only performed when you close the file. That could perform tolerably or terribly (there’s no way of knowing), but certainly won’t be as good as what you get natively, where you can truly write only part of a file, especially once you get to large databases where it will certainly be drastically slower. If you want performance out of any of these sorts of things, you’re going to need to stop putting everything in one “file” and split it into many, so that you can avoid touching most of them on most writes.
I haven’t had a chance to dig into the details of what part/version of the spec Safari have implemented so far, and I’m aware the spec is somewhat a moving target. But I do know the developers working on the type of projects I’m talking about are talking to the working group behind the spec to ensure the spec meets these type of needs.
The point of this over “abusing” IndexedDB, jlongster who created Absurd SQL had to perform some pretty unholy hacks to get it to work. When porting these db engines to WASM they expect a FS to look and behave like a FS, that’s what this does that IndexedDB doesn’t.
Quite true you could build a new SQL/DB engin on top of IndexedDB with a storage architecture designed for it. But that’s not what the existing engines are expecting.
I'm not sure that SQL in the browser is such a good idea in the long run. Database migrations are hard enough when there's just one instance of the DB; managing and performing migrations in each individual user's browser is going to be incredibly painful. Performance is also likely to be an issue.
I would take the hassle of carefully constructed database migrations over a no-sql database with records in multiple unknown states from different versions requiring handling routines scattered throughout my codebase.
SQL will always win, as will the web, the potential of SQL in the browser for PWAs is incredible. It also allows alignment of mobile and web apps for local storage.
It was also the correct decision to drop WebSQL, it would have limited the api and held it back. With WASM SQL we can all use whatever sql db we want with whatever extensions we want. Row/Column oriented? Our decision (SQL.js or DuckDB right now). Need a specific full text search extension, or a custom one? Just compile it in. Have an idea for a funky CRDT based replication system? Write an extension and add it.
your phone already has some tens or hundreds of SQLite databases on it. it requires some care, but many already go through the effort in their apps, so i can imagine web developers going to the same effort if they need it.
though i can also imagine them going through that effort when they don’t, so…
managing and performing migrations in each
individual user's browser is going to be
incredibly painful
What's the alternative? Typical NoSQL slop where we basically just wind up doing the "painful" SQL stuff anyway in what ultimately winds up being an even more painful process?
I think this is an ok permissionless fallback if you dont want to bother the user, but tbis feels so radically useless compared to local filesystem access.
It would be be so much more powerful & useful a capability if your users could see the sqlite db, of whatever other store you have. It feels like such a Safari-ism that this great capability does nothing for the user, does not help the web export & share itself. It keeps the web & the system boxes fully isolated. Safari's murderous intent.
This is incorrect. The File System Access API 100% does contain a showOpenFilePicker API. As it says in the blog post, they’ve just begun this process by beginning with the private origin portion of the API, but they’ll be adding the other portions later.
It seems this is up to the implementation as the spec leaves it open. Safari is not mapping this to the actual file system, but what about other browsers?
For Origin Private File Systems, I would not expect any user agent to map things to the actual file system: there are notable functional problems especially on Windows, and it’s a very significant security hazard, all of which is neatly avoided by putting it all in the likes of a SQLite database.
First, the functional problems: file names are going to be a bit of a bother, to the point where if you back it by the actual file system you will require some kind of escaping mechanism, which loses you the exact one-to-one correspondence. Names are sequences of 16-bit code units, which means they need not be valid Unicode (ugh, I wish new additions to the browser platform would just start rejecting malformed Unicode), which will cause trouble on some platforms and will probably not be looked on favourably on others as having the potential to cause trouble in software not written to cope with that (e.g. on Windows I don’t think I’ve ever encountered a name with an unpaired surrogate, though it’s legal; on Linux, paths that aren’t UTF-8 are decidedly more common, though they still have a habit of breaking things). It also allows various file names that are forbidden in Windows at large, though in almost all cases you could with difficulty work around that on a per-process basis by throwing it into POSIX mode and/or using fully qualified (\\.\ or \\?\) paths.
Second, the security hazard: putting files with arbitrary names and contents on the file system is dangerous for a few reasons. The simplest is that there’s various buggy software that automatically reads files that get added to the file system, and more than a few times such software has had bugs in parsers that have led to privileged arbitrary code execution. A key purpose of Origin Private File Systems is that you don’t need to prompt the user for permission, because it’s supposed to be safe. I have no idea if this sort of vector was ever used back in the days when IE persisted its internet cache directly to files on the disk, but I can easily imagine such a thing happening now, and it could be really bad, given the right bug.
I’m curious how much use something like this would ever get; the Origin Private File System is basically just a file system implemented atop a database: it doesn’t provide any new fundamental functionality, but can be shimmed perfectly atop IndexedDB (though performance characteristics will probably differ). When people hear “file system access”, they’re going to expect that this means you can access arbitrary files and folders on the file system, whether unbounded or within a certain scope, but the Origin Private File System is not that. There need be no correlation between the file system this API exposes and your file system. (In theory there perhaps could be, but it’s not the preferred way and there’s no obvious reason to do it that way, and the question of valid file names would be rather messy, even if the spec kinda makes a weird half-compromise for backslash in file names; and I think the stuff about atomic operations could make it difficult to do sanely, too.) It doesn’t put things in a private folder so that you could open its contents in other locally-installed apps.
So this is basically the harmless but also fairly useless part of File System Access. The only real reasons I can think of straight away for it potentially being useful are (a) if it performs better than IndexedDB, and (b) for simpler API compatibility with something using the dangerous parts of File System Access, which Chromium has implemented but which Firefox and Safari are both utterly rejecting.
I’m also a bit concerned about browsers shipping this stuff given how much flux there still is in the draft spec, which is still thoroughly in its incubation phase and hasn’t been adopted by a working group.
In the end, I can’t understand why they’d implement this part of the spec. It will get some people’s hopes up, only to dash them, since they’re not implementing the useful but dangerous part which is what people actually want.
If a browser syncs a private file system access across devices, I could see this being quite useful; e.g. you could have a Google Docs -like browser based document application that doesn't depend on a Google hosting and snooping on it. I don't know about whether it adds anything above indexedDB.
"since they’re not implementing the useful but dangerous part which is what people actually want."
I want it like this. There are lots of use cases for a file system to store user data locally in a safe way.
Even though it all can be also done with indexedDB, too - it makes this task incredibly more complicated. I did basically this, implement a file system on top of indexedDB - but with great pain.
and I did, because there was no other option - but only with great unnecessary pain.
I haven’t tried, but I think a minimal polyfill of exactly this atop IndexedDB might take 300–400 hundred lines of code (and I’d be interested in the difference in performance characteristics, especially on large “files”, and with different types of edit strategies). A mildly simplified but close enough to feature-complete form of it could probably be done in under 200 lines, though I doubt 100 is attainable honestly (IndexedDB is comparatively verbose).
It’s not that this is nothing, but I think it is and will be being fairly heavily oversold by some people.
Unfortunately, this does not seem to be the "local file access" that Chromium offers and that Safari lacks.
With better browser support for the File System Access API, web applications that store their data in files might become a common thing.
Currently, when building a web application, you usually build some backend system that lets the user log in and then stores the data. Or you use IndexedDB and let the browser handle it. In both cases, the user does not have good access to the data.
If instead on the first run, the application asked the user "Where do you want to store the data" and the user selects a file or a directory, that puts the user in full control.
Then they later can backup the data however they like, edit it with other tools, version it etc etc.
The browser is an awesome platform. I love to write local tools in HTML. It is just so easy to tweak browser based applications to your needs. Open the html file, change a line or two, save - boom! you got what you want.
> If instead on the first run, the application asked the user "Where do you want to store the data" and the user selects a file or a directory, that puts the user in full control.
It doesn't, not realy.
Both Safari and Firefox are now very wary and weary of asking the user for anything. There are so many things browsers already ask the user for: camera, location, notifications, microphone... In case of Chrome, additionally: USB, Serial, HID, ...
All the discussions about such features inevitably boil down: it's impossible to properly explain to the user what the hell is going on and what the implications are. At one point the user will just click "ok" without reading.
> It is very common for an application to interact with local files. For example, a general workflow is opening a file, making some changes, and saving the file. For web apps, this might be hard to implement. It is possible to simulate the file operations using IndexedDB API, an HTML input element with the file type, an HTML anchor element with the download attribute, etc, but that would require a good understanding of these standards and careful design for a good user experience. Also, the performance may not be satisfactory for frequent operations and large files.
> Based on the implementation of different browsers, one entry in the origin private file system does not necessarily map to an entry in the user’s local filesystem — it can be an object stored in some database. That means a file or directory created via the File System Access API may not be easily retrieved from outside of the browser.
Honestly, this makes no sense to me. The motivating example is exactly about being unable to interact with the host file system, but then they present a solution that does something completely different.
If this is supposed to be yet another storage API, then so be it, but this won't be able to solve the UX problems that the first quote was talking about.
Edit: Aha, the spec clears this up a bit: The standard defines different implementations of the file system API. Two of them are the "local" [1] and the "origin private" [2] file systems. The former is indeed a view of the host file system, with picker and all, while the latter is completely independent of the host.
Looks like this feature announcement was exclusively about the latter.
(The motivating example still make no sense to me as that is a clear use-case for the "local" filesystem implementation, but that seems to be more an issue with the announcement, not with the feature itself)
Edit2: Another important distinction is that the local file system requires user interaction to use (with good reason) : You have to show a file picker before you can use it, and subsequently can only access the files and directories the user selected in the picker. So you cannot use this as a behind-the-scenes storage mechanism.
In contrast, the origin-private file system requires no permissions and no user interaction.
Why does the implementation detail around origin file systems affect the motivating use case from the announcement?
Web apps can now create their own file systems and treat objects as files (and it seems directly point to such files) with this api. That wasn’t easily possible before.
The type of files accessed is radically different: The local FS acesses already existing files that users typically manage by hand, whereas the origin-private FS accesses files that the webapp created itself and will typically manage internally.
In Unix terms, the former gives you access to parts of /home, the latter to parts of /var or /tmp.
I'm also not sure if calling them different "implementations" (as I did in the GP) is really the best way to put it. Both filesystem objects implement the same interface, but you call different APIs to obtain the objects in the first place and would use them in different contexts.
My issue was that the motivating example was about rebuilding the "open"/"save as" functionality from desktop apps, which the local FS API was tailor-made for - but the announcement is about the origin-private FS API, which has a completely different purpose.
Thanks for clarifying this for me. I'd gone through some of the specs of the File System API, but not nearly thoroughly enough to pick up on this distinction.
I've read this a few times, but I don't see how you can regard this as anything other than false advertising. In short, it's basically a file-like wrapper around the hard-to-use IndexedDB. So it pretty much competes with localForage and idb-keyval.
The promise of File System Access API is in interop. There is no interop with Safari's implementation.
I understand the desire to have file access as a developer, but as a user, Safari and Firefox's decision to not implement the rest of it is a great decision. It's not worth the attack surface.
It is possible to simulate the file operations using IndexedDB API, an HTML input element with the file type, an HTML anchor element with the download attribute, etc, but that would require a good understanding of these standards and careful design for a good user experience. Also, the performance may not be satisfactory for frequent operations and large files.
…
Based on the implementation of different browsers, one entry in the origin private file system does not necessarily map to an entry in the user’s local filesystem — it can be an object stored in some database. That means a file or directory created via the File System Access API may not be easily retrieved from outside of the browser.
So what is the point of using this instead of IndexedDB, then? In IndexedDB we can even save private non-extractable keys!
Fortunately for you this shouldn't be outdated for a little while longer, since this only covers "origin private" filesystems. Which, in other words, are just a file-structured IndexedDB and not actual local file access.
Readit News
The one I am most excited by is for persistent SQLite with proper acid transaction in the browser, not having to load the whole db into memory. Absurd SQL [0] currently does this by creating a VFS on top of IndexedDB. This would let it do it properly, and is likely to be upstreamed to SQL.JS which is the main SQLite WASM project.
0: https://github.com/jlongster/absurd-sql
Many new in browser DB engines are going to get built on top of this. Others that I could see happing are:
- Relm from MongoDB being ported to WASM and use this for storage.
- If I were Supabase I would be looking to create a “Mini Supabase” for mobile, and make it work in browser too.
- Couchbase Mobile as an alternative to PouchDB
Implementing SQLite on top of this would require that a commit write the changes, close the file, and then reopen it, since writes are only performed when you close the file. That could perform tolerably or terribly (there’s no way of knowing), but certainly won’t be as good as what you get natively, where you can truly write only part of a file, especially once you get to large databases where it will certainly be drastically slower. If you want performance out of any of these sorts of things, you’re going to need to stop putting everything in one “file” and split it into many, so that you can avoid touching most of them on most writes.
The point of this over “abusing” IndexedDB, jlongster who created Absurd SQL had to perform some pretty unholy hacks to get it to work. When porting these db engines to WASM they expect a FS to look and behave like a FS, that’s what this does that IndexedDB doesn’t.
Quite true you could build a new SQL/DB engin on top of IndexedDB with a storage architecture designed for it. But that’s not what the existing engines are expecting.
SQL will always win, as will the web, the potential of SQL in the browser for PWAs is incredible. It also allows alignment of mobile and web apps for local storage.
It was also the correct decision to drop WebSQL, it would have limited the api and held it back. With WASM SQL we can all use whatever sql db we want with whatever extensions we want. Row/Column oriented? Our decision (SQL.js or DuckDB right now). Need a specific full text search extension, or a custom one? Just compile it in. Have an idea for a funky CRDT based replication system? Write an extension and add it.
though i can also imagine them going through that effort when they don’t, so…
It would be be so much more powerful & useful a capability if your users could see the sqlite db, of whatever other store you have. It feels like such a Safari-ism that this great capability does nothing for the user, does not help the web export & share itself. It keeps the web & the system boxes fully isolated. Safari's murderous intent.
First, the functional problems: file names are going to be a bit of a bother, to the point where if you back it by the actual file system you will require some kind of escaping mechanism, which loses you the exact one-to-one correspondence. Names are sequences of 16-bit code units, which means they need not be valid Unicode (ugh, I wish new additions to the browser platform would just start rejecting malformed Unicode), which will cause trouble on some platforms and will probably not be looked on favourably on others as having the potential to cause trouble in software not written to cope with that (e.g. on Windows I don’t think I’ve ever encountered a name with an unpaired surrogate, though it’s legal; on Linux, paths that aren’t UTF-8 are decidedly more common, though they still have a habit of breaking things). It also allows various file names that are forbidden in Windows at large, though in almost all cases you could with difficulty work around that on a per-process basis by throwing it into POSIX mode and/or using fully qualified (\\.\ or \\?\) paths.
Second, the security hazard: putting files with arbitrary names and contents on the file system is dangerous for a few reasons. The simplest is that there’s various buggy software that automatically reads files that get added to the file system, and more than a few times such software has had bugs in parsers that have led to privileged arbitrary code execution. A key purpose of Origin Private File Systems is that you don’t need to prompt the user for permission, because it’s supposed to be safe. I have no idea if this sort of vector was ever used back in the days when IE persisted its internet cache directly to files on the disk, but I can easily imagine such a thing happening now, and it could be really bad, given the right bug.
So this is basically the harmless but also fairly useless part of File System Access. The only real reasons I can think of straight away for it potentially being useful are (a) if it performs better than IndexedDB, and (b) for simpler API compatibility with something using the dangerous parts of File System Access, which Chromium has implemented but which Firefox and Safari are both utterly rejecting.
I’m also a bit concerned about browsers shipping this stuff given how much flux there still is in the draft spec, which is still thoroughly in its incubation phase and hasn’t been adopted by a working group.
In the end, I can’t understand why they’d implement this part of the spec. It will get some people’s hopes up, only to dash them, since they’re not implementing the useful but dangerous part which is what people actually want.
I want it like this. There are lots of use cases for a file system to store user data locally in a safe way.
Even though it all can be also done with indexedDB, too - it makes this task incredibly more complicated. I did basically this, implement a file system on top of indexedDB - but with great pain.
and I did, because there was no other option - but only with great unnecessary pain.
It’s not that this is nothing, but I think it is and will be being fairly heavily oversold by some people.
With better browser support for the File System Access API, web applications that store their data in files might become a common thing.
Currently, when building a web application, you usually build some backend system that lets the user log in and then stores the data. Or you use IndexedDB and let the browser handle it. In both cases, the user does not have good access to the data.
If instead on the first run, the application asked the user "Where do you want to store the data" and the user selects a file or a directory, that puts the user in full control.
Then they later can backup the data however they like, edit it with other tools, version it etc etc.
The browser is an awesome platform. I love to write local tools in HTML. It is just so easy to tweak browser based applications to your needs. Open the html file, change a line or two, save - boom! you got what you want.
This is entirely contingent on the format used. Netflix or any other media streaming company would store DRM compatible formatted content locally.
I think the feature in of itself opens up interesting possibilities but not sure how it will get used.
It doesn't, not realy.
Both Safari and Firefox are now very wary and weary of asking the user for anything. There are so many things browsers already ask the user for: camera, location, notifications, microphone... In case of Chrome, additionally: USB, Serial, HID, ...
All the discussions about such features inevitably boil down: it's impossible to properly explain to the user what the hell is going on and what the implications are. At one point the user will just click "ok" without reading.
> Based on the implementation of different browsers, one entry in the origin private file system does not necessarily map to an entry in the user’s local filesystem — it can be an object stored in some database. That means a file or directory created via the File System Access API may not be easily retrieved from outside of the browser.
Honestly, this makes no sense to me. The motivating example is exactly about being unable to interact with the host file system, but then they present a solution that does something completely different.
If this is supposed to be yet another storage API, then so be it, but this won't be able to solve the UX problems that the first quote was talking about.
Edit: Aha, the spec clears this up a bit: The standard defines different implementations of the file system API. Two of them are the "local" [1] and the "origin private" [2] file systems. The former is indeed a view of the host file system, with picker and all, while the latter is completely independent of the host.
Looks like this feature announcement was exclusively about the latter.
(The motivating example still make no sense to me as that is a clear use-case for the "local" filesystem implementation, but that seems to be more an issue with the announcement, not with the feature itself)
Edit2: Another important distinction is that the local file system requires user interaction to use (with good reason) : You have to show a file picker before you can use it, and subsequently can only access the files and directories the user selected in the picker. So you cannot use this as a behind-the-scenes storage mechanism.
In contrast, the origin-private file system requires no permissions and no user interaction.
[1] https://wicg.github.io/file-system-access/#local-filesystem
[2] https://wicg.github.io/file-system-access/#sandboxed-filesys...
Web apps can now create their own file systems and treat objects as files (and it seems directly point to such files) with this api. That wasn’t easily possible before.
In Unix terms, the former gives you access to parts of /home, the latter to parts of /var or /tmp.
I'm also not sure if calling them different "implementations" (as I did in the GP) is really the best way to put it. Both filesystem objects implement the same interface, but you call different APIs to obtain the objects in the first place and would use them in different contexts.
My issue was that the motivating example was about rebuilding the "open"/"save as" functionality from desktop apps, which the local FS API was tailor-made for - but the announcement is about the origin-private FS API, which has a completely different purpose.
The promise of File System Access API is in interop. There is no interop with Safari's implementation.
So what is the point of using this instead of IndexedDB, then? In IndexedDB we can even save private non-extractable keys!
Does anyone know if there's a list of small, little-known extensions just like this?
(and safari does this automatically after some idle time, I think)