The vast majority of the time we spend diagnosing and working around browser bugs is devoted to Safari. As an iPhone/Mac user and lover of all things Web, I really, really want Apple to get their act together with Safari. We sadly keep running into issues like this.
Safari is the red-headed stepchild of browsers. The dev tools are weird and different, the bugs are obscure and it's pretty much impossible to do anything on it. I love browser diversity but I wouldn't be sad to see it go away.
Wormhole.app uses peer-to-peer transfer to send your files directly from your browser to the recipient, in addition to using cloud hosting.
We use WebTorrent behind-the-scenes, so we need random access to the end-to-end encrypted version of the files to satisfy requests from peers. So we try to put the files on disk using Filesystem Access API or IndexedDB, falling back to in-memory if there's not enough storage.
Easy, stop supporting Safari.
No-one cares when IE support is dropped (people usually cheer for this) and user needs to download and install another browser.
So, stop supporting Safari and concentrate on Chrome (I use Firefox but their market share is even smaller so who cares).
If enough people show they don't support Safari, THEN Apple will listen.
If you keep complaining about having to support Safari then why would Apple care? Not only are you doing all the work for free, you are paying them money for their devices and software.
Any browser on iOS must use safari browser engine (otherwise won't be approved in app store) - the same issue will arise while using Chrome or Firefox on iPhone. Unless you can afford not support iOS/iPadOS users, devs have not choice but complain about bugs.
Chrome has other issues once in a while as well. Ironically, they broke IndexedDB in 2018 when a device had less than 1 GB of available space (which happened surprisingly often in 2018 on Android).
A few months ago, Chrome broke all WebView-based apps. Fortunately, they fixed it within a day. The IndexedDb bug was fixed after 6 weeks.
On iOS you don't have a choice other than Safari (other browsers are just wrappers around safari die to Apple policies). And most of us can't afford to drop iOS support (and most of us aren't even in the position to make that call).
The thing is, Apple WANTS us all to stop supporting Safari. It doesn’t help them make money to keep maintaining and extending WebKit to compete with the other browsers. Their (shareholders’) goal is to get more app purchases. I think the only realizable solution is something like antitrust litigation or class action lawsuit to allow web browser competition on iOS, allowing Firefox and Chrome to outcompete Safari. Then we can all drop support and give them the middle finger like IE
I wrote up the bug report for this a couple of weeks ago when iOS 14.6 first rolled out. iOS 14.6 at least did finally fix using HLS streams in gl.texUpload2D() on iPhone 12 series (which was broken since launch) but this new bug did mean our delight at that fact was somewhat short-lived!
Well done on your viral tweet that got some personal attention from Apple. Usually when I spend days narrowing down Safari bugs to minimal reproducible test cases, finding workarounds, and writing everything up in detail on the webkit bug tracker the only public response from Apple is the radar importer bot, and then everything goes quiet.
Btw, your original twitter thread implies you were the author of the bug report. Minor annoyance, but I did spend quite a while writing that up :)
Chrome 91 on Android also managed to break WebAssembly memory growth which affects our sites too, so it's been a rough couple of weeks working around browser bugs for us.
https://bugs.chromium.org/p/v8/issues/detail?id=11863
I find the Chromium team much more willing to engage directly and constructively on their bug trackers at least.
I’ve been shocked at the number of show-stopper bugs shipping in Safari recently so I’m doing what I can to draw attention to it. I hope it helps the Safari team get the resources they need from the big wigs at Apple.
Also: Didn’t mean to imply I wrote that report. We were about to open a duplicate issue before noticing yours. Thanks!
No worries, I imagine many teams have been through the same process over the past couple of weeks!
We've definitely had the same experience of Safari introducing way more issues with new releases than Chrome, and the fact it's part of the iOS image seems to make quick patching just never really an option.
I just wish they'd engage a bit more directly on the bug tracker - find it hard to believe that twitter is a more effective means to get attention on bugs.
It's a terrible API anyway and if it died I wouldn't shed a tear. If you need a good laugh (or more likely, a heart attack from the blood pressure rise it will induce) have a look at this blog post[1] by Mozilla justifying not using SQLite and choosing to implement the much more "elegant" IndexedDB. That post has too many examples of hubris and delusion for me to bother quoting it here, I'd just be reproducing most of it.
Then you can go to their code comparison page[2] and wonder what the hell they were on about.
> IndexedDB generally simplifies the programming model for interacting with databases, and allows for a wide number of use cases.
They can't be looking at the same code - what have they improved other than to use more Javascript?
The sooner it dies the sooner an alternative can appear… SQLite3, anyone?
I mean I can understand why they dropped websql, as great a project as SQLite is, mandating that browsers expose SQLite is not a spec, and expecting a complete reimplementation of sql by browser developers would be foolish.
But yeah the assertion that indexeddb is any good is hardly convincing.
Nowadays you can use sqlite3 through wasm if you want, though I don’t know how you’d handle the persistence layer.
> Nowadays you can use sqlite3 through wasm if you want, though I don’t know how you’d handle the persistence layer.
There was a post on here in the last month or 2, atleast for reading, that used http range headers for reading the db pages.
as for writing, you're gonna hate this, but a vfs impl. for SQLite using indexedDB as the backing store is your best option (Safari stupidity not withstanding)
Granted that the DOM event API feels clunky, but I actually think they got things mostly right:
- low-level storage primitive with support for indices, "you pay only for what you use".
- very fast if used in the right way
- very large sizes supported
- built-in version support, making it easy to do schema and data migrations.
- basic query api that again gives you a lot of flexibility and control over performance
- can be abstracted away by higher level libraries like Dexie
I've built a web-app heavily relying on IDB, and storage operations are generally much faster than the native app on Android using Realm DB. IDB is backed by leveldb on Chrome, which in benchmarks is consistently faster than sqlite. I'm happy we didn't end up with websql for one.
My biggest gripes are:
- The incompatibility with ES6 promises early on (but IE11 still suffers from) was unfortunate.
- The auto-commit behaviour of transactions is somewhat annoying and usually has an impact on your architecture if you want to write multiple things in one transaction, but it would have been easy to have memory leaks and deadlocks without this.
- Quota management and eviction is wildly inconsistent between browsers, even on Chrome you get evicted when storage gets critically low.
- Safari has a history of pretty bad IDB bugs.
SQLite is amazing. But after having worked with it for many years now, I believe it should never be a platform-level API, and not standardizing it for browsers was the right choice.
SQLite has a _lot_ of great features. Many of them only introduced in recent versions, and many of them are optional extensions enabled at compile-time. There is no feasible way to standardize this to get the same functionality across all browsers.
IndexedDB may not be great to work with (at least not without helper libraries), but at least the API surface is small, and much easier to ensure you get the same behaviour in all browsers.
I'd rather advocate for a proper block-level filesystem API
that could be used as backing storage for a WASM SQLite library.
> SQLite is amazing. But after having worked with it for many years now, I believe it should never be a platform-level API, and not standardizing it for browsers was the right choice.
No it was never the right choice and it didn't need to become standardized at first place. Writing a query builder is an easy task, implementing a RDBMS on top of indexedDB, ironically implemented on top of SQlite in Firefoxis a much harder tasks.
IndexedDB is absolutely atrocious API wise, and largely useless to query structured data.
IndexedDB's ugliness is due to being from a time before ES6+ really kicked into gear and we still had the old crud of weird callback stacks (See FileReader for another example).
As for SQLite, as lovely as that'd be, NOONE wanted to spend R&D on building a "SQLite clone" for the web SQL spec, and I can't blame them, how the hell can you justify that effort when SQLite exists and is crazy well tested and proven?
Have you ever tried to use it on a project? It's been a while for me as I moved to PouchDB or SQLite in WASM and never looked back.
Let's say you need to search for age > 19 and less than 30 for a person with the name of Jane or John.
That's nothing complicated, but indexedDB CANNOT DO THIS. I suppose you could do it by iterating every single data entry, but that's a performance no-go for even relatively trivial databases.
Likewise, joins on different stores simply don't exist. Iterating every entry in this case is O(N^2).
Might as well just store a giant JSON blob instead.
Quite literally, my only use of indexedDB these days is storing a SQLite file and whatever interactions PouchDB has (I think they are mostly limited to linking an ID with a JSON blob). Adding insult to injury, despite being a simple key/val store, it's SLOWER than SQLite too.
WebSQL was what we all wanted. But if I remember right, it couldn't be standardised because the W3C requirement insists that there should be atleast 2 different implementations of it in the clients - however all the browsers that supported it only built it using SQLite (which is a very obvious choice). If atleast one of the browser had built an implementation with a different database backend (like perhaps BerkleyDB), it could have become a web standard and a reality today. Instead, we are stuck with crap like IndexedDB.
If a least one browser had used a different backend, the API behavior would have had to actually be defined instead of just saying "behaves like SQLite". And that defined behavior would have to be implementable on top of both SQLite an the other backend.
All doable, but quite a lot of work. Once you start looking at the details, every single database has its own SQL dialect and they're not all that compatible except for the simplest queries.
IndexedDB does indeed suck, but there's really no alternative to it if you want Safari and Firefox support. Fortunately, in Chrome we have the FileSystem Access API which has a feature that lets you write into an "origin private" store, similar to how IndexedDB partitions storage per origin.
> There's even recent work to add even higher performance APIs into the Filesystem Access API and the reception from Firefox and Safari has been positive.
> That post has too many examples of hubris and delusion for me to bother quoting it here
Well, I could not help myself.
>> While versions of Safari, Chrome, and Opera support a technology called Web SQL Database, which uses SQL statements as string arguments passed to a JavaScript API, we think developer aesthetics are an important consideration, and that this is a particularly inelegant solution for client-side web applications.
>> We .. also spoke with Microsoft, who agree with us that IndexedDB is a good option for the web
OMG!! MS thought not doing what Apple and Google wanted was a good option. Earth shattering.
As for the code examples, both sets look horrible, but this and "_developer aesthetics_" don't go together. This is how they think their `JOIN` is "better" than SQL `JOIN`.
candyEaters = [];
function displayCandyEaters(event) {
var display = document.getElementById("purchaseList");
for (var i in candyEaters) {
display.textContent += ", " + candyEaters[i].name + "bought " +
candyEaters[i].count + "pieces";
}
};
var request = window.indexedDB.open("CandyDB",
"My candy store database");
request.onsuccess = function(event) {
var db = event.result;
var transaction = db.transaction(["kids", "candySales"]);
transaction.oncomplete = displayCandyEaters;
var kidCursor;
var saleCursor;
var salesLoaded = false;
var count;
var kidsStore = transaction.objectStore("kids");
kidsStore.openCursor().onsuccess = function(event) {
kidCursor = event.result;
count = 0;
attemptWalk();
}
var salesStore = transaction.objectStore("candySales");
var kidIndex = salesStore.index("kidId");
kidIndex.openObjectCursor().onsuccess = function(event) {
saleCursor = event.result;
salesLoaded = true;
attemptWalk();
}
function attemptWalk() {
if (!kidCursor || !salesLoaded)
return;
if (saleCursor && kidCursor.value.id == saleCursor.kidId) {
count++;
saleCursor.continue();
}
else {
candyEaters.push({ name: kidCursor.value.name, count: count });
kidCursor.continue();
}
}
}
I think indexedDb is just too low level for your use case. Because it’s actually very good! This low level primitive allows to implement various approaches on top of it, such as the excellent Dexie.js. You should have a look at it, it made me like indexedDb
I agree with everything you said, and it was the first time I got disappointed by Mozilla. Their excuse was "well we'd have to spec a common SQL and we don't want to do that". No there was no need to spec a common SQL for all browsers, let vendors choose whatever SQL version they want to implement. People have been building query builders for years, it's easy, much easier than re-implementing a RDBMS on top of IndexedDB. Then I learned that Firefox implemented IndexedDB on top of SQlite, what a freaking joke...
It made me really angry at Mozilla. WebSQL was so invaluable to me.
IndexedDB is indeed cancer. Usually it's also unnecessary, as you can store a significant amount of data in localStorage and the API is extremely simple to use.
If you need to store more data, why not do it on the server? What are the use cases for having a full-blown relational db client-side?
> it's also unnecessary, as you can store a significant amount of data in localStorage and the API is extremely simple to use.
Either you've never actually tried to store significant amounts of data in localstorage, or you're from the era of "640k is enough".
To say nothing of having no indexing functionality with localstorage, or paying the 33% base64 tax to put anything remotely binary into it.
> If you need to store more data, why not do it on the server?
"Because there is always, ALWAYS a high speed, low latency 100% uptime connection" /s
In answer to your last question, we have developed two PWAs reliant on local databases.
1. Was a product catalogue for use in stores, letting users filter down to select a particular appliance. two key requirements were that not all stores had great wifi coverage, and they wanted to easily update the catalogue (which meant no app recompilation).
So, Dexie atop IndexedDB, with indexes on the various facets for searching, and a simple settings page to run a sync.
Dexie stored the product data, and a "immutable" SW cache strategy was used to keep the images available.
2. A detailed report generation app, Key requirement, must work with patchy connectivity as reports need to be taken in the field with unknown signal availability. Reports are locally stored and synced periodically (connection permitting).
Again, Dexie atop IndexedDB, this time the images stored in the db as blobs prior to upload.
It's surprising how confident you are when you know so little. localStorage and indexedDB are very different in how they store data and read data, you also don't know the advantages both have over storing data on a server without using it as a full blown relational db.
I'm glad people are finally speaking out after all these years. I hated IndexedDB and got lambasted for expressing my views on this very forum. And I also recently just learned that IndexeDB was build on top of Sqlite in Firefox... what a cynical joke...
I just added a section on Safari's storage related bugs, of which there are many. Amazingly, IndexedDB broke in Safari 14.1.1 after localStorage broke in Safari 14.1, released a month earlier.
Yes... I've felt for years now that Safari in particular prioritized UI speed over correctness. They don't seem to care what they have to do to get a buttery UI. It is indeed buttery, much more than any other browser, but at what cost?
I assume you're aware of the WPT project (contributed to by all of the major browser vendors these days), and its CSS 2.1 coverage is lacking in a way I'm unaware of.
Forget automated tests; this is something the most cursory QA would have caught. The fact that they evidently have neither is a reflection not of technical ability but plain neglect.
QA very probably caught it, did a refresh (thinking that it's a problem with their test setup, as it is frequently the case) and then went about testing.
Apple never test safari very well, I have seen so many bugs that are easy to spot and reproduce.
My favorite one was websites added to the home screen on an iPad would have the clock displayed above the websites if you rotate the iPad after opening a website. It wasn't fixed for years and it may still be there.
It's opening the database _immediately_ after the browser starts that's the problem, which is easily missed given the browser isn't restarted for each test. Provided an IndexedDB test isn't run immediately after any restart, it'll pass.
As with every race condition (cross process, too, so tools like TSAN don't help), testing for the absence of race conditions is hard.
Readit News
The vast majority of the time we spend diagnosing and working around browser bugs is devoted to Safari. As an iPhone/Mac user and lover of all things Web, I really, really want Apple to get their act together with Safari. We sadly keep running into issues like this.
Here are a few more recent examples:
https://twitter.com/feross/status/1386835131780931587
https://twitter.com/feross/status/1392671684419735552
https://twitter.com/feross/status/1263544033135038464
I've been reporting bugs to WK for years and none have been solved.
Did you know you can't set the dimensions of an iframe in iOS?
We use WebTorrent behind-the-scenes, so we need random access to the end-to-end encrypted version of the files to satisfy requests from peers. So we try to put the files on disk using Filesystem Access API or IndexedDB, falling back to in-memory if there's not enough storage.
Dead Comment
A few months ago, Chrome broke all WebView-based apps. Fortunately, they fixed it within a day. The IndexedDb bug was fixed after 6 weeks.
Well done on your viral tweet that got some personal attention from Apple. Usually when I spend days narrowing down Safari bugs to minimal reproducible test cases, finding workarounds, and writing everything up in detail on the webkit bug tracker the only public response from Apple is the radar importer bot, and then everything goes quiet.
Btw, your original twitter thread implies you were the author of the bug report. Minor annoyance, but I did spend quite a while writing that up :)
I find the Chromium team much more willing to engage directly and constructively on their bug trackers at least.
Also: Didn’t mean to imply I wrote that report. We were about to open a duplicate issue before noticing yours. Thanks!
We've definitely had the same experience of Safari introducing way more issues with new releases than Chrome, and the fact it's part of the iOS image seems to make quick patching just never really an option.
I just wish they'd engage a bit more directly on the bug tracker - find it hard to believe that twitter is a more effective means to get attention on bugs.
Then I loaded up my 2005-vintage personal wiki which has been working fine for 15 years on all the browsers I use (Chrome and Firefox).
And it has a tabindex bug in Safari!
https://stackoverflow.com/questions/1848390/safari-ignoring-...
I'm not sure but isn't this a violation of the spec? Or is there some newer thing for tabindex? Either way I switched back to Firefox and Chrome.
I kinda like the idea of a browser that doesn't implement all the Chrome crap. But I also feel like it should interpret 15 year old pages correctly!
That's been my experience as well.
When reporting a bug to Chromium you get an answer in hours.
When reporting a bug to WebKit it's mostly crickets.
Then you can go to their code comparison page[2] and wonder what the hell they were on about.
> IndexedDB generally simplifies the programming model for interacting with databases, and allows for a wide number of use cases.
They can't be looking at the same code - what have they improved other than to use more Javascript?
The sooner it dies the sooner an alternative can appear… SQLite3, anyone?
[1] https://hacks.mozilla.org/2010/06/beyond-html5-database-apis...
[2] https://hacks.mozilla.org/2010/06/comparing-indexeddb-and-we...
But yeah the assertion that indexeddb is any good is hardly convincing.
Nowadays you can use sqlite3 through wasm if you want, though I don’t know how you’d handle the persistence layer.
There was a post on here in the last month or 2, atleast for reading, that used http range headers for reading the db pages.
as for writing, you're gonna hate this, but a vfs impl. for SQLite using indexedDB as the backing store is your best option (Safari stupidity not withstanding)
I've built a web-app heavily relying on IDB, and storage operations are generally much faster than the native app on Android using Realm DB. IDB is backed by leveldb on Chrome, which in benchmarks is consistently faster than sqlite. I'm happy we didn't end up with websql for one.
My biggest gripes are: - The incompatibility with ES6 promises early on (but IE11 still suffers from) was unfortunate. - The auto-commit behaviour of transactions is somewhat annoying and usually has an impact on your architecture if you want to write multiple things in one transaction, but it would have been easy to have memory leaks and deadlocks without this. - Quota management and eviction is wildly inconsistent between browsers, even on Chrome you get evicted when storage gets critically low. - Safari has a history of pretty bad IDB bugs.
SQLite has a _lot_ of great features. Many of them only introduced in recent versions, and many of them are optional extensions enabled at compile-time. There is no feasible way to standardize this to get the same functionality across all browsers.
IndexedDB may not be great to work with (at least not without helper libraries), but at least the API surface is small, and much easier to ensure you get the same behaviour in all browsers.
I'd rather advocate for a proper block-level filesystem API that could be used as backing storage for a WASM SQLite library.
No it was never the right choice and it didn't need to become standardized at first place. Writing a query builder is an easy task, implementing a RDBMS on top of indexedDB, ironically implemented on top of SQlite in Firefoxis a much harder tasks.
IndexedDB is absolutely atrocious API wise, and largely useless to query structured data.
This is happening at https://web.dev/storage-foundation/ fwiw
As for SQLite, as lovely as that'd be, NOONE wanted to spend R&D on building a "SQLite clone" for the web SQL spec, and I can't blame them, how the hell can you justify that effort when SQLite exists and is crazy well tested and proven?
Let's say you need to search for age > 19 and less than 30 for a person with the name of Jane or John.
That's nothing complicated, but indexedDB CANNOT DO THIS. I suppose you could do it by iterating every single data entry, but that's a performance no-go for even relatively trivial databases.
Likewise, joins on different stores simply don't exist. Iterating every entry in this case is O(N^2).
Might as well just store a giant JSON blob instead.
Quite literally, my only use of indexedDB these days is storing a SQLite file and whatever interactions PouchDB has (I think they are mostly limited to linking an ID with a JSON blob). Adding insult to injury, despite being a simple key/val store, it's SLOWER than SQLite too.
All doable, but quite a lot of work. Once you start looking at the details, every single database has its own SQL dialect and they're not all that compatible except for the simplest queries.
Which is implemented on top of Sqlite in many browsers, what a joke.
It's easier to implement IndexedDB on top of a SQL database than the other way around, so people claiming IndexedDB is "lower level" are ignorant.
There's even recent work to add even higher performance APIs into the Filesystem Access API and the reception from Firefox and Safari has been positive. Really early work is happening here: https://github.com/WICG/storage-foundation-api-explainer and the latest proposal is here: https://docs.google.com/document/d/121OZpRk7bKSF7qU3kQLqAEUV...
Positive as in: "FileSystem Access is a significant security risk and we're not going to implement this"? https://www.chromestatus.com/feature/6284708426022912
Positive as in "No, we don't want fifteen different file access apis, and we don't think Storage Foundation API is going anywhere"? https://www.chromestatus.com/feature/5670244905385984
And even though this is a draft created and authored exclusively by Googlers, even other Googlers are confused: https://github.com/WICG/storage-foundation-api-explainer/iss...
Well, I could not help myself.
>> While versions of Safari, Chrome, and Opera support a technology called Web SQL Database, which uses SQL statements as string arguments passed to a JavaScript API, we think developer aesthetics are an important consideration, and that this is a particularly inelegant solution for client-side web applications.
>> We .. also spoke with Microsoft, who agree with us that IndexedDB is a good option for the web
OMG!! MS thought not doing what Apple and Google wanted was a good option. Earth shattering.
As for the code examples, both sets look horrible, but this and "_developer aesthetics_" don't go together. This is how they think their `JOIN` is "better" than SQL `JOIN`.
It's built on top of Sqlite in Firefox, it isn't "low level". Give me Sqlite directly instead of that horrible stuff.
It made me really angry at Mozilla. WebSQL was so invaluable to me.
If you need to store more data, why not do it on the server? What are the use cases for having a full-blown relational db client-side?
So is PHP according the infallible SV pundits.
> it's also unnecessary, as you can store a significant amount of data in localStorage and the API is extremely simple to use.
Either you've never actually tried to store significant amounts of data in localstorage, or you're from the era of "640k is enough".
To say nothing of having no indexing functionality with localstorage, or paying the 33% base64 tax to put anything remotely binary into it.
> If you need to store more data, why not do it on the server?
"Because there is always, ALWAYS a high speed, low latency 100% uptime connection" /s
In answer to your last question, we have developed two PWAs reliant on local databases.
1. Was a product catalogue for use in stores, letting users filter down to select a particular appliance. two key requirements were that not all stores had great wifi coverage, and they wanted to easily update the catalogue (which meant no app recompilation).
So, Dexie atop IndexedDB, with indexes on the various facets for searching, and a simple settings page to run a sync. Dexie stored the product data, and a "immutable" SW cache strategy was used to keep the images available.
2. A detailed report generation app, Key requirement, must work with patchy connectivity as reports need to be taken in the field with unknown signal availability. Reports are locally stored and synced periodically (connection permitting). Again, Dexie atop IndexedDB, this time the images stored in the db as blobs prior to upload.
Easier / cheaper to host only static files.
I must admit there's a bit of schadenfreude in knowing that Firefox is almost irrelevant now, while SQLite keeps going from strength to strength.
> I have identified the regression point and assigned the bug to somebody. Thank you for the bug report. We will try and get this fixed ASAP.
I just added a section on Safari's storage related bugs, of which there are many. Amazingly, IndexedDB broke in Safari 14.1.1 after localStorage broke in Safari 14.1, released a month earlier.
Bugs.
Especially for a browser engine where you have a ridiculous number of testing permutations to cater for.
This is something that should have been caught by automated tests.
There’s basically no automated test suite even for CSS 2.1 that all vendors can rely on that matches behavior specified by the standards.
Luckily standards like that are so mature that it’s not needed as badly, but it definitely keeps incumbents safe from competition.
But for others who don't know:
https://web-platform-tests.org/
https://github.com/web-platform-tests/wpt/tree/master/css/CS...
Because a cursory glance at Chromium src tells me they do have automated tests for IndexedDB:
https://source.chromium.org/chromium/chromium/src/+/main:con...
Testing browsers is a daunting task. A modern browser is 10-15 million lines of code with literally thousands of exposed APIs [2]
[1] https://www.pcgamer.com/a-google-chrome-update-breaks-the-au...
[2] https://web-confluence.appspot.com/#!/confluence
My favorite one was websites added to the home screen on an iPad would have the clock displayed above the websites if you rotate the iPad after opening a website. It wasn't fixed for years and it may still be there.
As with every race condition (cross process, too, so tools like TSAN don't help), testing for the absence of race conditions is hard.