Readit News logoReadit News
jdvh commented on Lessons learned from building a sync-engine and reactivity system with SQLite   finkelstein.fr/sqlite-syn... · Posted by u/engmarketer
jdvh · 25 days ago
We've spend the last couple of years building a local-first end-to-end encrypted multiplayer notes app (Thymer). We also explored running sqlite in the browser but ultimately concluded that we didn't really sqlite or a real database at all in the browser.

What does a document system need? A few simple indexes that track links and backlinks, mentions, and hashtags. You’re broadcasting change notifications anyway (for reactive DOM updates), so updating the indexes for those manually isn’t much extra work. You apply updates eagerly and then notify other users so they can apply the same updates, with some rules to guarantee that all users end up with the same state regardless of the order in which they receive the updates. But a relational database doesn’t help with any of this. Document systems tend to be versioned, so every user action turns into another entry in a transaction log. Even queries like “last Monday’s version of my document” don’t map naturally to SQL. You can query for transactions in a given time period, but unless you have snapshots, you’re still forced to re-apply transactions all the way from t=0 if you want to re-create the document state at a given date.

jdvh commented on Tell HN: Notion Desktop is monitoring your audio and network    · Posted by u/HoyaSaxa
bschne · 2 months ago
Most intriguing thing in that vein I've seen: https://thymer.com (haven't used it, am not affiliated, just looked promising in a demo video esp. on performance grounds)
jdvh · 2 months ago
Hey thanks for mentioning us!

With Thymer we really care about performance, but Thymer is also end-to-end encrypted because we don't want to compromise on privacy. And it's real-time collaborative and offline first.

Thymer has optional self-hosting. Then you can upgrade (or not) at your own leisure, or intentionally stick to an older version you like better. Enshittification is a big problem in our industry. We've all been burned by it -- we certainly have -- and being able to opt out of a "new and improved!" version is a real feature.

Thymer will also be very extensible. Today we launched our plugin SDK: https://thymer.com/plugins and https://github.com/thymerapp/thymer-plugin-sdk/ with a bunch of examples. With Thymer you will be able to "vibe code" the very simple plugins and with VSCode/Cursor you can make more complex plugins with hot-reload.

jdvh commented on First-Class Models: The Missing Productivity Revolution   frest.substack.com/p/firs... · Posted by u/surprisetalk
jancsika · 2 months ago
> But having a version tree, forking and merging, having to deal multiple timelines and the graphs that represent them -- that's where you lose people.

There's no good reason why that should be the case. E.g., one could imagine the guts of the "copy-pasting files" UI being a VCS. That would keep the original 100% of the userbase plus allow whatever percentage to level up if/when the need arises (or go "back in time" in the event of a major screw-up).

It's just that software UX in 2025 is typically very bad. The real axiom: the longer you run an application, the more likely it will do the opposite of its intended purpose.

Oops, the word "stash" in git has an idiosyncratic meaning. That content has been removed from the history I was trying to keep. Fuck.

Oops, "Start" in Windows pauses interactivity and animation until ads are ready to be displayed in the upcoming dialog. Fuck!

Especially in the latter case, I don't think users are deterred by the cognitive load required to interact with the interface. It's probably more a case of them being deterred because the goddamned stupid thing isn't doing what it's supposed to.

jdvh · 2 months ago
In theory you can have these "zero cost abstractions" but in practice I don't think so. The user manual gets thicker. Concepts like 'delete permanently' and backup/restore get more complicated. Users will get confronted by scary "advanced users only" warnings in the interface. Some enthusiast blogger or youtuber will create content highlighting those advanced features and then regular users will get themselves in trouble. Customer support gets way more complicated because you always have to consider the possibility that the user has (unknowingly) used these advanced features. If you put buttons in the interface users will press those buttons. That's just a fact of life. Advanced features always come at a cost. Sometimes that cost is worth it, but only sometimes.
jdvh commented on First-Class Models: The Missing Productivity Revolution   frest.substack.com/p/firs... · Posted by u/surprisetalk
jdvh · 2 months ago
Rule of thumb: every 10% increase in complexity cuts your potential user base in half.

This is why people make backups by copy-pasting files. This is why Excel is so dominant. This is why systems like hypercard and git are not mainstream and never will be.

There is a large universe of tools people would love if only they would bother to learn how they worked. If only. Most people will just stick to whatever tools they know.

For most people the ability to go back and forward in time (linear history) is something they grasp immediately. Being able to go back in time and make a copy also requires no explanation. But having a version tree, forking and merging, having to deal multiple timelines and the graphs that represent them -- that's where you lose people.

jdvh commented on Sync Engines Are the Future   instantdb.com/essays/sync... · Posted by u/GarethX
dboreham · 6 months ago
That's not how the CRDT concept works.
jdvh · 6 months ago
You're free to argue that this isn't "pure" CRDT, but the CRDT algorithm still runs normally, just a bit later than it otherwise would.
jdvh commented on Sync Engines Are the Future   instantdb.com/essays/sync... · Posted by u/GarethX
jakelazaroff · 6 months ago
Unfortunately, a hard constraint of (state-based) CRDTs is that merging causally concurrent changes must be commutative. ie it is possible that clients will not be able to agree on the order of CRDT operations, and they must be able to arrive at the same state after applying them in any order.
jdvh · 6 months ago
I don't think that's required, unless you definitionally believe otherwise.

When clients disagree about the the order of events and a conflict results then clients can be required to roll back (apply the inverse of each change) to the last point in time where all clients were in agreement about the world state. Then, all clients re-apply all changes in the new now-agreed-upon order. Now all changes have been applied and there is agreement about the world state and the process starts anew.

This way multiple clients can work offline for extended periods of time and then reconcile with other clients.

jdvh commented on Sync Engines Are the Future   instantdb.com/essays/sync... · Posted by u/GarethX
mrkeen · 6 months ago
I've looked at CRDTs, and the concept really appeals to me in the general case, but in the specific cases, my design always ends up being "keep-all-the-facts" about a particular item. But then you defer the problem of 'which facts can I throw away?'. It's like inventing a domain-specific GC.

I'd love to hear about any success cases people have had with CRDTs.

jdvh · 6 months ago
It's still early, but we have a checkpointing system that works very well for us. And once you have checkpoints you can start dropping inconsequential transactions in between checkpoints, which you're right, can be considered GC. However, checkpointing is desirable anyway otherwise new users have to replay the transaction log from T=0 when they join, and that's impractical.
jdvh commented on Sync Engines Are the Future   instantdb.com/essays/sync... · Posted by u/GarethX
danielvaughn · 6 months ago
CRDTs work well for linear data structures, but there are known issues with hierarchical ones. For instance, if you have a tree, then two clients could send a transaction that would cause a node to be a parent of itself.

That said, there’s work that has been done towards fixing some of those issues.

Evan Wallace (I think he’s the CTO of Figma) has written about a few solutions he tried for Figma’s collaborative features. And then Martin Kleppmann has a paper proposing a solution:

https://martin.kleppmann.com/papers/move-op.pdf

jdvh · 6 months ago
As long as all clients agree on the order of CRDT operations then cycles are no problem. It's just an invalid transaction that can be dropped. Invalid or contradictory updates can always happen (regardless of sync mechanism) and the resolution is a UX issue. In some cases you might want to inform the user, in other cases the user can choose how to resolve the conflict, in other cases quiet failure is fine.
jdvh commented on Sync Engines Are the Future   instantdb.com/essays/sync... · Posted by u/GarethX
mentalgear · 6 months ago
Optimally, a sync engine would have the ability to be configed to have the best settings for the project (e.g. central server or completely decentralised). It'd be great if one engine would be so performant/configurable, but having a lot of sync engines to choose from for your project is the best alternative.

btw: excellent questions to ask / insights - about the same I also came across in my lo-fi ventures.

Would be great if someone could assemble all these questions in a "walkthrough" step-by-step interface and in the end, the user gets a list of the best matching engines.

Edit: Mh ... maybe something small enough to vibe code ... if someone is interested to help let me know!

jdvh · 6 months ago
Completely decentralized is cool, but I think there are two key problems with it.

1) in a decentralized system who is responsible for backups? What happens when you restore from a backup?

2) in a decentralized system who sends push notifications and syncs with mobile devices?

I think that in an age of $5/mo cloud vms and free SSL having a single coordination server has all the advantages and none of the downsides.

jdvh commented on Local-First and Ejectable   thymer.com/local-first-ej... · Posted by u/rzk
danjl · 6 months ago
If you use a local, browser-based database, and allow your app to be installed as a PWA, then you are ejectable? The only "server" is just a host for HTML+JS that your browser automatically downloads when you visit the site. The app runs locally, like a desktop app, inside your browser. No need for the items listed in the article.
jdvh · 6 months ago
You can think of ejectable as the opposite of being locked-in.

u/jdvh

KarmaCake day1705October 2, 2008
About
thymer.com - a local first, end-to-end encrypted task and planning app. Syncs with CRDTs. Optionally self-hosted. Programmable.
View Original