I got to the end of that without really understanding what this is solving, what it does, or how.
How do you handle changing upstream files locally without forking? Do you just, keep changes in a separate configuration format that is applied lazily at built time?
I've never had issues with maintaining a fork anyways.
The main advantage over plain "patch" is that it is more powerful in the face of upstream changes. For example if you rename the upstream file, you have no good way to represent this in .patch, but that project allows is. There also a way to specify patch using function name, which should make it more robust in face of upstream changes.
As for my opinion, this seems like an incremental improvement over existing tools. I'd prefer a simple shell script that does "git checkout ... && mv ... && cp .. && patch ..." over something fancy like this.
Don't we just call these out-of-tree patches? Lots of those for Linux lying about, it's not a new idea. I guess the difference is that they have multiple upstreams, so really they're more of a new project that consists of:
- A set of sqlite patches,
- Other upstreams and patches?
- A custom toolchain to build all the above together into one artefact
If a trunk is a series of patches then isn’t maintaining a series of patches against multiple upstreams just maintaining multiple forks? Feels like mostly semantics to try to differentiate at this point. I mean what is the thing that you would get from doing it this other way? Isn’t it fundamentally still going to require pulling upstream changes regularly and resolving conflicts when they occur? Reading the treatise they say “this allows us to address conflicts that normally you would have to resolve manually”. So Git has tools to pick and choose and auto resolve conflicts, you just have to customize the behavior of your tooling a bit.
Seems like they’re just ditching the inbuilt tools git/github offers to achieve this and doing the exact same thing with custom bespoke tooling. Instead of doing that, I’d be more interested in leveraging what Git has baked in with some sort of wrapper tool to perform my automation with some bespoke algorithm. There are merge drivers and strategies end users can implement themselves for this kind of power user behavior that don’t require some weird reinvention of concepts already built into Git
let's say you need to copy config.h.in to config.h and patch a few lines. In case of "subtree merge", you will have to do it manually, every time you upgrade upstream.
I feel like this best explains what added benefits LumoSQL tries to add:
> LumoSQL is a derivative of SQLite, the most-used software in the world. Our focus is on privacy, at-rest encryption, reproducibility and the things needed to support these goals. [...]
The home page says, "LumoSQL can swap back end key-value store engines in and out of SQLite. LMDB is the most famous (but not the only) example of an alternative key-value store, and LumoSQL can combine dozes of versions of LMDB and SQLite source code(...)" and also, "Encryption and corruption detection, optionally per-row".
> LumoSQL exists to demonstrate changes to SQLite that might be useful, but which SQLite probably cannot consider for many years because of SQLite's unique position of being used by a majority of the world's population.
> SQLite is used by thousands of software projects, just three being Google's Android, Mozilla's Firefox and Apple's iOS which between them have billions of users. That is a main reason why SQLite is so careful and conservative with all changes.
That's a great perspective. How well does the SQLite team work with them? How well does it work in production, especially if you need SQLite compatibility? And
How does LumoSQL deal with locking? If I proposed Wildcat as a storage engine which is lockless. Can LumoSQL be optimized for atomicy and multi writer? C shared library too if you're curious https://wildcatdb.com
I've played with SQLite when it was still available in-browser, and I felt that was on the brink of being a game-changer. If it was still supported in-browser and we had replication from the browser, peer-to-peer, I think we'd be living in a much more useful world. It's a lovely tech, but I never built anything serious around it. At this point, as a front-end web technology that seems to be gone. I know I could conceivably use it to back a NodeJS server, keep all the data in memory and local files, but I don't see a great use case for that. I do lots of small projects that could use SQLite, but I usually scaffold them around a single shot Mysql DB for testing, which is easy to spin up and easy to detach from any given back-end instance. So I'm not sure what I'd gain by trying to make a tiny databse on the back-end live in Sqlite files. I'm totally enchanted by stuff like Litestream, and I'm actually dying to find a reason to try it. But every good use case for Sqlite that I could think of sort of died when it stopped being a viable client-side store.
TL;DR, what are people using SQLite for? What's the advantage over spinning up a tiny MySQL instance in a cloud somewhere, where you don't have to deal with managing replication and failover by yourself?
Chances are you have an installation or several running in your pocket right now. It's one of the most widely deployed pieces of software in existence. It's not supposed to be a 'traditional' DB in that running a webapp for many users sense (although it can do that), but to back client based software that need a data store/query tool and don't want to implement their own.
Agree with all of this. Also want to add that SQLite is a perfect jumping off point for full stack web-devs doing react-native (or similar) and want a familiar data query pattern they were already used to from MySQL, Postgres, etc.
Having a consistency of SQL everywhere is really appealing for data management.
> All supported versions of Windows support SQLite, so your app does not have to package SQLite libraries. Instead, your app can use the version of SQLite that comes installed with Windows.
That's quite fun to know! Although, it's also funny that I downloaded the demo .acorn files on a Mac and the OS has no idea how to open them without searching the App Store.
I feel like a JSON file would be more compact and easier to read, but wtf do I know. Harder to query, I guess?
1. There's almost certainly a port of Sqlite3 to WASM that would be more than glad to run in your browser.
2. I'd really love to know what applications fit in the "we had replication from the browser, peer-to-peer, I think we'd be living in a much more useful world" situation. We've had GunDB, IPFS, etc. that live in the browser for decades (and projects like Urbit), and the killer app just... doesn't seem to exist? Let alone anything useful as just a basic demo? Anyone have anything to point to? I just don't see it, personally.
Heh. Well, #2, brilliant question. But no, I'm not thinking of anything as sexy as totally distributed filesystems. 15 years ago when I was into crypto and ran a bitcoin casino I would have had much bigger ideas for fully distributed DBs (which surely would have tanked and caused me ruin). Currently, I deal with a lot of site-specific software installations that run their own MySQL servers, some of which have unexpected downtime or go offline. I have a lot of custom code to align them with a master source of truth when they come back up. At least a few times a year, one of them gets so corrupted that I have to just login remotely and rebuild its database by hand. If I could have designed them to share a single database peer to peer, it would have saved me a lot of personal time.
There are probably a lot of hub-and-spoke systems like this flying way under the radar that would be a lot better if there were a reliable technology to keep them synchronized. I keep looking at Litestream and thinking about it.
Most of my python projects use SQLite; 1 exception where i need multiprocessing access to the database and no locking problems and speed so i need to run the entire db in memory.
The built in library makes it really quick and easy to use it. Whereas mysql or in my case id use postgresql if i needed a full db. You're looking for a third party library? I have used Psycopg before but its just not needed.
Yes, ive come up against the sqlite locked database performance troubles; and failed to actually get the multi user thing working properly. But I moreso just needed to reapproach the issue.
My new startup http://mapleintel.ca is db.sqlite3 based. thousands of lines in it so far and growing every day.
I've used sqlite3 in node, and it's nice and performant for small cases, yes. Mostly I've used it for things small enough where a user could download an entire sqlite file of their data, and then re-upload it in case their data got lost. But ultimately this data gets stored in a true MySQL DB. I don't think I'd trust it to run a whole system with thousands of users and millions of entries... honestly, maybe my issue is that I don't trust NodeJS enough...
Definitely can still be used on a client, you just have to be creative with running it. https://github.com/orbitinghail/sqlsync uses rusqlite compiled to WASM within a Web Worker, for instance.
Most uses of SQLite are client-side apps. Basically everything that's not web: from mobile apps and desktop apps to embedded software in things like cars, tvs, kiosks, etc. There are probably more apps using SQLite than not using SQLite for these kind of apps.
Are these things just running sql.js? Doesn't that use a kind of unstable webstorage instead of writing client-side files? I don't have a good handle on the state of SQLite these days as a way to store semi-permanent data on the client. In a locked-in environment or a backend I feel like it might make sense, but... isn't there like a 50Mb limit on localStorage, and how does that play nicely with a potentially larger DB...?
The general test suite is not proprietary, and is a standard part of the code. You can run make test. It uses TCL to run the testing, and covers virtually everything.
There is a separate TH3 test suite which is proprietary. It generates C code of the tests so you can run the testing in embedded and similar environments, as well as coverage of more obscure test cases.
Readit News
1. https://lumosql.org/src/not-forking/doc/trunk/README.md
How do you handle changing upstream files locally without forking? Do you just, keep changes in a separate configuration format that is applied lazily at built time?
I've never had issues with maintaining a fork anyways.
The main advantage over plain "patch" is that it is more powerful in the face of upstream changes. For example if you rename the upstream file, you have no good way to represent this in .patch, but that project allows is. There also a way to specify patch using function name, which should make it more robust in face of upstream changes.
As for my opinion, this seems like an incremental improvement over existing tools. I'd prefer a simple shell script that does "git checkout ... && mv ... && cp .. && patch ..." over something fancy like this.
- A set of sqlite patches,
- Other upstreams and patches?
- A custom toolchain to build all the above together into one artefact
Seems like they’re just ditching the inbuilt tools git/github offers to achieve this and doing the exact same thing with custom bespoke tooling. Instead of doing that, I’d be more interested in leveraging what Git has baked in with some sort of wrapper tool to perform my automation with some bespoke algorithm. There are merge drivers and strategies end users can implement themselves for this kind of power user behavior that don’t require some weird reinvention of concepts already built into Git
But if you give this a cool name, it's a New Idea.
Made by the same people who brought us SQLite.
We really need a way to customise software at the source code level without forking.
Instead of having a fork, you have a configuration that says "checkout x then apply changes a,b,c".
> LumoSQL is a derivative of SQLite, the most-used software in the world. Our focus is on privacy, at-rest encryption, reproducibility and the things needed to support these goals. [...]
https://lumosql.org/src/lumosql/file?name=doc/project-announ...
I'm unsure what Phase 1 was about, or if there is a planned Phase 3, but seems to outline what they're currently aiming for at least.
=>
"LumoSQL can swap SQLite backend, with Key-value store engines"
===
"LMDB is the most famous (but not the only) example of an alternative key-value store"
=>
"We currently only support LMDB as an alternative KV store"
===
"and LumoSQL can combine dozes of versions of LMDB and SQLite source code like this:"
=>
"LumoSQL will allow you to use different versions of SQLite and LMDB in parallel as different backends"
> SQLite is used by thousands of software projects, just three being Google's Android, Mozilla's Firefox and Apple's iOS which between them have billions of users. That is a main reason why SQLite is so careful and conservative with all changes.
That's a great perspective. How well does the SQLite team work with them? How well does it work in production, especially if you need SQLite compatibility? And
https://github.com/hypermodeinc/badger
I've played with SQLite when it was still available in-browser, and I felt that was on the brink of being a game-changer. If it was still supported in-browser and we had replication from the browser, peer-to-peer, I think we'd be living in a much more useful world. It's a lovely tech, but I never built anything serious around it. At this point, as a front-end web technology that seems to be gone. I know I could conceivably use it to back a NodeJS server, keep all the data in memory and local files, but I don't see a great use case for that. I do lots of small projects that could use SQLite, but I usually scaffold them around a single shot Mysql DB for testing, which is easy to spin up and easy to detach from any given back-end instance. So I'm not sure what I'd gain by trying to make a tiny databse on the back-end live in Sqlite files. I'm totally enchanted by stuff like Litestream, and I'm actually dying to find a reason to try it. But every good use case for Sqlite that I could think of sort of died when it stopped being a viable client-side store.
TL;DR, what are people using SQLite for? What's the advantage over spinning up a tiny MySQL instance in a cloud somewhere, where you don't have to deal with managing replication and failover by yourself?
Having a consistency of SQL everywhere is really appealing for data management.
Personally I use it a bunch in mobile and desktop apps.
> All supported versions of Windows support SQLite, so your app does not have to package SQLite libraries. Instead, your app can use the version of SQLite that comes installed with Windows.
https://learn.microsoft.com/en-us/windows/apps/develop/data-...
I feel like a JSON file would be more compact and easier to read, but wtf do I know. Harder to query, I guess?
1. There's almost certainly a port of Sqlite3 to WASM that would be more than glad to run in your browser.
2. I'd really love to know what applications fit in the "we had replication from the browser, peer-to-peer, I think we'd be living in a much more useful world" situation. We've had GunDB, IPFS, etc. that live in the browser for decades (and projects like Urbit), and the killer app just... doesn't seem to exist? Let alone anything useful as just a basic demo? Anyone have anything to point to? I just don't see it, personally.
There are probably a lot of hub-and-spoke systems like this flying way under the radar that would be a lot better if there were a reliable technology to keep them synchronized. I keep looking at Litestream and thinking about it.
you're doing something wrong if that is easier than using sqlite
> What's the advantage over spinning up a tiny MySQL instance in a cloud somewhere
one advantage is your thing will work without needing network access
https://docs.python.org/3/library/sqlite3.html
The built in library makes it really quick and easy to use it. Whereas mysql or in my case id use postgresql if i needed a full db. You're looking for a third party library? I have used Psycopg before but its just not needed.
Yes, ive come up against the sqlite locked database performance troubles; and failed to actually get the multi user thing working properly. But I moreso just needed to reapproach the issue.
My new startup http://mapleintel.ca is db.sqlite3 based. thousands of lines in it so far and growing every day.
Managing profiles and inventory in a solo game where crafting results are random and I don't like limited inventories.
How does a "fork" like this be tested if everything stays working and compatible to upstream after the change?
There is a separate TH3 test suite which is proprietary. It generates C code of the tests so you can run the testing in embedded and similar environments, as well as coverage of more obscure test cases.
https://sqlite.org/th3.html