> no matter how small the edit was, the entire file gets rewritten
SQLite doesn't fix this, because you would still need to encrypt the whole file (at least with standard sqlite). If you just encrypted the data in the cells of the table, then you would expose metadata in plaintext.
SQLCipher does provide that, but as mentioned by others, it isn't quite the same thing as sqlite, and is maintained by a different entity.
> The primary issue is that new features cannot be added natively to the XML tree without causing breaking changes for older clients or third-party clients which have not adopted the change yet.
That isn't a limitation of xml, and could also be an issue with sqlite. The real problem here is if clients fail if they encounter tags or attributes they don't recognize. The fix here is for clients to just ignore data it doesn't know about, whether that is xml or sqlite.
The complaints about compatibility between different implementations would be just as bad with sqlite. You would still have some implementations storing data in custom attributes, and others using builtin fields. In fact it could be even worse if separate implementations have diverging schemas for the tables.
> Governance Issues
None of this has anything to do with sqlite vs xml. It is a social issue that could be solved without switching the underlying format, or remain an issue even if it was changed.
The biggest weakness is the cost. Each client would have to purchase an expensive license. The source code is provided upon purchase though, but essentially destroys the ability to build a client from source due to the compiled binary distribution.
I'm not sure why it's a concern that the whole file needs a rewrite.
Naiively perhaps I thought that was helpful with solid state storage because it means that old data is trimmed faster?
It mentions it near the entire file being in memory but that seems a dubious concern. If the key is in memory the entire file can be comprised either way. Nothing can really stop that if you have access to the programs memory.
It's an issue if you have a really big password database, for example because you are storing large attachments in there. Especially if you are also syncing the file over the network. Also if your file is multiple GB, having it all in memory is an issue because of the amount of memory used.
That isn't really how keepass is meant to be used, but apparently people do use it this way.
> Devising a new schema based on SQLite would allow for current features that are being jerry-rigged into the attributes to have their own real place in the database
Perfectly possible with XML too
> An SQLite based store is one of the most tested and optimal formats for document and application storage
It's optimized for things that largely don't matter for password storage. The testing is admirable, but there's no issue of keepass clients crashing or corrupting data so again, not very relevant (probably because of low concurrency, simple writes etc).
> A switch this big is a major chance to fix the governance structure and align it more with a democratic consortium than a benevolent-dictator-for-life style of project management
You don't need a technical change to solve this. In fact, a fork that would fracture clients is the last thing you need when making governance changes.
> So many quality of life features can be added where the old schema disallowed it
All of the features they list can be achieved with an XML format. The format isn't what's holding them back.
I don't know if this could count as "corrupting": I made the mistake of syncing my keepassxc database to my macbook with finder webdav client (nextcloud backend) it read the file alright but when I tried to write a new secret it helpfully wrote an empty file in place, wiping nextcloud file versions in the process.
Thankfully, Nextcloud was smart enough to move the previous file in the trash bin and I could restore it.
It seems keepassxc "save" procedure here was to delete the old file and replace it with a new one and something went catastrophically wrong in the process ?
Looking at the settings there's is a parameter for this method for particular circumstances but I didn't enable it back then. Now I just have a second database only on this mac synced to icloud and never letting it near my nextcloud again.
I actually had a similar thing happen with gvfs-fuse (Google drive). It was a bug in gvfs using the quota usage of a file as it's file size (because libgdata didn't provide a method to get file size), but I was using a file shared to me so it had zero quota usage.
All of which is to say I would bet on something in the webdav-nextcloud line being at fault instead of keepassxc.
Not a problem for XML per se (you can work with byte positions, and with fixed-size blocks to avoid resizing/relocation), but in the case of KDBX there is the issue that it is encrypted as a whole. Not encrypting as a whole, on the other hand, risks leaking more information about the contents, like you can see which parts/how much changed between one update and the next.
I actually prefer this. It's how most user-facing file formats work. KDBX in particular is often used in conjunction with syncing software, and I don't want a half changed file to sync and then the connection to be lost. The usual paradigm of "write new file then move and replace over old file" works more safely for atomic changes.
The schema issues aren't solved by moving to sqlite, or the proposed solution is doable with XML too. I can see the same thing with the attributes (some described it as a shadow schema) happening in an attributes table just as easily. And in my experience relational schemas are a lot harder to modify than a document schema like XML.
EDIT: also you don't need to have just one password vault and I'd say you probably shouldn't, separate entries also assist with separation of concerns. This last adds a little overhead but is a reasonable workaround.
However on the whole I like sqlite for app persistence. It can, however, leak data (forensically) if not managed properly.
Is kdbx broken or has it been causing data loss? I've been using KeepassXC as long as it has existed and no issues for me thus far. If kdbx is not problematic for it's intended use then I think moving to SQLite just makes it more hacker friendly which I have no need for. I have no need for other applications extending the use of my passwords.
If anything, maybe give people the option to export to SQLite and then use that going forward but keep it entirely optional.
Exporting to a relational db is beside the point. You can very easily just export a CSV file and double click it in DB Browser and have something working. Or you could export the XML document and write a quick importer for it. But genuinely what benefit does it provide other than allowing me to hack at my unencrypted db with sql statements? All the benefits that I was referring to are for daily usage of a KeePass client, not for a one off export.
But genuinely what benefit does it provide other than allowing me to hack at my unencrypted db with sql statements?
None at all. That is why I am saying it should be entirely optional so most people can just ignore it and those wanting the alternate schema can get their feature request fulfilled.
I've glanced over the article - I think using SQLCypher makes sense if it benefits Keepass project internally, meaning makes it easier to implement new features or fix bugs.
I do not see how it would benefits end users in any significant way. Sure, you can look inside the database, how many people need that?
Dumping the database to CSV is not a good backup, schema changes over time, what was dumped from one version of the app would not work for importing into another version if schema changes. Backup it needs a versioned schema format, which would actually look like KDBX format if implemented in XML :-)
I don't want to assume you didn't read the article, but this isn't really about the database engine. It's about the shadow schema that has grown up around the format. The database switch would serve as a flag day to unify things. It won't be a permanent fix, nothing at this scale ever is, and we'll probably need another migration in a few decades. Still worth doing.
I read it and I get it but I have never run into problems with KP using its existing schema. The only reason I could see this debate making sense would be if third parties want to integrate with it otherwise it works perfectly fine. And sure if it's worth doing then make it optional like I said. Have an export as SQLite, MySQL, Postgres, Oracle, DB2 but keep it optional given there is no need otherwise. This is solving a perceived problem that does not exist or creating a problem where there isn't one.
As for scale this app is for one person doing one query at a time or saving one password at a time. One person will not be saving millions of passwords or if for some reason they are then they can export to a format that an enterprise solution could take it's place. People have written team / company password managers that can use Oracle, MySQL and Postgres that can track and audit user and team password changes. This is not something that should ever be expected of an individual personal password manager like KeepassXC.
This is just my take but I would rather have the maintainers of KP focus on bugs, usability issues, quality of life enhancements which they have been doing great thus far in my opinion. Forcing a schema migration is just asking for trouble and potentially causing bugs that turn some people away from using it or cause people to lose data if they do not have snapshots of their database. Or if forcing a schema change make for damn sure there are many backups in each format and encourage the users to back up all the files to many external encrypted drives and store some of the encrypted devices off-site. Something everyone should be doing regardless.
I think that's where I see the most concerns with this proposal: XML is already an extensible format (that's what the X stands for) and XML schema changes should be simpler than needing to run SQL migrations (alter/drop/etc). A switch to SQL doesn't guarantee that schema extensions better align as standards or allow for easier schema modifications. I think it more extends the risk that schema components get ossified and harder to extend.
That problem that the current schema doesn't have enough ways to declare custom memory-protected fields outside of user-facing attributes seems just as likely to be replicated and maybe worsened in an SQL schema.
Changing the database engine doesn't fix the architecture problems nor schema migration problems. It's certainly a good time to reevaluate the architecture problems and the schema migration problems. But the huge caution I'd suggest here would be look at the ossification complaints about the current XML schema and expect SQL migrations to be worse and plan for worse multi-schema operations and intenser ossification. (Especially because these files are expected to live in a large multi-vendor ecosystem, SQLite schema migration management is going to be much worse than any XML schema management.)
Maybe it's irrational, and I cannot actually justify it (and of course safe writing is of primary importance), but somehow rewriting the whole file feels like a good thing for a secrets storage. Updating only part of a file obviously reveals something, even though it probably shouldn't matter if it doesn't reveal anything useful. But the default mode of thinking is we can never assume the leaked information cannot be used somehow.
KDBX seems unique in this threat model out of the major providers. The cloud ones all use a relational DB, while major local ones like Enpass & Codebook just use a SQLCipher store. I wish someone with some real experience here would chime in: What metadata does a SQLCipher DB leak that a KDBX file does not? I mention that both of them obviously reveal the size of your vault to an attacker (w/ KDBX reporting the size more accurately, ultimately irrelevant), which is pragmatically unavoidable information leakage.
I do use it, and rewriting the whole file annoys me especially when the storage is not local and the database contains sizable blobs. For storing passwords and short secrets, it makes little to no difference but if I have 10 1MB blobs stored in there, it becomes upsetting.
Well, yes, this is what OP is saying, and I'm not arguing against that. However, this is not what *.kdbx was designed for. And I am only talking about what cryptographically changes for the intended use case if we encrypt every page separately.
> So rather than risking sync issues uploading your 20MiB KDBX file on every minor change, you can upload just the 4KiB or so comprising that data.
Why is your KDBX file 20 MiB? It seems you are storing something that is not actually a good fit for a password manager, and expecting the entire world to change around you instead of storing those files in a more appropriate place.
I don't use KP, but I have a pdf for my floor safe in my password manager. I only open it a few times per year and I need more than just the combination, I need instructions on whether the first number is cw or ccw. While I could no doubt look it up on the internet every time, I was fearful that the user's manual might some day disappear from the internet. Some things that aren't obviously passwords still belong in a password manager.
In my mental model, the PDF is not a secret and can be stored anywhere -- encrypted, if desired, but it sounds like a public document.
The safe combination is a secret, and obviously belongs in secret storage.
In this specific example, if I had trouble remembering whether the first number of the combination was reached via cw or ccw rotations, I'd include that in the secret, e.g. "cw34-12-22-45".
(Some safe combinations require multiple rotations. I unintentionally became the owner of one that is something like "cw3x34-ccw2x12-cw5x22-ccw2x45". I still can't open it actually, but that pattern is what the Internet tells me. :)
Were that me (I used KP), it would be in a different kdbx file. This is one of the benefits of KP, I have about 8 different vaults for various things. I don't like putting my eggs in one basket.
We have a single kdbx with nearly 7000 entries. It is about 45MB at the moment with very few text docs in it. It once got to over 100MB when I found people using it as a doc store but it had a bit of a clear out a few years back.
"The reasonable man adapts himself to the world; the unreasonable man persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man."
My kdbx is only 173 KB, but I don't hate this idea.
Some decent arguments for development concerns, but the users will certainly never notice.
For 10 years I've managed a family of 4 in a single KeePass db. Unique passwords across all accounts, random passwords instead of PII for "security questions", fake DOBs, and all other random security related stuff. Never had a single issue. Everything happens instantly as far as I can tell. And the file is 67kb.
My work dbs, separate only because they should be, are much smaller and simpler.
I love KeePass. Changing everything under the hood probably only has potential to make pain for the user. Best case is that nothing is made noticeably worse. Doesn't seem worth the risk.
Of course, the technical backbone of a product is rarely a user concern. But, changing the store allows for new design decisions when making the schema that are not tied to the old (IMO) technical debt of the KDBX format.
Users have wanted proper custom entry templates for years in order to input their card details into the manager. BitWarden, CodeBook, and others all have this built in without needing to fiddle with attributes and autofill to make it work.
I think a major change like this is worth it when we consider the KeePass of the next decade or two. This is genuinely my most important file by a longshot. Basing it on SQLite is just the right move for future proofing.
I'm working on an alternative to KeePass/KeepassXC called Lockstep - it is local-first password manager that supports sync natively.
It uses SQLCypher as local data store and keeps KeePass-compatible data model - supports import of .kdbx database.
SQLCypher and different schema used in Lockstep alone did not solve any problems that I have with Keepass. Those problems are sync and sharing.
Solving sync and sharing cannot be done on whole database file level, as it implemented now in KeePass. Changes need to be tracked at the password record level, all changes need to persisted as operations log and that log needs to be distributed across devices.
The above means writing a whole protocol, and that's a lot more work than changing local storage.
Readit News
SQLite doesn't fix this, because you would still need to encrypt the whole file (at least with standard sqlite). If you just encrypted the data in the cells of the table, then you would expose metadata in plaintext.
SQLCipher does provide that, but as mentioned by others, it isn't quite the same thing as sqlite, and is maintained by a different entity.
> The primary issue is that new features cannot be added natively to the XML tree without causing breaking changes for older clients or third-party clients which have not adopted the change yet.
That isn't a limitation of xml, and could also be an issue with sqlite. The real problem here is if clients fail if they encounter tags or attributes they don't recognize. The fix here is for clients to just ignore data it doesn't know about, whether that is xml or sqlite.
The complaints about compatibility between different implementations would be just as bad with sqlite. You would still have some implementations storing data in custom attributes, and others using builtin fields. In fact it could be even worse if separate implementations have diverging schemas for the tables.
> Governance Issues
None of this has anything to do with sqlite vs xml. It is a social issue that could be solved without switching the underlying format, or remain an issue even if it was changed.
https://www.sqlite.org/see/doc/trunk/www/readme.wiki
A weakness though, again, is that this is closed source...
Naiively perhaps I thought that was helpful with solid state storage because it means that old data is trimmed faster?
It mentions it near the entire file being in memory but that seems a dubious concern. If the key is in memory the entire file can be comprised either way. Nothing can really stop that if you have access to the programs memory.
That isn't really how keepass is meant to be used, but apparently people do use it this way.
Perfectly possible with XML too
> An SQLite based store is one of the most tested and optimal formats for document and application storage
It's optimized for things that largely don't matter for password storage. The testing is admirable, but there's no issue of keepass clients crashing or corrupting data so again, not very relevant (probably because of low concurrency, simple writes etc).
> A switch this big is a major chance to fix the governance structure and align it more with a democratic consortium than a benevolent-dictator-for-life style of project management
You don't need a technical change to solve this. In fact, a fork that would fracture clients is the last thing you need when making governance changes.
> So many quality of life features can be added where the old schema disallowed it
All of the features they list can be achieved with an XML format. The format isn't what's holding them back.
All of which is to say I would bet on something in the webdav-nextcloud line being at fault instead of keepassxc.
Not writing the entire database on every save?
EDIT: also you don't need to have just one password vault and I'd say you probably shouldn't, separate entries also assist with separation of concerns. This last adds a little overhead but is a reasonable workaround.
However on the whole I like sqlite for app persistence. It can, however, leak data (forensically) if not managed properly.
If anything, maybe give people the option to export to SQLite and then use that going forward but keep it entirely optional.
None at all. That is why I am saying it should be entirely optional so most people can just ignore it and those wanting the alternate schema can get their feature request fulfilled.
I do not see how it would benefits end users in any significant way. Sure, you can look inside the database, how many people need that?
Dumping the database to CSV is not a good backup, schema changes over time, what was dumped from one version of the app would not work for importing into another version if schema changes. Backup it needs a versioned schema format, which would actually look like KDBX format if implemented in XML :-)
As for scale this app is for one person doing one query at a time or saving one password at a time. One person will not be saving millions of passwords or if for some reason they are then they can export to a format that an enterprise solution could take it's place. People have written team / company password managers that can use Oracle, MySQL and Postgres that can track and audit user and team password changes. This is not something that should ever be expected of an individual personal password manager like KeepassXC.
This is just my take but I would rather have the maintainers of KP focus on bugs, usability issues, quality of life enhancements which they have been doing great thus far in my opinion. Forcing a schema migration is just asking for trouble and potentially causing bugs that turn some people away from using it or cause people to lose data if they do not have snapshots of their database. Or if forcing a schema change make for damn sure there are many backups in each format and encourage the users to back up all the files to many external encrypted drives and store some of the encrypted devices off-site. Something everyone should be doing regardless.
That problem that the current schema doesn't have enough ways to declare custom memory-protected fields outside of user-facing attributes seems just as likely to be replicated and maybe worsened in an SQL schema.
Changing the database engine doesn't fix the architecture problems nor schema migration problems. It's certainly a good time to reevaluate the architecture problems and the schema migration problems. But the huge caution I'd suggest here would be look at the ossification complaints about the current XML schema and expect SQL migrations to be worse and plan for worse multi-schema operations and intenser ossification. (Especially because these files are expected to live in a large multi-vendor ecosystem, SQLite schema migration management is going to be much worse than any XML schema management.)
Why is your KDBX file 20 MiB? It seems you are storing something that is not actually a good fit for a password manager, and expecting the entire world to change around you instead of storing those files in a more appropriate place.
The safe combination is a secret, and obviously belongs in secret storage.
In this specific example, if I had trouble remembering whether the first number of the combination was reached via cw or ccw rotations, I'd include that in the secret, e.g. "cw34-12-22-45".
(Some safe combinations require multiple rotations. I unintentionally became the owner of one that is something like "cw3x34-ccw2x12-cw5x22-ccw2x45". I still can't open it actually, but that pattern is what the Internet tells me. :)
My kdbx is only 173 KB, but I don't hate this idea.
For 10 years I've managed a family of 4 in a single KeePass db. Unique passwords across all accounts, random passwords instead of PII for "security questions", fake DOBs, and all other random security related stuff. Never had a single issue. Everything happens instantly as far as I can tell. And the file is 67kb.
My work dbs, separate only because they should be, are much smaller and simpler.
I love KeePass. Changing everything under the hood probably only has potential to make pain for the user. Best case is that nothing is made noticeably worse. Doesn't seem worth the risk.
Users have wanted proper custom entry templates for years in order to input their card details into the manager. BitWarden, CodeBook, and others all have this built in without needing to fiddle with attributes and autofill to make it work.
I think a major change like this is worth it when we consider the KeePass of the next decade or two. This is genuinely my most important file by a longshot. Basing it on SQLite is just the right move for future proofing.
It uses SQLCypher as local data store and keeps KeePass-compatible data model - supports import of .kdbx database.
SQLCypher and different schema used in Lockstep alone did not solve any problems that I have with Keepass. Those problems are sync and sharing.
Solving sync and sharing cannot be done on whole database file level, as it implemented now in KeePass. Changes need to be tracked at the password record level, all changes need to persisted as operations log and that log needs to be distributed across devices.
The above means writing a whole protocol, and that's a lot more work than changing local storage.
The project is pre-alfa https://github.com/lockstepvault-hq/lockstep