Ask HN: Is anyone successfully self-hosting Firefox Sync?

I've self-hosted the original Firefox syncserver[0] for a couple of years now. It's a now unmaintained Python 2 + gunicorn web app.

It still uses the Firefox identities service, since I haven't bothered to try setting that one up myself. That means I need a Firefox/Mozilla account even with a self-hosted sync server, which isn't ideal.

Note that Mozilla do provide a script to delete all your data from their servers[1], in case you were using Firefox sync before going self-hosted.

I originally went through the setup instructions manually. Now I've created a Nix package[2] and NixOS module for it[3], which was an adventure by itself. I can now easily reproduce the setup, which I did while migrating my webserver to a different provider.

As for why I didn't try syncstorage-rs, it's rather petty, but the old syncserver was working well for me, and it doesn't yet support SQLite.

[0] https://github.com/mozilla-services/syncserver

[1] https://github.com/mozilla-services/syncserver#removing-mozi...

[2] https://git.sr.ht/~williamvds/config/tree/master/item/nix/pk...

[3] https://git.sr.ht/~williamvds/config/tree/master/item/nix/mo...

wkat4242 · 3 years ago

The first few lines in the repo you mention illustrate the problem:

> Note that this repository is no longer being maintained. Use this at your own risk, and with the understanding that it is not being maintained, work is being done on its replacement, and that no support or assistance will be offered.

And its replacement syncserver-rs is in this forever-unfinished state so they've effectively killed self-hosting. At least from a support standpoint.

toyg · 3 years ago

I don't understand. The original was barely 500 lines of code in two files, everything else being packaging machinery. Yes it's python 2, but how hard could have it been to port 500 LOCs to 3.x...? Even assuming some crypto libraries broke, surely there are working equivalents?

Instead, they went for a Big Rewrite in Rust, resulting in a much bigger codebase that will be much harder to maintain... and broke it anyway.

williamvds · 3 years ago

Good point, I'll update the comment to indicate syncserver is now unmaintained.

Of course, everything self-hosted is at my own risk etc. And official support is a luxury in this kind of FOSS, I generally don't count on it.

But since it works for me, it may be worth sharing the details.

MawKKe · 3 years ago

Sounds like classic case of ”Second System Effect”

kevincox · 3 years ago

Have you considered upstreaming the package and module to nixpkgs? It would be awesome to have those one line of config away for all nixpkgs users.

williamvds · 3 years ago

I briefly considered it, but there are some big problems:

- The new replacement package, syncserver-rs, has already been added to Nixpkgs, along with a NixOS module [0]

- Python 2 and the package itself are no longer supported, meaning...

- support in Nixpkgs is pretty rough - dependencies have been updated such that they no longer support Python 2, or have been removed outright, meaning...

- syncserver has already been removed from Nixpkgs

- in order to avoid the problem of unsupported dependencies, I delegated dependency resolution to a community project, poetry2nix[1]. Even so, it was a nightmare to get working, lots of hacks and workarounds were needed

Of course, my repo is public, so anyone is welcome to use the Flake within. If they do, I'd suggest dropping me a line so I don't do anything drastic to the package or module.

[0] https://github.com/NixOS/nixpkgs/pull/176835

[1] https://github.com/nix-community/poetry2nix

pieter_mj · 3 years ago

>Note that Mozilla do provide a script to delete all your data from their servers[1], in case you were using Firefox sync before going self-hosted.

Note : You'll need to disable 2fa first on your account before you can delete the data.

pmarreck · 3 years ago

I think Nix, as struggle-full as it was, was the right choice, here, although I don't see the flake.nix file

williamvds · 3 years ago

The flake starts at this level: https://git.sr.ht/~williamvds/config/tree/master/item/nix

Struggle is a good way to put it! I feel I hit one of Nix's pain points here: packaging software whose dependencies are largely missing from Nixpkgs. On top of that, it was a Python app, and the state of Python packaging is quite a mess.

I also feel it Nix was the right choice. My journey was rather harsh, but I do believe in its approach.

wasmitnetzen · 3 years ago

Yes, I'm running it with the mozilla/syncserver docker image[1], but still using the hosted identity service. The setup procedure is a bit fickle, the error handling on Firefox' side is quite bad, sometimes the first sync times out when you have a larger data set, and you need to manually enable each sync type to reduce the size. But once it's up and running, I don't really have any issues.

I do hope that syncstorage-rs will become usable eventually and hopefully improve performance.

Note: contrary to your post, cookies are not synced.

[1]: https://hub.docker.com/r/mozilla/syncserver

heywoodlh · 3 years ago

What is gained from self hosting the container? Are you able to sync more data than Firefox's official Sync service? Less reliance on a third party service? Would love to understand the value-add.

Their docs[0] make it seem like little would be gained from a privacy/security perspective as it seems pretty secure/private (at least to me).

[0] https://support.mozilla.org/en-US/kb/how-firefox-sync-keeps-...

eqvinox · 3 years ago

You could put your self-hosted container in a self-hosted VPN to add a layer of protection.

But honestly, while encryption theoretically can make any data "safe" to hand to a 3rd party, it's just good "data hygiene" to keep it under your own control. If someone steals the Firefox sync database now, and some part of the encryption scheme is later broken by e.g. quantum computers, the thieves could retroactively decrypt your sync account data.

And there's also the question of metadata - when does your Firefox send data to sync, and how much? Are you sure there's nothing to glean from that?

boring_twenties · 3 years ago

Am I missing something, or does using the hosted identity service defeat the whole purpose?

I guess that depends on your threat model. I want to defend against Mozilla being breached and all data getting accessed, and (legal) data mining by Mozilla. My setup doesn't defend against Mozilla going rogue, or someone getting malicious code into Firefox.

Deleted Comment

Mikescher · 3 years ago

Shameless plug: A while ago I wrote a cli for firefox sync [1] (in my case mostly to automate some stuff when I create bookmarks).

It does not really solve your problem for browser-history/credentials etc. But as long as you use a client that you verified you can be kinda sure the data is E2E encrypted and can't be read by Mozilla.

Also on a side note: To this day the weave protocol is one of the more (needlessly) complex protocols I ever implemented and I'm still not sure what problems some of the design decisions solve...

[1] github.com/mikescher/firefox-sync-client

Wicher · 3 years ago

This looks very useful, thanks for creating!

darylfritz · 3 years ago

Does this have the ability to read the open tabs that are synced from all devices?

arkadiyt · 3 years ago

> However, I do not want to risk storing absolutely sensitive data (browsing history, auth cookies, credentials, etc) at some third party.

FWIW Firefox says they do not have the capability to read your data: https://hacks.mozilla.org/2018/11/firefox-sync-privacy/

gorgoiler · 3 years ago

Unfortunately, this isn’t really true. The party which holds your data is also the party which ships evergreen versions of the software used to decrypt that data. There’s nothing to stop the government in Firefox’s jurisdiction from bullying them into adding a backdoor. We can assume the bullying is ongoing.

I like to imagine it as having a safe deposit box provided by a local bank. I have the only key and have proven that to be the case. I also store the key at home in another safe which is regularly serviced by bank employees. While they probably don’t abuse their position in the latter to access the former, it would be completely wrong to say they do not have the capability to get at my stuff.

All that aside, I love FF sync and rely on it everyday for work stuff. It is like magic and I’m very grateful for the feature. I am realistic about the NSA’s level of interest in my work calendar and the threat they pose to my day to day life.

> The party which holds your data is also the party which ships evergreen versions of the software used to decrypt that data. There’s nothing to stop the government in Firefox’s jurisdiction from bullying them into adding a backdoor.

Hosting your own sync server doesn't prevent this.

iso1631 · 3 years ago

> I have the only key and have proven that to be the case

Locksmiths will get through that lock in a couple of minutes, probably without you ever knowing

The only thing stopping the bank and/or government from accessing your box is the law

dodgerdan · 3 years ago

Then you can’t trust the FF code not to send a copy to their own servers also.

bluGill · 3 years ago

Banks always drill out the lock on a safe deposit box if they need to get in, even though it wouldn't be hard for someone to pick the lock (or so lockpicks who claim to have the ability to check this say, I've never tried to pick the lock on a safe deposit box, and my lockpick skills are not very good so I doubt I could)

hosteur · 3 years ago

> FWIW Firefox says they do not have the capability to read your data: https://hacks.mozilla.org/2018/11/firefox-sync-privacy/

The article is from 2018. Isn't this only relating to the old version of Firefox Sync - the one in which self-hosting is actually supported?

The design looks reasonable. However, my main problem with it is that putting all this extremely valuable data in one place makes it a huge target. A supply-chain attack or an implementation error in the crypto would be devastating.

HPsquared · 3 years ago

That's also an option in Chrome, "encrypt sync data with your own sync passphrase"

the_third_wave · 3 years ago

No, I used to host the previous version but when that stopped working (including the workarounds which kept it going for a bit longer) I ceased using it. While the sync server itself is not that much of an effort to get going, the auth server is. I do not use external auth services when I can avoid it so I chose to forego on using sync instead of creating a "Firefox account". If I ever find an easier way to get the auth server running and if I still use Firefox by that time I might start using it again but until that time it is a no-go.

To Mozilla: fire your overpaid activist CEO Mitchell Baker and use the resulting funds to hire (back) the developers she got rid of, clean up the mess, get rid of the social justice nonsense and take back your place as a browser development organisation instead of a 'colours of change' peddler.

captn3m0 · 3 years ago

I faced enough issues that I switched to the Mozilla endpoints. Problems ranged from no clear documentation for configuration, unclear resource usage, no documented method to host identity as well. My one hope was to get my data exported via SQLite for fun stuff, as an automated export. But the data format was also not well documented, so that never happened.

charles_f · 3 years ago

I self host most of my stuff for the reasons you mentioned, and that included Sync for a few years. I never got the auth part to work so I stopped trying and just used the sync server itself.

I stopped doing that after some time, it seemed too futile. Errors are hard to debug and you're on your own, plus I was not using it for passwords or CC, so it just wasn't worth it.

rayusher · 3 years ago

What other stuff do you host?

mbwgh · 3 years ago

Why does every "Ask HN" I see come with a grey text color? Is this a stylistic thing or do people take issue with people asking questions and downvote?

aamargulies · 3 years ago

HN does this purposefully to show its preference for external content rather than self posts.

KronisLV · 3 years ago

I wonder if there could be an alternative way of getting the point across, as opposed to having low contrast text, which some tools report as an accessibility problem: https://wave.webaim.org/report#/https://news.ycombinator.com...

But other than that, it's an interesting design choice.

pancrufty · 3 years ago

HN does this because they don’t care. It wouldn’t take much to make this site usable, but they refuse to add 5 lines of CSS to make the site work in dark mode.

groestl · 3 years ago

Aaaaah facepalm. Using this for a decade, never had a clue.

MintPaw · 3 years ago

Wow, TIL after many years. Kinda scummy to show some content as artificially downvoted to drive engagement. I thought everyone hated questions...

fIREpOK · 3 years ago

HN does the same when you go through your settings and select "show hidden"... You "see" them but you dont.. The light gray goes even closer to the background color in this case.

thrdbndndn · 3 years ago

div.toptext { color: black; }

@plibither8: Maybe this is something Refined Hacker News extension could do? https://github.com/plibither8/refined-hacker-news

japanman425 · 3 years ago

Okay but why should I do that