Firefox Sync Storage / syncstorage-rs

[wbk]

Hi all,

(Related apps: syncstorage-rs and (depricated) syncstorage)

After keeping off upgrading to YNH 11 for quite a while because it would mean no more Firefox Sync (Python 2.x), I bit the bullet and instead hosted the syncserver on a VPS with Debian 10 / buster, and upgraded my home-YNH to 11 / bullseye.

A few days back I noticed Firefox SyncStorage in the catalog! I knew there was work at Mozilla to have a reimplementation of the SyncServer, but did not know it had completed let alone been packaged for Yunohost. Thanks all involved!

I started this post to have something on the forum that reacts to the tag ‘syncserver-rs’, and to kill the time while the app installs. The step Building the sources (it will take some time)... indeed does take some time; my Yunohost has been allocated 3 out of 4 cores of a Pentium J5005 (low power, but still almost four times as powerfull as a Raspberry 4b, according to some benchmarksite)

The compilation step took about 30 minutes; after that it took less than 10 minutes to display the ‘post install’ screen.

I’m quite happy so far:

• the domain is the same, I just recreated the one that was moved to the VPS
• so, after updating DNS, the clients will connect with the new sync storage instead of the old sync server
• one caveat:
  • the old syncserver allows for paths in a domain (which I have)
  • the new syncserver-rs does not allow a path in the domain
• as a result, the custom identity.sync.tokenserver.uri’s have to be changed on the clients.

The last point turns out a blessing in disguise: for my own Firefox I created backups of at least logins and bookmarks (because sync is not backup, on the old platform), so that in case of major reset when starting the sync, I at least have something to fall back on.

For the rest of the family I did not do that and hoped for the best.

As it is now, their Firefoxes will not be able to sync for a while, until I tested the new platform and update their URI’s. Before that I could switch back the DNS, to allow their clients to sync before being switched to the new platform.

So far for tonight, possibly for this year. To be continued!

[hook]

Thanks for kicking this discussion off!

I’m using Firefox to sync tabs (etc.) between many devices – in fact, I’m (ab)using the devices to use several Firefox Profiles (one for each KDE Plasma Activity) to have different. Which means I typically have a dozen or two of “devices”/Profiles bound to the same Firefox Sync account.

What I noticed with Mozilla’s official Firefox Sync is that after some time devices you haven’t used in a long time get forgotten and you can’t restore the tabs from them.

Is this limitation lifted if you self-host?

Any drawbacks for self-hosting?

[wbk]

@hook

several Firefox Profiles (one for each KDE Plasma Activity)

That is an interesting way of seperating identities, for lack of a better word.

Risking thread derailment: how do other applications hold up with the different identities per workspace?

@hook

What I noticed with Mozilla’s official Firefox Sync

I never used the ‘hosted’ sync. What I do know, is that the old sync server (either hosted by Mozilla or self-hosted) only did that: sync between active devices. One device offline while the other is online? No sync. Phone and PC synced, but lost your telephone while your PC crashed? No backup at the sync server.

The new sync server is supposed to overcome that perceived / unexpected shortcoming. I guess holds a copy of the profile as if it were another client.

I’d say that when you use the Mozilla-provided service, your devices sync via the new server and there should be a copy of your synced things a their servers. Perhaps they clear stale profiles after some amount of time?

@hook

Is this limitation lifted if you self-host?

I have not perceived this limit in a reproducible way. You might know that the sync service consists of two separate legs: the Mozilla account for authorization, and the Sync account for synchronization.

The authorization server can also be self hosted, but is (even) more complex to set up than the sync server (which ‘just works’ via Yunohost). That aside: when I log in to my Mozilla account, I see connected devices that have been last used up to four years ago. I’d have to delve up such a device to see whether sync catches up.

For these old sessions the newer sync features, such as ‘send tab to device’ are not available; I guess the session is there, but since the client has not connected for four years, Sync has no way to know which version of Firefox is running other than ‘probably four years old’. The four years old version does not have the ‘send tab to device’ feature, so I can’t chose that device on my current Firefox to send a tab to.

@hook

Any drawbacks for self-hosting?

Only one: the URI string for the sync server has to be changed on each client. No trouble on desktop clients, but more of a hassle on mobile devices (but quite doable recently).

It had been nice if Firefox asked you which sync server you want to use once you turn synchronization on, but while it would promote their ‘freedom of choice’-iness, it would complicate matters for >80% >99.9% of their users

[wbk]

TL:WR: so far I have mixed results:

• Profiles that existed before connecting to the new syncservice-rs, still sync with each other after switching to the new service.
• Profiles that are created after switching and only have experience with the new syncservice-rs, only sync with each other
• It seems these two groups don’t mix : no synchronization from one group to the other
• I can send tabs from one group to the other though

Longer version:

I set out to test the new syncstorage-rs before decomissioning the old (still very much functional, albeit also very much Python-end-of-life) syncservice.

Test outline:

• I started with my regular profiles on a laptop, desktop and a phone being in-sync. I added a new profile to the desktop version, and let that sync as well. (Sync was maybe not instantaneous, but done within a minute, including copying and installing of existing plugins)
• Switched DNS from syncservice to syncservice-rs (with a path from https://sync.domain.tld/ffsync/token/1.0/sync/1.5, to https://sync.domain.tld/token/1.0/sync/1.5 )
• Installed a new Firefox-based browser (nightly vs regular beta; “FFUpdater” makes it easy to find various Firefox-variants) on the mobile (as Firefox on mobile lacks profiles), set up the syncserver URI to point to https://sync.domain.tld/token/1.0/sync/1.5 and logged in to my Firefox-account
• open logs on my Yunohost to see if anything is happening
  tail -f /var/log/syncserver-rs/syncserver-rs.log and
  tail -f /var/log/nginx/sync.domain.tld-access.log
• Confirm that, yes, when I press ‘sync’ on either desktop or mobile, a call is done to the sync service
• Wait and find that, no, the phone does not receive any profile data

I let the mobile browser run for a while (an hour? Could be more, could be a bit less). No results.
Next:

• I uninstalled Firefox Nightly, redid the installation and configuration. No improvement for the most part. It does sync with the instance of Nightly that I had just uninstalled, so sync at some level does work (and storage does as well)
• I installed Firefox Klar. It does not seem to have Firefox/Mozilla account integration or sync capability out of the box.
• I installed Fennec F-Droid. Nice: this browser has quite a few settings by default that I have to change on regular Firefox. Synchronization though, no go. It does sync with the two Nightly ‘profiles’, but not with the others.
• As a test, on Fennec, I disabled all syncing, except for logins. After that I tried on Nightly to open a tab from Fennec: not possible: because the ‘send tabs’ sync feature was disabled also in Nightly: the settings had been synced (this was via the third tab on ‘open tabs’ popup, left of ‘incognito’ tabs; long-click on an open tab and share → send to device still works, and can also send to the clients that came from the ‘legacy’ sysnservice.)
• Finally, I thought of creating another profile on the desktop, that would sync with the new syncserver-rs, so would never have seen the old syncserver. That profile is somewhere in the middle, worst-of-both-worlds:
  • It has a single extension from my synced profiles (desktop-side) but lacks all the rest
  • It is not visible (and does not sync) with my mobile ‘new only’ synchronizations
  • “Send tab to device” does work, for sending and for receiving
  • As opposed to the other desktop-profiles, this profile does not have an “about:sync” page, only the “about:sync-log” page
  • What? While checking the error logs (about:sync-log, or ~/.mozilla/firefox/profileano.t.ao.3/weave/logs ) I just found I made a copy/paste mistake, and tried syncing to the non-existent (at the moment) old syncservice! No wonder nothing got synchronized; very peculiar the tabs could be sent to another device none the less!
  • upon correcting the URI, I still get a 404 on the correct link for the service

It seems that ‘new’ synchronization targets can sync with none of the accounts that have ever touched an old sync service.

To be continued, for sure!

[wbk]

@wbk

very peculiar the tabs could be sent to another device none the less!

On further reading, maybe that is a feature of the accounts-server, as described in Device Registration (Sync) | Firefox Ecosystem Platform

[hook]

@wbk

That is an interesting way of seperating identities, for lack of a better word.

Risking thread derailment: how do other applications hold up with the different identities per workspace?

This is how I get Firefox to do it:

Introducing Activity-aware Firefox - A script to make KDE Plasma and Firefox...

On KDE Plasma I use Activities quite a bit. Most KDE Gears/Apps handle it fine (there’s some Kirigami ones that don’t), and with non-KDE apps, it’s a hit-or-miss, but I rarely run into trouble TBH.

…back on topic then

[hook]

@wbk

I’d say that when you use the Mozilla-provided service, your devices sync via the new server and there should be a copy of your synced things a their servers. Perhaps they clear stale profiles after some amount of time?

@wbk

That aside: when I log in to my Mozilla account, I see connected devices that have been last used up to four years ago. I’d have to delve up such a device to see whether sync catches up.

IIRC, someone on the official Mozilla Matrix channel told me that they indeed clear stale profiles (IIRC, you can still see the profile, but it’s empty), so they don’t run out of space. I wonder if that’s the same with self-hosted.

[wbk]

@hook

I wonder if [there’s a space limit] with self-hosted.

Troubleshooting my non-syncing syncserver, I came across the configuration (/var/www/syncserver-rs/config.toml). Quota are disabled by default:

# enable quota limits
syncstorage.enable_quota = 0
# set the quota limit to 2GB.
# max_quota_limit = 200000000
syncstorage.enabled = true
syncstorage.limits.max_total_records = 1666 # See issues #298/#333

[hook]

Awesome, thanks!

[Danceswithcats]

Hello everyone,

I’d like to have my Firefox account self-hosted. However, the app on Yunohost catalogue says it is not maintained anymore. There is a second one, which says it is ‘implemented in Rust’, but there is no information about it. Is it the one you are all using?

[wbk]

Hi Danceswithcats,

The complete ‘solution’ consists of multiple services, each of which can be self-hosted.

Authentication is handled by the account server (it used to be called ‘Firefox account’, but has been rebranded to ‘Mozilla account’, I believe; perhaps to make it a logical place to authenticate for Thunderbird sync, once that becomes available). The authentication part is not available for Yunohost, and used to be somewhat more complex to set up than the sync service.

The part that is available for Yunohost is the synchronization part. The original syncserver ran on Python 2.x, which is not available on Debian 11 with as a result that this package does not run on any halfway modern distribution.

The replacement for the syncserver is syncserver-rs. Besides being written in Rust instead of Python, it also stores instead of only enabling synchronization. The benefit of the last bit is that it also works as a backup for your settings even if you don’t have another machine or profile to sync with.

To use your self-hosted syncserver-rs, you still need an authentication server. I use the Mozilla-provided authorization server (and thus, a regular Mozilla/Firefox account), with my self-hosted syncservice-rs.

Make sure to point your Firefox to the right tokenserver URI (as described by the app after installing) before logging in, or if you already have your Firefox connected, log out before setting the correct URI.

In short:

• Install syncserver-rs in its own subdomain
• (log your Firefox out of its connection with Mozilla, if it is already signed in)
• Correct the token URI as described, something like https://sync.domain.tld/1.0/sync/1.5
• Log your Firefox into your Firefox/Mozilla account
• You will be taken to the Mozilla/Firefox accountserver to log in, but synchronization will take place in the background via your own server

Good luck!

[Danceswithcats]

wbk,

This is an incredibly thorough and generous answer. Thank you so much for this. You’ve boosted my confidence. I will get on to this this weekend.

[Danceswithcats]

It appears to have worked. Syncing now. I’ve got half a dozen laptops that need to be signed into a new account, but it’s looking promising.

You should think about getting a link to your post put in the app documentation page in the Yunohost applications catalogue.

Thank you again.

[wbk]

@Danceswithcats

You should think about getting a link

The app’s admin page has a link to all posts with the corresponding tags (was a reason to start the post in the first place: there was not any result when I clicked that link :-P)

I didn’t think of posting a backlink to the catalog, that I have added now.

PS: Nice you got it running!