[Solved] Firefox Sync Storage / Mozilla Syncserver-rs not syncing

[wbk]

Hi all,

My YunoHost server

Hardware: computer at home
YunoHost version:
- yunohost version: 11.2.8.2 (stable) , upgraded to 11.2.9.1 (stable)
- yunohost-admin version: 11.2.3 (stable) , upgraded to 11.2.4 (stable)
- moulinette version: 11.2 (stable) (no upgrade)
- ssowat version: 11.2 (stable) (no upgrade)

I have access to my server : Through SSH | through the webadmin | direct access via keyboard / screen
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no /
If your request is related to an app, specify its name and version:
id: syncserver-rs
name: Firefox SyncStorage
version: 0.14.4~ynh1 (no upgrade)

Description of my issue

I have installed syncserver-rs, expecting it to be a drop-in replacement for the old syncserver (1.9.1~ynh3) that ran on Python 2. After installing and connecting Firefox, sync does not happen; about:sync-log in Firefox shows errors.

To keep sync functionality and also upgrade to Bullseye, I had a small VPS running Debian 10 / Buster with Yunohost 4.4.3. It ran at https://ffs.domain.tld/ffsync/.

The VPS is still running, only DNS points to Yunohost at home now, where I also added the ffs-subdomain.

Because syncserver-rs does not like to be installed in a subdirectory, the identity.sync.tokenserver.uri changed a bit:

• https://ffs.domain.tld/token/1.0/sync/1.5 vs
• https://ffs.domain.tld/ffsync/token/1.0/sync/1.5 previously

I started out with a new profile in Firefox and an alternative Firefox-based browser on my phone for the new syncserver-rs. No synchronization was happening. I switched my main profile as well, to prevent the syncserver-rs or nginx log to show mentions of the old URL.

Firefox logs the synchronization client side. Logs are written to the profile directory, and can conveniently be accessed via the special about:sync-logs URL. I uploaded a sample of this mornings’ error log

Searching for ‘ffs’, this section pops out:

1704950836337	FirefoxAccounts	INFO	fetching updated device list
1704950836393	Services.Common.RESTRequest	DEBUG	GET request to https://api.accounts.firefox.com/v1/account/devices?filterIdleDevicesTimestamp=1703136436340
1704950836464	Services.Common.RESTRequest	DEBUG	GET https://ffs.maindomain.tld/token/1.0/sync/1.5 404
1704950836464	Services.Common.TokenServerClient	DEBUG	Got token response: 404
1704950836465	Services.Common.TokenServerClient	WARN	Error processing token server response: TypeError: right-hand side of 'in' should be an object, got number(resource://services-common/tokenserverclient.sys.mjs:297:11) JS Stack trace: _processTokenResponse@tokenserverclient.sys.mjs:297:11
_tokenServerExchangeRequest@tokenserverclient.sys.mjs:239:19
1704950836465	Sync.SyncAuthManager	ERROR	Non-authentication error in _fetchTokenForUser: TokenServerClientError({"message":{}})(resource://services-common/tokenserverclient.sys.mjs:28:36) JS Stack trace: TokenServerClientError@tokenserverclient.sys.mjs:25:16
_tokenServerExchangeRequest@tokenserverclient.sys.mjs:245:19
1704950836465	Sync.Status	DEBUG	Status.login: success.status_ok => error.login.reason.network
1704950836465	Sync.Status	DEBUG	Status.service: error.login.failed => error.login.failed

When visiting the domain ‘manually’, a note is displayed telling “Syncstorage is running”, and some JSON-info when visiting the token-URL. There the response header mentions SSO:

Response headers
X-Firefox-Spdy	h2
content-length	1
content-security-policy	upgrade-insecure-requests
content-type	application/json
date	Thu, 11 Jan 2024 05:54:00 GMT
permissions-policy	interest-cohort=()
server	nginx
strict-transport-security	max-age=63072000; includeSubDomains; preload
x-content-type-options	nosniff
x-download-options	noopen
x-frame-options	SAMEORIGIN
x-permitted-cross-domain-policies	none
x-sso-wat	You’ve just been SSOed
x-weave-timestamp	1704952440.28
x-xss-protection	1; mode=block

I’m not familiar with the workings of SSO-wat. I found the config in /etc/ssowat/conf.json, can/should I add the token URI there? Or add a custom web app at the location (not sure whether syncserver-rs’s greedy ‘this is my domain’-needs would allow).

How can I prevent SSO-wat from blocking access to https://ffs.domain.tld/token/1.0/sync/1.5 ?

[orhtej2]

The app needs to be installed with visitors access. Are you able to reach https://ffs.domain.tld/__heartbeat__ ?

[orhtej2]

Also, there’s broader discussion on setting it up here: Firefox Sync Storage / syncstorage-rs - #4 by wbk

[wbk]

Hi orthej2,

Thank you for the suggestions!

@orhtej2

The app needs to be installed with visitors access.

The app does not allow changing the property from the ‘apps’ page, not on install and not afterwards.

The property is ‘hardcoded’ in the groups, here is a detail of my ‘Visitors’ section in ‘Groups and permissions’ :

Notice it does not have the x behind it, as Cypht has: it is not possible to (accidentally) disable guest/visitor/anonymous access.

Even so,

@orhtej2

Are you able to reach https://ffs.domain.tld/__heartbeat__ ?

shows:

Response Headers
X-Firefox-Spdy	h2
content-length	85
content-security-policy	upgrade-insecure-requests
content-type	application/json
date	Thu, 11 Jan 2024 19:18:51 GMT
permissions-policy	interest-cohort=()
server	nginx
strict-transport-security	max-age=63072000; includeSubDomains; preload
x-content-type-options	nosniff
x-download-options	noopen
x-frame-options	SAMEORIGIN
x-permitted-cross-domain-policies	none
x-sso-wat	You’ve just been SSOed
x-xss-protection	1; mode=block

Notice the last two lines, also SSOed and mode=block

I changed the group permissions to allow “All users” access to Firefox Syncstorage, thinking that as I am logged in anyway, ssowat might just allow me access to the page. Unfortunately, it does not make a change though.

[wbk]

I had a look at the SSO/LDAP-documentation and skimmed the SSOwat-repository, to no avail.

The documentation page mentions …

TODO/FIXME: moar explanations of how this is done for non-PHP apps?

… with syncstorage-rs being a Rust-app, is that todo applicable?

Some other SSO-related problems on the forums were solved after upgrading Yunohost. It was upgraded to version 11.2.9.1 ; that made no difference.

I guessed the client would already have gotten a token from the Mozilla account, giving access to the syncstorage without needing basic auth.

My next try was to install “My Webapp” on the same domain, with the /token/1.0/sync/1.5 path as the URL where it was to be installed.

No-go either:

This URL is either unavailable, or conflicts with the already installed app(s):

• ffs.domain.tld/ → Firefox SyncStorage (syncserver-rs)

Any suggestions?

[Aleks]

@wbk

How can I prevent SSO-wat from blocking access to https://ffs.domain.tld/token/1.0/sync/1.5 ?

It’s not very clear to me what doesn’t work exactly, but if the app is indeed allowed for visitors, and if you are indeed able to see the note “Syncstorage is running” and see JSON info when visiting the token URL, then SSOwat is not blocking it (assuming you’re doing this not-logged-in)

The fact that you see X-SSO-Wat You’ve just been SSOed in the header means nothing, it’s a generic header added on all request, wether blocked or not (clearly kind of misleading, i’m not sure what’s the initial motivation behind it)

[wbk]

Hi Aleks,

Thanks,

@Aleks

(assuming you’re doing this not-logged-in)

that is an important distinction that I overlooked!

Testing again in a private/incognito browser window, the behaviour when visiting the /token/1.0/sync/1.5/ or __heartbeat__ URLs gives the same result, this for the …token… URL:

and this for the heartbeat URL:

(in both cases showing the most interesting bit; the headers for heartbeat are about the same, but with content-length 1 for the token URL, there’s nothing interesting in JSON/raw data).

@Aleks

what doesn’t work exactly (?)

I also don’t know what the problem exactly is; the result is that synchronizing does not work. All synchronization attempts fail with the 404 in the log in the first post.

There are ‘error.login failed’ as well (see this log, pasted without going through yunohost-log-upload). I blamed Firefox not being able to find (404) the syncstorage for that , because I’d expected 403 if it had found it but was unable to log in.

When starting the thread I was quite sure SSO was the culprit, and did not post any serverside logs. If these signs are not of being blocked by SSO, serverside logs are relevant.

Actually, something does happen. tail -f /var/log/syncserver-rs/syncserver-rs.log gives a new line every time I open the Firefox-account submenu by clicking on my accountname in Firefox’ hamburgermenu (top-right, in my case). This line is written:

Jan 13 09:53:15.845 INFO {“ua.os.family”:“Linux”,“ua.name”:“Firefox”,“ua.browser.ver”:“115.0”,“ua”:“115.0”,“uri.method”:“GET”,“ua.browser.family”:“Firefox”,“uri.path”:“/token/1.0/sync/1.5”,“ua.os.ver”:“UNKNOWN”}

Upon subsequent ‘sync now’, the line changes to ‘syncing’ for a couple of seconds, and returns to ‘sync now’. There is a new line in the log serverside, and a new log client side (see timestamp 2024-01-13 10:55:17 in the clientlog); excerpt from the serverlog:

Jan 13 09:53:15.845 INFO {"ua.os.family":"Linux","ua.name":"Firefox","ua.browser.ver":"115.0","ua":"115.0","uri.method":"GET","ua.browser.family":"Firefox","uri.path":"/token/1.0/sync/1.5","ua.os.ver":"UNKNOWN"}
Jan 13 09:55:10.275 INFO {"ua.browser.ver":"115.0","ua.os.ver":"UNKNOWN","uri.method":"GET","ua.name":"Firefox","ua":"115.0","ua.browser.family":"Firefox","uri.path":"/token/1.0/sync/1.5","ua.os.family":"Linux"}
Jan 13 09:55:17.449 INFO {"ua.os.family":"Linux","ua.os.ver":"UNKNOWN","ua.name":"Firefox","ua.browser.family":"Firefox","uri.path":"/token/1.0/sync/1.5","uri.method":"GET","ua.browser.ver":"115.0","ua":"115.0"}
Jan 13 09:56:58.223 INFO {"ua.os.family":"Linux","ua.name":"Firefox","ua.browser.ver":"115.0","uri.method":"GET","ua":"115.0","ua.browser.family":"Firefox","uri.path":"/token/1.0/sync/1.5","ua.os.ver":"UNKNOWN"}

The records seem different, but as far as I notice, the information is the same only in a different order.

In the nginx logs there is nothing interesting either, it seems. The last err.log is empty, err.log.1 is from days ago. The access.log shows the same 404 that is mentioned clientside by Firefox.

It makes me wonder whether I copied the URI correctly. The old syncserver would print the manual when visiting the public URL, but while the new syncserver does have a troubleshooting section in the app section of the admin pages, the actual sync URI was only showed after installation. Even so, while visiting the domain of syncstorage-rs logs 200 followed by 499 in acces.log, and heartbeat gives 200, copying the token URI from the firefox log and visiting it, does give me 404 in nginx’s access.log.

Some other facts and assumptions that may or may not be relevant:

• The passwords (and users) for the Mozilla account (previously Firefox account) for authenticating are not the same as for Yunohost
• The old syncserver only synchronized stuff, and allowed to do so after authenticating with an authentication server (I used the default Mozilla one for auth); the new syncstorage stores stuff as well; I assume the accounts are still defined at the Mozilla auth server, not in Yunohost
• I tested ‘migrated’ profiles as well as new profiles that never saw the old syncservice (or the old URL, for that matter), as well as desktop Firefox as some mobile variants (lacking profiles).
• I am not aware of any configuration to be done serverside (at my Yunohost for syncserver-rs), I only pointed the token URI in about:config in the right direction. That was enough for the old syncservice.
• Installation of the syncserver went super easy (as usually with any app in the catalog), without any reason to retry installation. That is something I could still try.

Sorry for the long text. I’ll try re-installing and report whether the problem persists. I can have another look at the correct sync URI, and make a summary of symptoms.

[wbk]

Wohoooo!

(Did I make myself clear? WOHOOO!!)

After reinstalling, I payed close attention for a change to the actual URI that is shown.

• /1.0/sync/1.5 is what it says
• /token/1.0/sync/1.5 is what I had in my Firefox

My old syncserver installation had /ffsync/ as installation directory. I never noticed the superfluous ‘token’ in my configuration!

My desktop Firefox took up the change quickly enough, but on Android it took, of course, more trouble.

Apart from unlocking the sync debug by tapping the Firefox logo and exiting Firefox as described, it was also necessary to log out of the Firefox-account and log in anew (quite easily done via Mozilla accounts → the forum translates the actual URL to the page title, it is the firefox website .com/pair where you have to go)

So, for Firefox on Android, what I did:

• Firefox → menu → settings → about FF → tap logo → back to settings → go to top, tap “Sync Debug” just under the account, tap “Custom Sync server” and put https://ffs.domain.tld/1.0/sync/1.5 in the popup. OK, back, ‘Exit and restart Firefox’ (there’s a button)
• Restart Firefox; it did not work for me yet, in the server log I actually saw that Firefox was requesting the old URL from a couple of weeks ago
• Log out of the Firefox account, quit Firefox and app properties → kill Firefox (just to be sure)
• Restart Firefox, log in to account (via the pairing method)

The phone started syncing immediately, after being out of sync for two weeks.

I’m so happy it works now and I can upgrade the VPS I ran for this to a more recent version

Thanks for making the storage server available. Also a big thank you for everyone reading and sympathizing silently, and even more for those taking their precious time to help me out trying to correct this silly PEBKAC

PS: want to know an easy way to recognize that your profile has been synced for the first time with another profile? Autoscroll will be disabled on Linux (It is the default setting, and one way or another it is difficult to recognize that someone really likes to have autoscroll enabled with an extra tick or something in Firefox settings Anyway, if you wonder why your autoscroll gets disabled sometimes, search in that corner)