Hardware: VM in a computer YunoHost version: 11.2.4 I have access to my server : Through SSH, through the webadmin Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no If your request is related to an app, specify its name and version: synapse v1.92.3~ynh1
Description of my issue
It’s been a few days (at least) that synpase is overloading my server. Load is consistently around 4 and never goes down.
CPU usage of synapse process (/opt/yunohost/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ as reported by ps) is constantly around 8-10%, sometimes much higher, RAM usage is around 50%.
I don’t know if that’s supposed to be normal, I never checked it in the past, but it doesn’t seem too normal. I found it out because the whole server started being slow and other apps would bug.
The yunohost VM containing synapse has 8Go of RAM, matrix users are probably quite few (less than 10). It has been operating for years without any issues with synapse. No setup was changed recently, apart from a RAID disk replacement, that completed without issues.
The issue started occurring with the previous version of synapse, I upgraded to the latest but nothing changed.
I was wondering if anyone had ever had such issue with synapse and if yes how they debugged it to understand what’s going wrong with it.
Can you try the Synapse Admin app and have a look at big rooms ?
Maybe some of your users joined a heavy room, that is constantly sending information to your server ?
(On my side, Synapse have always been the heaviest process, but like 5% of the CPU, sometime more, but not for long)
Hey @Mamie, thanks for your suggestion. I’m trying to check out synapse admin but it’s giving some difficulties.
Installing worked fine, but then I can’t log into it. As soon as I enter the homeserver url, I can only click onto “sign in with sso”, I do it log in, get redirected to the sso portal, and nothing more, if I go back to synapse admin it asks again to sign in.
I checked all I could think could cause issues.
“Synapse (Server administration API.)” permission is given to visitors, sounds a bit extreme to me, but I didn’t do it, maybe installing Synapse admin did it.
The user I try to log in with is a synapse administrator (at least I ran the following command: su --command="psql matrix_synapse" postgres <<< "UPDATE users SET admin = 1 WHERE name = '@user2beAdmin:domain.tld'" with the appropriate username and domain).
I’m running out of ideas of what could be going wrong. But it definitely seems to be that Synapse admin doesn’t manage to have access to synapse administration api, since in the web console it logs: GET https://synapsedomain.tld/_synapse/admin/v2/users?deactivated=false&dir=f&from=0&guests=true&limit=10&order_by=name [HTTP/2 401 Unauthorized 165ms]
Well I came with a problem, now I found some more of them
I’m running out of ideas on how to debug the issue. It seems to be an issue with SSO. I posted about the situation on synapse_ynh repository.
I always found it very hard to debug issues between synapse, element, and SSO. But I don’t think it’s even something synapse-admin packager can do anything about, it’s probably down to synapse-admin app itself, maybe even react-admin.
I find synapse/matrix/element very hard to deal with at a small hosting level. It’s often quite discouraging, because every time a small issue comes up, it becomes an endless chain of issues to resolve, and oftentime I just end up giving up because it’s too many complications and I’m not able to put into it the big learning time it would require.
It’s a complex software stack that requires quite some knowledge for debugging. Also it’s very fast changing, so new issues come up regularly. I wonder if it makes sense for a small hosting level. Which is a pity, because it seemed to be the point of it.
Just some personal considerations, I wonder about other yunohost users experiences with it.
Just my experience : Matrix is great, but the encryption mode can be quite complexe, and needs just a little bit of brain to make it work, and sadly, users prefers Whatsapp because it works.
I forced Matrix for my family and some association, but it was clearly forced, and my choice is sometime disliked. (But not by me).
Another point is that Synapse is maybe not the best server anymore, it is thinked to hoste HUGE amount of people, not at all whatus YunoHost’s user are.
Maybe the other servers will be better for us : Matrix.org - Servers
But they are all still in beta.
[UPDATE] This is solved! You just need to update your synapse.conf and nginx conf for your synapse domain. See here for more details. It may also be fixed in a package update which seems to be being worked on.
I seem to be having this error as well. As soon as I edit the home server in the Element Android app or Element Web, the only option it gives me is “Sign in with CAS” which takes me to the Yunohost SSO login and then never returns to the app after signing in. It seems that the URL it’s opening is just DOMAINNAME.TLD/yunohost/sso/ without any kind of asking for a token to return to the app or anything.
I’ve updated the issue you listed with logs and further details. @ericg I see you seem active in synapse_ynh. If there’s further info/testing I can help with, just let me know.
@chateau Could you maybe update the title of this post to reflect that the issue isn’t just resource usage, but now not being able to log in?
Hey @Zinkie, thanks a lot for the update, ok I’ll wait for the synapse_ynh update to check if I can use synapse admin then.
For the title of this thread, I’ve added something about the login issue, I hope it doesn’t make it just more confusing. The issue I was having about synapse using too much resources is still ongoing, and I realize now that not being able to use Synapse Admin was actually a most probably unrelated side issue trying to resolve the first one. Maybe the best would be to separate the posts about login issues to a dedicated thread (though I believe I’d have to be a forum admin to be able to do that).
Anyway, sorry for the confusion. When I’m able to use synapse admin I’ll get back to trying to resolve the original resources consumption issue and post updates on it here.
