Metronome using all available memory

mouse · February 1, 2022, 10:58am

My YunoHost server

Hardware: Raspberry Pi at home
YunoHost version: 4.3.6.2 (stable).
I have access to my server : Through SSH | through the webadmin
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no

Description of my issue

hey ynh-land.

recently i have noticed my server being really slow, and xmpp messages not sending. when i look into htop on the server, metronome is using 100% of a CPU core and gradually swallowing all free memory.

i firstly restarted metronome visa systemctl, it worked fine for the rest of the day. then next the same issue has come up again. now if i restart it only takes 5-10 minutes for metronome to use all available memory and stall the whole server.

i am running a modified metronome.cfg.lua file: i added modules and ports for a/v calls, turn/stun server. but that was long ago, and it has worked fine since.

i’m not sure if there have been any config updates that i should merge with my version?

i’m a bit surprised by this behaviour, as i have run prosody on other servers for years at a time without ever seeing this.

metronome’s error log file is empty, while its standard log file doesn’t show anything disastrous.

perhaps someone else has had a similar issue, or has a suggestion of where to start digging around?

sudo journalctl -u metronome.service also shows nothing other than my restarting it.

we only have a handful of users, and only connect to a handful of other servers.

one thing i noticed in metronome.log is that metronome is trying to connect to the subdomain jabber.my.beloved.ynh.server, about which is says this:

incoming s2s stream my.beloved.ynh->jabber.my.beloved.ynh closed: This host does not serve jabber.my.beloved.ynh

the only other error is a failed connected to a now defunct/offline server.

mouse · February 1, 2022, 2:28pm

logs are here
https://paste.yunohost.org/racozisidi

wbk · February 1, 2022, 9:52pm

I noticed there are XMPP CNAME’s for multiple domains, does Metronome serve multiple domains?
It should not be a problem of course, but it reminds me that since a recent version there is an option to turn off XMPP for domains in the Yunohost domain settings. Perhaps in that same set of changes something happened that does not sit well with your added modules.

mouse · February 2, 2022, 9:48am

@wbk, thanks for your suggestion.

in the domain settings in the web admin, xmpp is only enabled for the main domain, not any further subdomains.

but i recall seeing dns for xmpp for other domains. i never did anything about it because i thought it was just the ynh automated default.

mouse · February 2, 2022, 10:22am

my ynh does also serve other domains. i have external web-app to host other sites on my server, and these are the ones that metronome is trying to use with ‘xmpp-upload’.

but i have no config for them in metronome.cfg.lua.

wbk · February 2, 2022, 10:11pm

I don’t know whether by default XMPP is only enabled for the main domain, or that it is enabled for all domains, or whether there has been any change in that the past half year.

There has been a change in the options to configure it in the web admin, to suggest (and in some cases configure) DNS records.

On my Yunohost there is more than one domain, and without thinking about it I chat with an account on one domain via XMPP with an account on another domain, so XMPP is active for at least a couple of domains.

All this probably does not have a relation with your problem. You could try whether disabling XMPP in the domain settings and removing the CNAME’s from DNS has the effect that those not-related domains are not mentioned in the logs anymore.

If someone with better knowledge of XMPP and Metronome comes along, they can probably help you more than me!

mouse · February 19, 2022, 11:11am

hmm so i was wrong, it looks like theres a full xmpp conf for every single subdomain on the server!

i’m surprised ynh does that, as i’d only ever need a single domain, and why would all my apps each need their own xmpp config?

maybe adding extdisco/jingle things for turn/stun to 10 or so domains chokes up metronome…?

mouse · February 19, 2022, 11:27am

my config changes are these:

github.com/YunoHost/issues

make XMPP audio/video works with at least conversations

opened 08:20PM - 12 May 20 UTC

larchange

enhancement

XMPP Priority: high

I've enable audio video on my yunohost instance with the following changes. I …believe these configuration are enough to make A/V work even on 3G/different wifi etc. # Configure /etc/metronome/metronome.cfg.lua ## enable module ``` "extdisco"; -- XEP 215 "jingle_nodes"; ``` ## Add config: ``` external_services = { ["domain.tld"] = { [1] = { port = "3478", transport = "udp", type = "stun" }, [2] = { port = "3478", transport = "tcp", type = "stun" }, [3] = { port = "5349", transport = "tcp", type = "stuns" }, [4] = { port = "3478", transport = "tcp", type = "turn", turn_secret = "Some passphrase", turn_ttl = 300 }, [5] = { port = "3478", transport = "udp", type = "turn", turn_secret = "Some passphrase", turn_ttl = 7200 }, [6] = { port = "5349", transport = "tcp", type = "turns", turn_secret = "Some passphrase", turn_ttl = 7200 } } }; jingle_nodes_turn_credentials = true; jingle_nodes_turn_secret = "Some passphrase"; jingle_nodes_turn_credentials_ttl = 86400; jingle_nodes_restricted = false; ``` # Install coturn And create config file /etc/turnserver.conf ``` listening-port=3478 tls-listening-port=5349 listening-ip=0.0.0.0 listening-ip=<ipv6> min-port=49152 max-port=65535 verbose fingerprint use-auth-secret static-auth-secret="Some passphrase" realm=domain.tld max-allocate-lifetime=7200 cert=/etc/yunohost/certs/domain.tld/crt.pem pkey=/etc/yunohost/certs/domain.tld/key.pem syslog prod no-loopback-peers no-cli cli-ip=127.0.0.1 cli-port=5766 cli-password=clipassword ``` # Open corresponding ports ``` yunohost firewall allow UDP -4 49152:65535 yunohost firewall allow Both 3478 yunohost firewall allow Both 5349 yunohost firewall allow Both 3479 yunohost firewall allow Both 5350 ``` I'm open to make some changes, but I'm completely new on the project and will welcome some help.

mouse · February 19, 2022, 11:29am

i wonder it is is fine to just remove all the unneeded xmpp Virtual Hosts in metronome’s /conf.d/ directory so that it doesn’t worry about all those extra subdomains, with all their extra extdisco services?

mouse · February 19, 2022, 11:45am

i tried removing all other domains, and the problem persists

mouse · February 19, 2022, 11:56am

i tried removing all jingle/stun/turn config (details in the above github issue), and memory usage remained excessive.

system · March 21, 2022, 11:56am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.