Synapse logs are huge

My YunoHost server

Hardware: VPS bought online
YunoHost version: 4.3.6.2
I have access to my server : Through SSH | through the webadmin
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no

Description of my issue

I notice the disk space on my server was filling up and taking a look at what is taking up the most space, the latest homeserver.log file of synapse is 14 GB big. Which seems rather massive. A glance through the synapse docs, I can’t find anything helpful.

I assume I can delete the file with no consequences, but that may not solve the issue.

1 Like

Where is this .log located?

You make a statement about the log files. Do you have a question?

If you do have a question, don’t be shy and out with it! Don’t forget in that case to give some useful information, such as number of users, number of rooms and bridges, versions and such.

As a data point: my server does not see heavy traffic, has a handful of users, some 50 rooms, some of which are relatively active; synapse’s homeserver.log has grown to 60 MB. The previous log, homeserver.log.1, is a week old and is 100 MB.

The logs (by default) are in /var/log/matrix-synapse

2 Likes

Well, I want to ask something. Can we delete the log file? Is synapse going to create a new log file I delete it?

My question is… Should my homeserver.log file be so big?
From what you are saying you have a 60MB log for 50 rooms and a handful of users. I have only myself as an active user, and I’m in no more than 10 rooms.
So it seems something is very wrong with my logs.
Any advice on where to look to find out what is causing this? I asked in the Synapse admins matrix rooms and got an unhelpful ‘read the docs’.

Hmm… I can start with the not-so-helpful “What does it say in the logs?” When you follow the log, with tail -f , do messages race by? In my case:


tail -f /var/log/matrix-synapse/homeserver.log
2022-03-09 20:23:00,990 - twisted - 275 - INFO - sentinel- Timing out client: IPv6Address(type='TCP', host='::ffff:188.68.38.222', port=39794, flowInfo=0, scopeID=0)
2022-03-09 20:23:01,548 - synapse.handlers.presence - 772 - INFO - persist_presence_changes-44614- Persisting 4 unpersisted presence updates
2022-03-09 20:23:03,077 - synapse.federation.transport.server.federation - 95 - INFO - PUT-803398- Received txn 1644862894031 from leprette.fr. (PDUs: 0, EDUs: 1)
2022-03-09 20:23:03,130 - synapse.access.https.8448 - 421 - INFO - PUT-803398- ::ffff:89.234.177.95 - 8448 - {leprette.fr} Processed request: 0.054sec/0.001sec (0.006sec, 0.000sec) (0.003sec/0.043sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1644862894031 HTTP/1.1" "Synapse/1.47.1" [0 dbevts]
2022-03-09 20:23:06,381 - synapse.federation.transport.server.federation - 95 - INFO - PUT-803399- Received txn 1646758312371 from remi.im. (PDUs: 0, EDUs: 1)
2022-03-09 20:23:06,431 - synapse.access.https.8448 - 421 - INFO - PUT-803399- ::ffff:77.173.199.109 - 8448 - {remi.im} Processed request: 0.052sec/0.001sec (0.005sec, 0.001sec) (0.002sec/0.041sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1646758312371 HTTP/1.1" "Synapse/1.53.0" [0 dbevts]
2022-03-09 20:23:14,633 - synapse.access.http.8008 - 421 - INFO - GET-803400- 83.232.57.155 - 8008 - {@user:homeserver} Processed request: 0.006sec/0.002sec (0.003sec, 0.000sec) (0.001sec/0.001sec/1) 333B 200 "GET /_matrix/client/r0/sync?filter=1&set_presence=offline&timeout=6000&since=s259183_4202135_15841_212534_7423_43_10255_514257_1 HTTP/1.0" "Element/1.3.9 (Linux; U; Android 12; SM-G970F Build/SP1A.210812.01; Flavour FDroid; MatrixAndroidSdk2 1.3.9)" [0 dbevts]
2022-03-09 20:23:18,980 - twisted - 275 - INFO - sentinel- Timing out client: IPv6Address(type='TCP', host='::ffff:144.76.146.18', port=43686, flowInfo=0, scopeID=0)
2022-03-09 20:23:20,747 - twisted - 275 - INFO - sentinel- Timing out client: IPv6Address(type='TCP', host='::ffff:95.217.202.35', port=55482, flowInfo=0, scopeID=0)
2022-03-09 20:23:21,228 - twisted - 275 - INFO - sentinel- Timing out client: IPv6Address(type='TCP', host='::ffff:168.119.236.153', port=55096, flowInfo=0, scopeID=0)
2022-03-09 20:23:25,972 - synapse.federation.transport.server.federation - 95 - INFO - PUT-803401- Received txn 1646755402155 from tilosp.de. (PDUs: 0, EDUs: 1)

So, about a line per second.

If it is many more than that, maybe logging is at debug in the settings. Mine is at INFO, most places:

cat /etc/matrix-synapse/log.yaml 

version: 1

formatters:
  precise:
   format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s- %(message)s'

filters:
  context:
    (): synapse.util.logcontext.LoggingContextFilter
    request: ""

handlers:
  file:
    class: logging.handlers.RotatingFileHandler
    formatter: precise
    filename: /var/log/matrix-synapse/homeserver.log
    filters: [context]
    level: INFO
  console:
    class: logging.StreamHandler
    formatter: precise
    level: WARN

loggers:
    synapse:
        level: INFO

    synapse.storage.SQL:
        level: INFO

root:
    level: INFO
    handlers: [file, console]

There is also a homeserver-specific logconfig, /etc/matrix-synapse/domain.tld.log.config, it is also at INFO.

Yours?

Yes, log files can be deleted safely. I have not encountered a program that did not recreate the log file when it was missing. Log directories are often not created automatically, so only throw the files away.

There is (often) a setting ‘logrotate’ that will start a new log daily or weekly to help keep the size down (I thought I saw it somewhere in the configs, but I don’t find it back now).

1 Like

Thanks I had a look at the log and nothing stood out as unusual. my logging level was also set to INFO.

I decided to delete and live with the consequences, and all seems to be running fine still. Thanks for your time and advice.

I’ll mark this as solved and keep an eye on the file to see if it grows in size again.

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.