Synapse logs are huge

Support apps
,
1

My YunoHost server

Hardware: VPS bought online
YunoHost version: 4.3.6.2
I have access to my server : Through SSH | through the webadmin
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no

Description of my issue

I notice the disk space on my server was filling up and taking a look at what is taking up the most space, the latest homeserver.log file of synapse is 14 GB big. Which seems rather massive. A glance through the synapse docs, I canā€™t find anything helpful.

I assume I can delete the file with no consequences, but that may not solve the issue.

1 Like
2

Where is this .log located?

3

You make a statement about the log files. Do you have a question?

If you do have a question, donā€™t be shy and out with it! Donā€™t forget in that case to give some useful information, such as number of users, number of rooms and bridges, versions and such.

As a data point: my server does not see heavy traffic, has a handful of users, some 50 rooms, some of which are relatively active; synapseā€™s homeserver.log has grown to 60 MB. The previous log, homeserver.log.1, is a week old and is 100 MB.

The logs (by default) are in /var/log/matrix-synapse

2 Likes
4

Well, I want to ask something. Can we delete the log file? Is synapse going to create a new log file I delete it?

5

My question isā€¦ Should my homeserver.log file be so big?
From what you are saying you have a 60MB log for 50 rooms and a handful of users. I have only myself as an active user, and Iā€™m in no more than 10 rooms.
So it seems something is very wrong with my logs.
Any advice on where to look to find out what is causing this? I asked in the Synapse admins matrix rooms and got an unhelpful ā€˜read the docsā€™.

6

Hmmā€¦ I can start with the not-so-helpful ā€œWhat does it say in the logs?ā€ When you follow the log, with tail -f , do messages race by? In my case:


tail -f /var/log/matrix-synapse/homeserver.log
2022-03-09 20:23:00,990 - twisted - 275 - INFO - sentinel- Timing out client: IPv6Address(type='TCP', host='::ffff:188.68.38.222', port=39794, flowInfo=0, scopeID=0)
2022-03-09 20:23:01,548 - synapse.handlers.presence - 772 - INFO - persist_presence_changes-44614- Persisting 4 unpersisted presence updates
2022-03-09 20:23:03,077 - synapse.federation.transport.server.federation - 95 - INFO - PUT-803398- Received txn 1644862894031 from leprette.fr. (PDUs: 0, EDUs: 1)
2022-03-09 20:23:03,130 - synapse.access.https.8448 - 421 - INFO - PUT-803398- ::ffff:89.234.177.95 - 8448 - {leprette.fr} Processed request: 0.054sec/0.001sec (0.006sec, 0.000sec) (0.003sec/0.043sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1644862894031 HTTP/1.1" "Synapse/1.47.1" [0 dbevts]
2022-03-09 20:23:06,381 - synapse.federation.transport.server.federation - 95 - INFO - PUT-803399- Received txn 1646758312371 from remi.im. (PDUs: 0, EDUs: 1)
2022-03-09 20:23:06,431 - synapse.access.https.8448 - 421 - INFO - PUT-803399- ::ffff:77.173.199.109 - 8448 - {remi.im} Processed request: 0.052sec/0.001sec (0.005sec, 0.001sec) (0.002sec/0.041sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1646758312371 HTTP/1.1" "Synapse/1.53.0" [0 dbevts]
2022-03-09 20:23:14,633 - synapse.access.http.8008 - 421 - INFO - GET-803400- 83.232.57.155 - 8008 - {@user:homeserver} Processed request: 0.006sec/0.002sec (0.003sec, 0.000sec) (0.001sec/0.001sec/1) 333B 200 "GET /_matrix/client/r0/sync?filter=1&set_presence=offline&timeout=6000&since=s259183_4202135_15841_212534_7423_43_10255_514257_1 HTTP/1.0" "Element/1.3.9 (Linux; U; Android 12; SM-G970F Build/SP1A.210812.01; Flavour FDroid; MatrixAndroidSdk2 1.3.9)" [0 dbevts]
2022-03-09 20:23:18,980 - twisted - 275 - INFO - sentinel- Timing out client: IPv6Address(type='TCP', host='::ffff:144.76.146.18', port=43686, flowInfo=0, scopeID=0)
2022-03-09 20:23:20,747 - twisted - 275 - INFO - sentinel- Timing out client: IPv6Address(type='TCP', host='::ffff:95.217.202.35', port=55482, flowInfo=0, scopeID=0)
2022-03-09 20:23:21,228 - twisted - 275 - INFO - sentinel- Timing out client: IPv6Address(type='TCP', host='::ffff:168.119.236.153', port=55096, flowInfo=0, scopeID=0)
2022-03-09 20:23:25,972 - synapse.federation.transport.server.federation - 95 - INFO - PUT-803401- Received txn 1646755402155 from tilosp.de. (PDUs: 0, EDUs: 1)

So, about a line per second.

If it is many more than that, maybe logging is at debug in the settings. Mine is at INFO, most places:

cat /etc/matrix-synapse/log.yaml 

version: 1

formatters:
  precise:
   format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s- %(message)s'

filters:
  context:
    (): synapse.util.logcontext.LoggingContextFilter
    request: ""

handlers:
  file:
    class: logging.handlers.RotatingFileHandler
    formatter: precise
    filename: /var/log/matrix-synapse/homeserver.log
    filters: [context]
    level: INFO
  console:
    class: logging.StreamHandler
    formatter: precise
    level: WARN

loggers:
    synapse:
        level: INFO

    synapse.storage.SQL:
        level: INFO

root:
    level: INFO
    handlers: [file, console]

There is also a homeserver-specific logconfig, /etc/matrix-synapse/domain.tld.log.config, it is also at INFO.

Yours?

7

Yes, log files can be deleted safely. I have not encountered a program that did not recreate the log file when it was missing. Log directories are often not created automatically, so only throw the files away.

There is (often) a setting ā€˜logrotateā€™ that will start a new log daily or weekly to help keep the size down (I thought I saw it somewhere in the configs, but I donā€™t find it back now).

1 Like
8

Thanks I had a look at the log and nothing stood out as unusual. my logging level was also set to INFO.

I decided to delete and live with the consequences, and all seems to be running fine still. Thanks for your time and advice.

Iā€™ll mark this as solved and keep an eye on the file to see if it grows in size again.

1 Like
9

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.