Mattermost service failing after server out of storage

My YunoHost server

Hardware: VPS OVH 2G/20G
YunoHost version: 11.2.10.3 (stable).
I have access to my server : Through SSH | through the webadmin
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : yes
If yes, please explain: errors happened after the vps run out of space, but after resolving the low space issue they persist
If your request is related to an app, specify its name and version:
mattermost 9.5.2~ynh1

Description of my issue

My vps run out of storage space (nextcloud upload folder was the culprit, it seems that it is caused by the nextcloud client losing network, so I have deleted its contents), almost all services stopped working. So I cleaned up unnecessary files and data. All services were running after a reboot but mattermost didn’t.
Restarting it manually failed. I had a look at its log and it was about wrong login credentials to database. Here is the log :

https://paste.yunohost.org/kusitosafu

I will be grateful for any help. Thanks a lot

Hi jarod5001,

Is Mattermost the only application running on Postgres, or do other programs that use Postgres still work?

I’d try logging into Postgres directly, and see if the Mattermost database is accessible. Perhaps superfluous:

$ sudo su postgres - 
$ psql 
postgres=# \l  (backslash L, to list databases)
                                        List of databases
      Name      |     Owner      | Encoding |   Collate   |    Ctype    |   Access privileges    
----------------+----------------+----------+-------------+-------------+------------------------
 matrix_synapse | matrix_synapse | UTF8     | C           | C           | 
 postgres       | postgres       | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 mattermost ?
(x rows)
postgres=# \c mattermost
You are now connected to database "mattermost" as user "postgres".
forgejo=# 

My reasoning is that probably the credentials are fine, but the database itself not. This test would give some info about that.

It’s a bit of a shot in the dark, but here’s a thread on stackexchange from someone unable to restart PG after filling storage, postgresql - Postgres cannot be restarted after disk ran out of space - Database Administrators Stack Exchange

1 Like

Est-ce que tu peux te connecter à la base de données pgsql ?

su postgres
psql mattermost
\l
1 Like

I can connect to postgre :

mattermost=# \l
                                       List of databases
    Name    |   Owner    | Encoding |   Collate   |    Ctype    |      Access privileges       
------------+------------+----------+-------------+-------------+------------------------------
 hedgedoc   | hedgedoc   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/hedgedoc                +
            |            |          |             |             | hedgedoc=C*T*c*/hedgedoc
 mattermost | mattermost | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/mattermost              +
            |            |          |             |             | mattermost=C*T*c*/mattermost
 postgres   | postgres   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0  | postgres   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres                 +
            |            |          |             |             | postgres=CTc/postgres
 template1  | postgres   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres                 +
            |            |          |             |             | postgres=CTc/postgres
(5 rows)

Hedgdoc is running fine.
Thanks @wbk @rodinux
What’s the next step?

If Mattermost’s database is fine, this should work

postgres=# \c mattermost
You are now connected to database "mattermost" as user "postgres".
mattermost=#

After that… I’m out of words :stuck_out_tongue:

I think the password for mmuser that tries to connect the mattermost-service is encrypted/hashed somewhere in the Mattermost configuration. Perhaps you can set a new password, you’d need to know how to get from the new plaintext password to the hashed/encrypted one to set in the configuration.

Or, if mmuser has a password that is known to you, you might try using those credentials to log in to psql directly.

If the database is in order and all else fails with the login: maybe you can export/backup all data from Postgres directly, put restore a backup of Mattermost, and overwrite the data with what you just exported from Postgres. There may be tools to help with this, but it sounds a bit heavy handed.

1 Like

Thanks a lot for taking the time to respond.
And me have no idea how to check or where.
May be backup the mattermost data, remove the app and reinstall it.
I have no experience with posgresql, I have basic knowledge of mysql from cli. So I won’t be able to fix it by mayself.

After this error in the log you have
Error: failed to initialize platform: cannot create store: error setting up connections: pq: password authentication failed for user "mmuser"

Perhaps try this

GRANT ALL ON DATABASE mattermost TO mmuser;
ALTER DATABASE mattermost OWNER TO mmuser;
GRANT USAGE, CREATE ON SCHEMA PUBLIC TO mmuser;

source:

2 Likes

ERROR: role "mmuser" does not exist

II think I have to create a new one.
Another thing that I don’t understand, in the config.json of mattermost, the datasource line does not contain mmuser but mattermost. Compared to another instance of mattermost I have at home, I see a password in that line that is missing in the broken instance.
I’m not sure if I fix the psql problem, how many other issues will there be.

Strange!

There you have a point…

Is there a backup of the broken instance, that might have a correct config.json?

(I was unsuccessful in trying to create a reproduction)

I tried to install Mattermost to see if I could learn anything about the connection string, or the way the password or the login process is handled, but my installation (on a just-installed) Yunohost went awry on missing directories, https://paste.yunohost.org/raw/opigicaxax . Don’t let it distract you from trying to repair your instance, it was just a test :wink:

1 Like

I have removed the app and removed the role added in postgresql. Reinstall the app successfully. And unfortunately I don’t have a backup. That’s not a big deal since there was not much to lose in the app till now.

Glad you got it solved, more or less, without losing data :slight_smile:

I have heard of Mattermost, but never used it. What do you (intend to) use it for, if you don’t mind me asking?

1 Like

I am using it on my home server. Only family. I use it to exchange documents to my wife for her work and to send documents to my daughter for her studies. And when I send a file, I send it with tags in the comment. So it becomes easier to filter documents. I also use it for communication. It has clients for all my devices.
On my vps, I’m planning to use it with two colleagues to work on a project. It’s possible to create a private channel (hidden to members in the server that are not members of that channel).
There are more options of mattermost that I am not using till now. It’s a good option for team work

1 Like

Thanks for the write-up :slight_smile:

I use Nextcloud for the same, but haven’t explored tags in that context yet.

Even with Nextcloud I’m far away from the limits of the functionality, while Mattermost covers the same besides having interaction as a focus and even more integrations.

Have fun with the reinstalled instance :slight_smile:

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.