Need testers for the latest Mattermost upgrade

emilien · March 16, 2023, 11:17am

I can connect inside the browser on Android, so it’s public no ? If not where I can change this ?

I use the nohost address with let’s encrypt certificate, the server URL is like : https://myserver.nohost.me/chat is it ok or I must use a subdomain like https://chat.myserver.nohost.me ?

[EDIT] I changed to https://chat.myserver.nohost.me, it didn’t resolve my issue.

hibar84 · March 19, 2023, 1:43pm

Hello, j’ai tenté d’installer la mise à jour sur la branche testing, j’ai eu une erreur durant l’installation, voici les logs:
https://paste.yunohost.org/raw/amuzoxudip

Je suis sur une carte Olimex (ARM), c’est peut être la raison ?

Bonne journée!

ericg · March 19, 2023, 1:50pm

maybe a pgloader incompatibility with arm32… → KABOOM! FATAL error: There is no thread support in this instance. · Issue #1101 · dimitri/pgloader · GitHub

vins · April 1, 2023, 5:54pm

Big db for us I guess, 3,7GB… and an error for pgloader :

Info: Migrating to PostgreSQL database...
Warning: Heap exhausted during garbage collection: 32768 bytes available, 35264 requested.
Warning: Gen  Boxed   Code    Raw  LgBox LgCode  LgRaw  Pin       Alloc     Waste        Trig      WP GCs Mem-age
Warning:  1     245      0  35694      0      0      5    8   886813504 290999488    42949672   35944   0  1.0028
Warning:  2     575      0  61556     25      0     25   98  1532421408 505125600     2000000   62181   0  0.2489
Warning:  3       0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
Warning:  4       0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
Warning:  5       0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
Warning:  6    1323      7    985    199      0     56    0    82244624   1969136     2000000    2570   0  0.0000
Warning:  7     224      0  30092      0      0     15    0   748244848 245641360     2000000   30331   0  0.0000
Warning:            Total bytes allocated    =    3249724384
Warning:            Dynamic-space-size bytes =    4294967296
Warning: GC control variables:
Warning:    *GC-INHIBIT* = true
Warning:    *GC-PENDING* = true
Warning:    *STOP-FOR-GC-PENDING* = false
Warning: fatal error encountered in SBCL pid 2924(tid 0x7fd9c0e1f700):
Warning: Heap exhausted, game over.

The solution seems to be reduce the pgloader option :

github.com/dimitri/pgloader

Heap exhausted during garbage collection

opened 12:01PM - 07 May 19 UTC

AnrDaemon

While trying to run a full new import using fresh pgloader container, I stumbled… upon ``` 2019-05-07T11:56:36.063000Z DATA < #(" 0" "1e73536d-364a-4ac5-9556-26080ae675be" "a2159ea6-dcd4-4bfb-a659-8a30d17471a3" " 7" "000" "0" " 0" "008" "" " 2" "2014-06-03" "НСТ Весна-6" "5074" "5074" "c24a92ab-0163-469a-a56a-081cf40ddbe4" "НСТ Весна-6" "46436000000" "46736000" "21" "5a11404e-3a4f-4943-b60c-51480729dcfb" "000" "500000080000260" "142180" "02bf0bee-3882-4364-8ad1-dc004a0ef2be" "50" "тер" "2014-02-28" "0260" "" "" "2016-08-09" "000" "0000" "000" " 0" "3e5ba4cb-7b5e-4258-86e0-8b91ca39d601" "0000" "" "0") 2019-05-07T11:56:36.065000Z DATA < #(" 0" "2dbae092-4d28-4692-bfd7-509c92eb6896" "30e0f893-20d3-4c65-b476-7db50ad93492" " 7" "000" "0" " 0" "008" "" " 2" "2014-06-03" "Овальная" "5074" "5074" "5e1ee261-ee43-4004-8c93-2f5f33d3c864" "Овальная" "46436000000" "46736000" "21" "5a11404e-3a4f-4943-b60c-51480729dcfb" "000" "500000080000051" "142180" "2e68e5bf-0f7b-4bab-aac9-e99e57445dfc" "50" "ул" "2014-02-28" "0051" "" "" "2016-04-14" "000" "0000" "000" " 0" "3e5ba4cb-7b5e-4258-86e0-8b91ca39d601" "0000" "" "0") Heap exhausted during garbage collection: 288 bytes available, 432 requested. Gen StaPg UbSta LaSta LUbSt Boxed Unboxed LB LUB !move Alloc Waste Trig WP GCs Mem-age 0: 0 0 0 0 0 0 0 0 0 0 0 10737418 0 0 0.0000 1: 0 0 0 0 0 0 0 0 0 0 0 10737418 0 0 0.0000 2: 28005 28307 0 0 855 9423 0 0 12 334403376 2386128 10737418 0 0 0.5853 3: 10322 32767 0 0 793 7934 8 0 0 284273120 1955360 10737418 0 0 0.0000 4: 0 0 0 0 0 0 0 0 0 0 0 10737418 0 0 0.0000 5: 0 0 0 0 1027 9584 38 0 50 345980496 2965936 356717914 769 1 0.0000 6: 0 0 0 0 1886 1220 0 0 0 101777408 0 2000000 1787 0 0.0000 Total bytes allocated = 1066434400 Dynamic-space-size bytes = 1073741824 GC control variables: *GC-INHIBIT* = true *GC-PENDING* = false *STOP-FOR-GC-PENDING* = false fatal error encountered in SBCL pid 133(tid 140627020216064): Heap exhausted, game over. Welcome to LDB, a low-level debugger for the Lisp runtime environment. ldb> ``` I can assert that nothing has significantly changed on my end, this is the same exact unaltered files I've imported last time, and the only change was in loader script, and it was minimal: ```diff - SET search_path to 'fias' + SET search_path TO 'fias ```

kemenaran · April 2, 2023, 2:00pm

I made a branch that reduces the prefetch count. You can test it by running:

sudo yunohost app upgrade mattermost --url https://github.com/YunoHost-Apps/mattermost_ynh/tree/fix-large-migrations

Does it work better on your database?

vins · April 2, 2023, 5:15pm

just trying…

vins · April 2, 2023, 5:18pm

arf the database already exists because of the last failed upgrade…

vins · April 2, 2023, 5:41pm

oups sorry…

the exact output was :

root@yunohost:~# yunohost app upgrade mattermost --url https://github.com/YunoHost-Apps/mattermost_ynh/tree/fix-large-migrations
Info: Now upgrading mattermost...
Info: [+...................] > Loading installation settings...
Info: [#...................] > Checking version...
Info: [#+..................] > Backing up the app before upgrading (may take a while)...
Info: [##++++++............] > Upgrading dependencies...
Info: Migrating to PostgreSQL database...
Warning: ERROR:  database "mattermost" already exists
Warning: Failed to run Mattermost to create PostgreSQL database tables
Warning: [Error] Upgrade failed.
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: This action broke dpkg/APT (the system package managers)... You can try to solve this issue by connecting through SSH and running `sudo apt install --fix-broken` and/or `sudo dpkg --configure -a`.
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
Warning: sh: 0: getcwd() failed: No such file or directory
^CError: Could not upgrade mattermost: The operation was manually interrupted?
Info: The operation 'Upgrade the 'mattermost' app' could not be completed. Please share the full log of this operation using the command 'yunohost log share 20230402-171513-app_upgrade-mattermost' to get help
^CError: Operation interrupted
^CException ignored in: <module 'threading' from '/usr/lib/python3.9/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 1428, in _shutdown
    lock.acquire()
KeyboardInterrupt:

but when I sudo postgres and psql I donnot see with \l the mattermost db !
After that big trash :
the only backup available from which I can do a restore is mattermost-pre-upgrade1
https://paste.yunohost.org/raw/orurerumob

No time now to investigate I must leave, I restore a backup of the VM.
No lost for us, but big bug…

vins · April 2, 2023, 6:25pm

good, I have some time in the end. So after the backup (this night, the aborted migration was yesterday), I can see that the postgres table mattermost exists. So deleting it should permit to have a successfull upgrade.
What about having a check to handle these tricky case ?
So after a manual drop of the postgreSQL db mattermost the upgrade with the branch fix-large-migrations is successfull.
I thing that adding a check if such a mattermost db exists in postgreSQL could be a good thing

root@yunohost:~# yunohost app upgrade mattermost --url https://github.com/YunoHost-Apps/mattermost_ynh/tree/fix-large-migrations
Info: Now upgrading mattermost...
Info: [+...................] > Loading installation settings...
Info: [#...................] > Checking version...
Info: [#+..................] > Backing up the app before upgrading (may take a while)...
Info: [##++++++............] > Upgrading dependencies...
Info: Migrating to PostgreSQL database...
Warning: NOTICE:  word is too long to be indexed
Warning: DETAIL:  Words longer than 2047 characters are ignored.
Warning: NOTICE:  word is too long to be indexed
Warning: DETAIL:  Words longer than 2047 characters are ignored.
Warning: NOTICE:  word is too long to be indexed
Warning: DETAIL:  Words longer than 2047 characters are ignored.
Info: [########++..........] > Stopping a systemd service...
Info: [##########..........] > Making sure dedicated system user exists...
Info: [##########++........] > Upgrading source files...
Info: [############+++.....] > Upgrading NGINX web server configuration...
Info: [###############+....] > Upgrading systemd configuration...
Info: [################....] > Upgrading logrotate configuration...
Info: [################+...] > Integrating service in YunoHost...
Info: [#################+..] > Starting a systemd service...
Info: The service mattermost has correctly executed the action start.
Info: [##################+.] > Reloading NGINX web server...
Info: [####################] > Upgrade of mattermost completed
Success! mattermost upgraded
Success! Upgrade complete

kemenaran · April 3, 2023, 10:03am

Pfew, glad you could make it work.

In case of a failed migration, the script is supposed to delete the partial table in Postgres. For some reason it didn’t, maybe because of the way pgloader failed… Sorry for this. Anyway, I’ll try to merge the bugfix. Thanks for the report!

TheNomad11 · April 16, 2023, 9:34am

Hm, I am a bit confused about this upgrade. 7.8.1~ynh1 has been published several weeks ago. Is it safe to upgrade? There still seem to be issues?

Nevertheless thanks for your work, a complex issue with databases!

ptrx · April 18, 2023, 7:18pm

Hello!

I’m also having difficulties upgrading to v7.8.1~ynh1 (from 7.2.0~ynh1) on a vm (not arm).

The log extract looks like this:

Info : DEBUG - + systemctl stop mattermost
Info : DEBUG - + set +e
Info : DEBUG - + sudo -u mattermost timeout --preserve-status 180 ./bin/mattermost
Info : DEBUG - + '[' 143 '!=' 0 ']'
Info : DEBUG - + '[' 0 '!=' 143 ']'
Info : WARNING - Failed to run Mattermost to create PostgreSQL database tables
Info : DEBUG - + ynh_exit_properly

Here are the complete logs.
I’ve tried with the master branch as well as the fix-large-migrations one.
Not sure what to do now If anyone has an idea I’m open!

kemenaran · April 25, 2023, 4:14pm

Hmm, so the initial dummy run of Mattermost to create the tables structures failed, or timed out. If you’re running on a VM, maybe it just didn’t go enough time to complete.

Do you have better results using this branch, which has an increased timeout at this point?

sudo yunohost app upgrade mattermost --url https://github.com/YunoHost-Apps/mattermost_ynh/tree/increase-migration-timeout

Vanitarium · May 1, 2023, 2:03pm

Done the upgrade and here is the log:

yunohost log show 20230501-135527-app_upgrade-mattermost
description: Upgrade the 'mattermost' app
log_path: /var/log/yunohost/categories/operation/20230501-135527-app_upgrade-mattermost.log
logs: 
  - 2023-05-01 14:55:53,484: DEBUG - + set +o xtrace
  - 2023-05-01 14:55:53,534: DEBUG - + set +o xtrace
  - 2023-05-01 14:55:53,544: DEBUG - + echo '! Helper used in legacy mode !'
  - 2023-05-01 14:55:53,544: DEBUG - + set +x
  - 2023-05-01 14:55:53,545: DEBUG - + echo '[#################+..] > Reloading NGINX web server...'
  - 2023-05-01 14:55:53,546: DEBUG - + set -o xtrace
  - 2023-05-01 14:55:53,546: INFO - [#################+..] > Reloading NGINX web server...
  - 2023-05-01 14:55:53,546: DEBUG - + ynh_systemd_action --service_name=nginx --action=reload
  - 2023-05-01 14:55:53,546: DEBUG - + local legacy_args=nalpte
  - 2023-05-01 14:55:53,546: DEBUG - + args_array=(['n']='service_name=' ['a']='action=' ['l']='line_match=' ['p']='log_path=' ['t']='timeout=' ['e']='length=')
  - 2023-05-01 14:55:53,546: DEBUG - + local -A args_array
  - 2023-05-01 14:55:53,546: DEBUG - + local service_name
  - 2023-05-01 14:55:53,547: DEBUG - + local action
  - 2023-05-01 14:55:53,547: DEBUG - + local line_match
  - 2023-05-01 14:55:53,547: DEBUG - + local length
  - 2023-05-01 14:55:53,547: DEBUG - + local log_path
  - 2023-05-01 14:55:53,547: DEBUG - + local timeout
  - 2023-05-01 14:55:53,547: DEBUG - + ynh_handle_getopts_args --service_name=nginx --action=reload
  - 2023-05-01 14:55:53,547: DEBUG - + set +o xtrace
  - 2023-05-01 14:55:53,616: DEBUG - + service_name=nginx
  - 2023-05-01 14:55:53,616: DEBUG - + action=reload
  - 2023-05-01 14:55:53,616: DEBUG - + line_match=
  - 2023-05-01 14:55:53,616: DEBUG - + length=20
  - 2023-05-01 14:55:53,616: DEBUG - + log_path=/var/log/nginx/nginx.log
  - 2023-05-01 14:55:53,616: DEBUG - + timeout=300
  - 2023-05-01 14:55:53,617: DEBUG - + '[' reload == stop ']'
  - 2023-05-01 14:55:53,617: DEBUG - + [[ -n '' ]]
  - 2023-05-01 14:55:53,617: DEBUG - + '[' reload == reload ']'
  - 2023-05-01 14:55:53,617: DEBUG - + action=reload-or-restart
  - 2023-05-01 14:55:53,617: DEBUG - ++ date --utc --rfc-3339=seconds
  - 2023-05-01 14:55:53,617: DEBUG - ++ cut -d+ -f1
  - 2023-05-01 14:55:53,619: DEBUG - + local 'time_start=2023-05-01 13:55:53 UTC'
  - 2023-05-01 14:55:53,619: DEBUG - + systemctl reload-or-restart nginx
  - 2023-05-01 14:55:53,663: DEBUG - + [[ -n '' ]]
  - 2023-05-01 14:55:53,663: DEBUG - + ynh_script_progression '--message=Upgrade of mattermost completed' --last
  - 2023-05-01 14:55:53,664: DEBUG - + set +o xtrace
  - 2023-05-01 14:55:53,716: DEBUG - + set +o xtrace
  - 2023-05-01 14:55:53,726: DEBUG - + echo '! Helper used in legacy mode !'
  - 2023-05-01 14:55:53,726: DEBUG - + set +x
  - 2023-05-01 14:55:53,727: DEBUG - + echo '[####################] > Upgrade of mattermost completed'
  - 2023-05-01 14:55:53,728: INFO - [####################] > Upgrade of mattermost completed
  - 2023-05-01 14:55:53,728: DEBUG - + set -o xtrace
  - 2023-05-01 14:55:53,728: DEBUG - + ynh_exit_properly
  - 2023-05-01 14:55:53,728: DEBUG - + local exit_code=0
  - 2023-05-01 14:55:53,728: DEBUG - + [[ upgrade =~ ^install$|^upgrade$|^restore$ ]]
  - 2023-05-01 14:55:53,729: DEBUG - + rm -rf /var/cache/yunohost/download/
  - 2023-05-01 14:55:53,729: DEBUG - + '[' 0 -eq 0 ']'
  - 2023-05-01 14:55:53,729: DEBUG - + exit 0
  - 2023-05-01 14:55:54,730: DEBUG - Checking that required services are up and running...
  - 2023-05-01 14:55:55,348: SUCCESS - mattermost upgraded
metadata: 
  ended_at: 2023-05-01 14:55:55
  env: 
    BASH_XTRACEFD: 7
    NO_BACKUP_UPGRADE: 0
    YNH_APP_ACTION: upgrade
    YNH_APP_BASEDIR: /var/cache/yunohost/app_tmp_work_dirs/app_xx3dq9m4
    YNH_APP_CURRENT_VERSION: 7.8.1~ynh1
    YNH_APP_ID: mattermost
    YNH_APP_INSTANCE_NAME: mattermost
    YNH_APP_INSTANCE_NUMBER: 1
    YNH_APP_MANIFEST_VERSION: 7.8.1~ynh2
    YNH_APP_PACKAGING_FORMAT: 1.0
    YNH_APP_UPGRADE_TYPE: UPGRADE_PACKAGE
    YNH_ARCH: amd64
    YNH_CWD: /var/cache/yunohost/app_tmp_work_dirs/app_xx3dq9m4/scripts
    YNH_DEBIAN_VERSION: bullseye
    YNH_INTERFACE: cli
    YNH_STDRETURN: /tmp/tmpj_knvdjf/stdreturn
  error: None
  interface: cli
  operation: app_upgrade
  parent: None
  related_to: 
    - app
    - mattermost
  started_at: 2023-05-01 14:55:27
  success: True
  yunohost_version: 11.1.18
metadata_path: /var/log/yunohost/categories/operation/20230501-135527-app_upgrade-mattermost.yml
name: 20230501-135527-app_upgrade-mattermost

kemenaran · May 1, 2023, 2:26pm

Yay, it seems it worked! I’ll merge the bugfix into the mainline branch then. Thanks for the report and testing.

ptrx · May 8, 2023, 12:24pm

I was about to try the testing branch to upgrade but I decided to re-try the main branch first and… it worked!
Thanks a lot

TheNomad11 · July 7, 2023, 8:36pm

Finally I made the step and upgraded from 7.2 to 7.8.1 with this complex database migration, I was a bit scared but it worked!! Thanks so much for your work!