Update Mastodon fails with error

My YunoHost server

Hardware: RPI 4 at home
YunoHost version: 11.1
I have access to my server : SSH/Webadmin
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no
If yes, please explain:

Description of my issue

After upgrading to the latest Yunohost version and upgrading all packages trough the web-ui I saw that Mastodon required an update, i clicked it , it took about 2 hours during wich both Mastodon and the web-ui were unresponsive

After that, it failed in the background and mastodon + web-ui came back up , I tried to run the update trough the CLI to understand a bit more about the issue and here’s what it says :

Any idea why it might fail to upgrade ?

Thanks !

Could you go in webadmin > tools > logs and find the log of this upgrade operation ?
Click on the green button “Share with yunopaste” and copy the link here.

Oh there we go, terminal ended with this :

Full log : https://paste.yunohost.org/raw/ukilofajeb

Also, and obviously as seen in the screenshot, my mastodon service has been wiped from server.

All I did was try to upgrade, first trough GUI, then CLI

La mise à jour a échouée à cause de:

LoadError: Could not open library '/var/www/mastodon/live/vendor/bundle/ruby/3.0.0/gems/blurhash-0.1.6/lib/../ext/blurhash/encode.so': /var/www/mastodon/live/vendor/bundle/ruby/3.0.0/gems/blurhash-0.1.6/lib/../ext/blurhash/encode.so: cannot open shared object file: No such file or directory

Puis la restauration a échouée parce que

/var/cache/yunohost/app_tmp_work_dirs/app_fn_ktt9s/restore: line 123: ld_preload: unbound variable

Pour en savoir plus sur la raison de l’échec de restauration, il serait utile de voir le contenu du fichier _common.sh qui définit cette variable ld_preload

tar -axf /home/yunohost.backup/archives/mastodon-pre-upgrade1.tar apps/mastodon/settings/scripts/_common.sh -O

On pourra alors peut être réparer l’archive pour restaurer mastodon.

Et voila :slight_smile:

#!/bin/bash

#=================================================

COMMON VARIABLES

#=================================================

dependencies used by the app (must be on a single line)

pkg_dependencies=“imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev file git-core g++ libprotobuf-dev protobuf-compiler pkg-config gcc autoconf bison build-essential libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm3|libgdbm6 libgdbm-dev redis-tools redis-server postgresql postgresql-contrib libidn11-dev libicu-dev libjemalloc-dev curl apt-transport-https”
build_pkg_dependencies=“”

memory_needed=“2560”
ruby_version=3.0.3
nodejs_version=16

Workaround for Mastodon on Bullseye

See "bundle exec rails" command fail on Debian 11 Bullseye · Issue #15751 · mastodon/mastodon · GitHub

if [ “$(lsb_release --codename --short)” = “bullseye” ]; then
case $YNH_ARCH in
amd64)
ld_preload=“LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so”
;;
arm64)
ld_preload=“LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so”
;;
armhf)
ld_preload=“LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libjemalloc.so”
;;
armel)
ld_preload=“LD_PRELOAD=/usr/lib/arm-linux-gnueabi/libjemalloc.so”
;;
i386)
ld_preload=“LD_PRELOAD=/usr/lib/i386-linux-gnu/libjemalloc.so”
;;
esac
else
ld_preload=“”
fi

#=================================================

PERSONAL HELPERS

#=================================================

#=================================================

EXPERIMENTAL HELPERS

#=================================================

#=================================================

FUTURE OFFICIAL HELPERS

#=================================================

So for me it’s a bit strange… I don’t understand why ld_preload is unbound in the restore script !

Have you tried to restore one more time mastodon-pre-upgrade1 archive ?

Maybe you can try to force the ld_preload value in the restore script ?

The idea is to edit the archive restore script to put before the line 123:

ld_preload=“LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so”

If you know vim you can edit the .tar archive with it.

No idea and yeah I did , same error :confused:

What scares me is that I did nothing to trigger this, I simply clicked “update” in the webadmin so I must not be the only user with that issue

Sure I’ll try that but how do I edit a texte file inside a tar archive ?

Update : After trying different things, reinstalling mastodon, still no data anywhere and the original backup has disappeared.

I’m gonna try and pull from my remote Borg backup to see if I can salvage anything.

But I’m getting a bit worried about this, it’s fine if I never recover Mastodon and my data but it shouldn’t be normal for Yunohost to “blow up” on it’s own. I’ll let you know how I recover if I manage to

Updates :

→ The old “mastodon-pre-upgrade1.tar” was deleted

→ I tried restoring from a “mastodon-pre-upgrade2.tar” that was created in it’s place. I have ruby errors on Mastodon-Web when running it (unavailable right now to copy/paste since I’m doing step 3) and a 502 error when loading the web interface

→ After searching trough the entire forum’s search result for Mastodon I’m now trying to upgrade this “failed” restore to the latest Mastodon version to see if it restores access

→ If that doesn’t work can I pull an old backup from my borg repo and restore everything ?

→ Is there a way to surgically pull my followers/media/toots from an old borg repo backup and nuke my server to start all over again clean ? I know this isn’t a popular thing to do but the forums have no answers , I’m apparently the only one to have had such a catastrophic failure with Mastodon and I’m at the limits of my competence on the issue.

→ Does Borg even keep old version or did I loose everything ?

Bottom line for everyone combing trough the forum : Please backup your Mastodon profile/followers before doing a routine update, my server blew up on it’s own

Allright I’m SO CLOSE TO GETTING IT lol

So, I finally learned about how Borg Backup work, logged in trough my MacOS desktop and pulled an archive of Mastodon from a day before the explosion.

I downloaded it straight from the borg repo and created a .tar archive of it

However when I use the Yunohost CLI to import this backup I get the following error : Error: Could not load info for archive ‘/home/yunohost.backup/archives/mastodo040223.tar’… The info.json file cannot be retrieved (or is not a valid json).

Absolutely no idea what format Json this thing expect, there’s close to 0 information available both in docs and in the forums

Update : Managed to pull a BorgBackup mastodon backup from before the accident but it still fails with some Ruby error when restoring mastodon exactly like before. Probably because my Yunohost version is now higher than it was before the accident

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.