Pleroma Known Timeline wont load 500 Error

[Twizzay]

Hi Everyone, I have been using YunoHost successfully for a few months now. I love the OS. It has made it possible for me to keep my data and host my own software and applications right from my home. I would love to be able to contribute more to this project some day.

My YunoHost server

Hardware: Raspberry Pi 4B at home
YunoHost version: 4.3.6.2 (stable).
I have access to my server : Through SSH | through the webadmin | direct access via keyboard / screen
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : No, everything I have used has been vanilla YunoHost Applications

Description of my issue

I tried to install a few new FE from the Admin FE recently. After reboot, I received nginx 502 errors.

I rolled back to a previous local backup of my pleroma installation and now the Known timeline is not loading and the home timeline is very slow and only loads occasionally. I am still visible to other instances. Can still post etc. but nothing from anyone else gets through the known timeline. Frequently crashes with 502 error.

It should be noted that my current backup method is only a local backup of the pleroma installation. Not a full disk backup. I noticed after restoring from the backup that the FE’s I had installed were still there. So I manually deleted /var/lib/pleroma/static/frontends.

Everything worked great before I made these changes. I will shame myself and find better ways to backup my instance.

on some advice from a friend, I modified my postgresql.conf to recommendations from PGtune and opened up the timelines to be viewable from unsigned users.

After doing that, I noticed something unusual. That is, my timelines load perfectly fine for unsigned users. Its just me that is receiving the 500 internal server error for the Known and Home timelines.

Would love to provide as much information as needed. I am Very motivated to solve this problem. Very little experience with pleroma. TIA

Also

Using Masto FE this error does not occur however it does behave unusually slow.

Here my instance for reference.

Here are some logs

[ljf]

It seems you have an SQL timeout error on your postgresql database.

How many ram and CPU have you (which raspberry pi) ?
Which kind of storage do you use (SD card, SSD, HDD)?
Is the pleroma instance slow ?

We need the iowait indictor, run

apt install sysstat
iostat

Could you check network is working correctly inside your yunohost (what about the yunohost diagnosis ?).

ping wikipedia.org

[Twizzay]

I am running off of an Rpi4b with an SD Card.

Here is the configuration that PGtune recommended:

# DB Version: 11
# OS Type: linux
# DB Type: web
# Total Memory (RAM): 4 GB
# CPUs num: 4
# Connections num: 20
# Data Storage: ssd

max_connections = 20
shared_buffers = 1GB
effective_cache_size = 3GB
maintenance_work_mem = 256MB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 26214kB
min_wal_size = 1GB
max_wal_size = 4GB
max_worker_processes = 4
max_parallel_workers_per_gather = 2
max_parallel_workers = 4
max_parallel_maintenance_workers = 2

iostat std out:

Linux 5.10.103-v7l+ (gandalf.mylegendary.quest)         03/29/2022      _armv7l_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          37.94    0.00    8.55    1.46    0.00   52.05

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0          20.09       172.59       357.88   22155699   45941725

ping worked.

Diagnosis Logs They’re not very pretty but nothing has been out of the ordinary since before this issue started.

[ljf]

So your iowait is ok.

Maybe you could test to do some request inside your database to see if request are long or not.
If you find a way to get the SQL request that timeout (for example by activating temporarily some debugging feature in pleroma ?) it could be very useful.

Currently i don’t know how to fix your issue

[Twizzay]

I have multiple local backups of this application, but even the oldest backups seem to elicit the same problems.

However, a fresh installation of Pleroma doesn’t have any problems.

Any Idea why that might be?

It leads me to believe that something must be wrong with how YunoHost is restoring from local Backups…

I would really love to be able to keep my account, statuses, MRF settings, and followers if possible.

[IljaB]

It leads me to believe that something must be wrong with how YunoHost is restoring from local Backups…

I looked if i could see differences in how Pleroma recommends restoring and how ynh does it, and there are differences.

The Pleroma docs recommend to use pg_restore with the -1 flag while restoring[1]. Ynh doesn’t seem to do that. It just executes the file as is with the psql command and without the -1 flag[2].

With man pg_restore I see the following (The psql command has a similar option)

   -1
  --single-transaction
      Execute the restore as a single transaction (that is, wrap the emitted commands in BEGIN/COMMIT). This ensures that either all the commands complete successfully, or no changes are applied. This option
      implies --exit-on-error.

Whether something really went wrong during restoration, is another question of course. But from what I understand, it’s indeed possible that something went wrong during restoration, but ynh ignores it and just continues.

[1] https://docs-develop.pleroma.social/backend/administration/backup/

[2] To restore, it uses the helper ynh_psql_execute_file_as_root (see https://github.com/YunoHost-Apps/pleroma_ynh/blob/master/scripts/restore#L121)

I see that uses the helper ynh_psql_connect_as (https://github.com/YunoHost/yunohost/blob/dev/helpers/postgresql#L59)

Which in turn uses the psql command (https://github.com/YunoHost/yunohost/blob/dev/helpers/postgresql#L8)