Stuck in an upgrade/migration loop

What type of hardware are you using: Raspberry Pi 3, 4+
What YunoHost version are you running: 11.3.0.2
How are you able to access your server: The webadmin
SSH

Describe your issue

Hi!

I was about to upgrade an app when this error appeared:

$ sudo yunohost app upgrade --force bookstack                                                                   1 ↵
Info: Now upgrading bookstack…
Error: This app requires YunoHost >= 12.0.9 but current installed version is 11.3.0.2

Glad that I got the information, I ran the migration command to continue the process.

But then, I got this error:

$ sudo yunohost tools migrations run --accept-disclaimer
Info: Running migration 0027_migrate_to_bookworm…
Info: Fetching available upgrades for system packages…
Error: Migration 0027_migrate_to_bookworm did not complete, aborting. Error: Your system is not fully up-to-date. Please perform a regular upgrade before running the migration to Bookworm.
Info: The operation 'Run migrations' could not be completed. Please share the full log of this operation using the command 'yunohost log share 20250127-101219-tools_migrations_migrate_forward' to get help

It seems that both actions are blocked by the other, and I have no idea how to fix this.

This is not the only app that’s giving me the need for migration, btw.

Thanks!

Share relevant logs or error messages

ended_at: 2025-01-27 10:12:57.818465
error: ‘Migration 0027_migrate_to_bookworm did not complete, aborting. Error: Your
system is not fully up-to-date. Please perform a regular upgrade before running
the migration to Bookworm.’
interface: cli
operation: tools_migrations_migrate_forward
parent: null
started_at: 2025-01-27 10:12:19.206780
success: false
yunohost_version: 11.3.0.2

============

2025-01-27 11:12:19,323: INFO - Running migration 0027_migrate_to_bookworm…
2025-01-27 11:12:19,355: INFO - Fetching available upgrades for system packages…
2025-01-27 11:12:19,978: DEBUG - Hit:2 Index of /debian bullseye InRelease
2025-01-27 11:12:20,027: DEBUG - Hit:3 Index of /php/ bullseye InRelease
2025-01-27 11:12:20,124: DEBUG - Hit:1 Index of /debian/ bullseye InRelease
2025-01-27 11:12:20,211: DEBUG - Get:4 mirror.fra1.de.leaseweb.net | powered by Leaseweb bullseye InRelease [15.0 kB]
2025-01-27 11:12:22,210: DEBUG - Fetched 15.0 kB in 2s (6493 B/s)
2025-01-27 11:12:32,067: DEBUG - Reading package lists…
2025-01-27 11:12:54,747: DEBUG - Done
2025-01-27 11:12:54,764: DEBUG - Loading migration 0022_php73_to_php74_pools…
2025-01-27 11:12:54,765: DEBUG - Loading migration 0025_global_settings_to_configpanel…
2025-01-27 11:12:54,766: DEBUG - Loading migration 0024_rebuild_python_venv…
2025-01-27 11:12:54,767: DEBUG - Loading migration 0021_migrate_to_bullseye…
2025-01-27 11:12:54,769: DEBUG - Loading migration 0023_postgresql_11_to_13…
2025-01-27 11:12:54,769: DEBUG - Loading migration 0026_new_admins_group…
2025-01-27 11:12:54,770: DEBUG - Loading migration 0027_migrate_to_bookworm…
2025-01-27 11:12:57,808: ERROR - Migration 0027_migrate_to_bookworm did not complete, aborting. Error: Your system is not fully up-to-date. Please perform a regular upgrade before running the migration to Bookworm.
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/yunohost/tools.py”, line 786, in tools_migrations_run
migration.run()
File “/usr/lib/python3/dist-packages/yunohost/migrations/0027_migrate_to_bookworm.py”, line 98, in run
self.check_assertions()
File “/usr/lib/python3/dist-packages/yunohost/migrations/0027_migrate_to_bookworm.py”, line 376, in check_assertions
raise YunohostError(“migration_0027_system_not_fully_up_to_date”)
yunohost.utils.error.YunohostError: Your system is not fully up-to-date. Please perform a regular upgrade before running the migration to Bookworm.

Hi Von,

Welcome to the forums!

I think this refers to a regular apt upgrade && apt update . It makes sure all packages are at the (known) last available version for Debian 11 / Bullseye.

From there they can be upgraded to the versions provided by Debian 12 / Bookworm during the migration.

Please run

$ sudo apt update
$ sudo apt upgrade

and post the output if any errors occur.

1 Like

Hey! Thank you for your answer :slight_smile:

It seems like it. I ran the update + upgrade, and was greeted with:

$ sudo apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  gcc-8-base
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

After few searches, I read that it might not be recommended to call the --only-upgrade option.

Should I just wait then ?

You could try removing gcc-8-base.

If I suggest my Yunohost to do so, a small number of extra packages is also uninstalled:

$ sudo apt remove gcc-8-base
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libasan3 libcilkrts5 libubsan0
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  gcc-6 gcc-8-base libgcc-6-dev libgcc1 libmpx2
0 upgraded, 0 newly installed, 5 to remove and 0 not upgraded.
After this operation, 39.2 MB disk space will be freed.
Do you want to continue? [Y/n] n
Abort.
$

After that try apt update and apt upgrade once more to find if there are no more upgradeable packages, and restart the migration.

In case the migration still complains about being not fully up to date, about how many apps are installed on your Yunohost? How many of them have a new version, but state that they depend on YNH12 before they can upgrade? You may tray upgrading them one by one, to find out of any of them still can be upgraded on YNH11

The migration started!

But then it failed…

Here is the log: https://paste.yunohost.org/raw/uqanoxuwuv

And now, every yunohost command gets an exception:

Traceback (most recent call last):
  File "/usr/lib/python3.11/logging/config.py", line 389, in resolve
    found = getattr(found, frag)
            ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'moulinette.interfaces' has no attribute 'api'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/logging/config.py", line 391, in resolve
    self.importer(used)
  File "/usr/lib/python3/dist-packages/moulinette/interfaces/api.py", line 16, in <module>
    from bottle import request, response, Bottle, HTTPResponse, FileUpload
  File "/usr/lib/python3/dist-packages/bottle.py", line 44, in <module>
    from inspect import getargspec
ImportError: cannot import name 'getargspec' from 'inspect' (/usr/lib/python3.11/inspect.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.11/logging/config.py", line 562, in configure
    handler = self.configure_handler(handlers[name])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/config.py", line 724, in configure_handler
    klass = self.resolve(cname)
            ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/logging/config.py", line 396, in resolve
    raise v from e
ValueError: Cannot resolve 'moulinette.interfaces.api.APIQueueHandler': cannot import name 'getargspec' from 'inspect' (/usr/lib/python3.11/inspect.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/yunohost", line 77, in <module>
    yunohost.cli(
  File "/usr/lib/python3/dist-packages/yunohost/__init__.py", line 35, in cli
    init_logging(interface="cli", debug=debug, quiet=quiet)
  File "/usr/lib/python3/dist-packages/yunohost/__init__.py", line 168, in init_logging
    configure_logging(logging_configuration)
  File "/usr/lib/python3/dist-packages/moulinette/utils/log.py", line 67, in configure_logging
    dictConfig(logging_config)
  File "/usr/lib/python3.11/logging/config.py", line 812, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/lib/python3.11/logging/config.py", line 569, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'api'

Should I create a new topic? So sorry to bother…

Without that it’d be a silent forum… :rofl:

From the log:

2025-01-27 16:15:50,392: DEBUG - 793 packages upgraded, 179 newly installed, 89 to remove and 8 not upgraded.
2025-01-27 16:15:50,392: DEBUG - Need to get 576 MB/576 MB of archives. After unpacking 454 MB will be used.
2025-01-27 16:15:56,423: INFO - Downloading...
2025-01-27 16:23:32,881: WARNING - E: Failed to fetch http://mirror.de.leaseweb.net/raspbian/raspbian/pool/main/m/mailutils/libmailutils9_3.15-4_armhf.deb: Error reading from server - read (32: Broken pipe) [IP: 37.58.58.140 80]
2025-01-27 16:23:32,889: WARNING - E: Unable to fetch some packages; try '-o APT::Get::Fix-Missing=true' to continue with missing packages
2025-01-27 16:25:54,087: ERROR - Migration 0027_migrate_to_bookworm did not complete, aborting. Error: Failed to run command 'aptitude full-upgrade --show-why -o 

Checking that location,

it seems available.

With some luck, wget is installed on your Yunohost. Would you try

$ wget http://mirror.de.leaseweb.net/raspbian/raspbian/pool/main/m/mailutils/libmailutils9_3.15-4_armhf.deb

to confirm your Yunohost can reach the mirror?

It’s inconvenient that the yunohost-commands broke. You could try once again to kick off the migration command, but if it does not work, we can manually continue the upgrade from Bullseye to Bookworm with the command line that can be found in the log:

2025-01-27 16:15:14,992: DEBUG - Running: LC_ALL=C DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=none aptitude full-upgrade --show-why -o Dpkg::Options::='--force-confold' --quiet=2 -o=Dpkg::Use-Pty=0 -o "APT::Status-Fd=$YNH_STDINFO"

so,

$ sudo aptitude full-upgrade --show-why -o Dpkg::Options::='--force-confold' --quiet=2 -o=Dpkg::Use-Pty=0 -o "APT::Status-Fd=$YNH_STDINFO"

should return the whole bunch of complaints that you also see in the log (sooo many packages to install, unmet dependencies, suggested solution). Compare to see that it is about the similar list (should be exact, but as all planets are about to be aligned, you never know…)

Once that is done, run apt upgade && apgrade once again, then try running the migration once more via the yunohost-command.

If that still does not work, it may be time to get the pro’s involved. I’ve had the Yunohost commands broken and repaired again, but I’m not too familiar with the ways in which they can get broken.

So, to summarize:

  • migration was cancelled after the first package could not be downloaded
  • check that the mirror is actually available from your Yunohost
  • try running the yunohost-migration
    • if that one does not work, run the complicated aptitude command from the log
  • run apt update and upgrade
  • try restarting the yunohost-migration

Hey!

First of all, thank you for the detailed answer.

Here’s what happened:

I ran the command to manually continue the upgrade and it went through it without error. I accepted that the sshd and dovecot config would upgrade and override the current config well.

Then I proceeded to update/upgrade again, and this is what I was greeted with:

$ sudo apt update
sudo: you do not exist in the passwd database

That didn’t felt right, but I managed to run the commands by becoming root instead of admin. 8 packages were upgraded.

Then I ran the migration again, and got few LDAP errors. It continued and broke at the Regenerate system configurations 'nsswitch' operation.
Here are the related logs:

I wanted to check the users in the webadmin, and got the same error:
Error: "500"
Action: "GET" /yunohost/api/domains?locale=en
Error message:

error during LDAP search operation with: base='ou=domains,dc=yunohost,dc=org', filter='virtualdomain=*', attrs=['virtualdomain'] and exception {'desc': 'No such object'}

It seems kinda bad from my perspective :grimacing: , but I’m not an expert on LDAP

Edit: I thought it was a good idea to reboot, when in doubt… I can’t access by ssh, stating “Too many authentication failures”
But I can access the webadmin by local IP, enven though my login is granted by “Wrong password or username”

So, that started off well enough, but the fun didn’t last.

When faced with authentication failures, it’s always nice to have a root-session at hand. Rebooting may let you lose it. So, SSH does not allow you back in? If you still can get in via the webadmin, you may be able to install a web-terminal. It’s less comfortable than a regular terminal, but will let you repair the system in times of need.

In case the configuration for nsswitch is broken, I’ll paste mine below so you can compare, and seeing no item related to my specific server, try copying it to yours:

$ cat /etc/nsswitch.conf 
# /etc/nsswitch.conf

passwd:         files systemd ldap
group:          files systemd ldap
shadow:         files ldap
gshadow:        files

hosts:          files myhostname mdns4_minimal [NOTFOUND=return] dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis
sudoers:        files ldap

There was something else I’d thought to write, but between starting the post and coming back home, it slipped my mind.

If there’s any other user you can use to gain sudo, that would be helpful of course!

Yes, it was too late when I realized I had a bad idea rebooting…

Unfortunately, I can’t get in via the webadmin, the users are not found.

I think the only user left is root, so I’ll try to find the time to physically connect to the server today, for direct connection on the machine.

I’ll keep you posted. Thanks!

Ah, that’s nasty :frowning:

If that does not work, you could try mounting the SD card on another (Linux) computer, and check the contents of /etc/nsswitch.conf there.

You could then also change some files to (be able to) reset passwords (either by changing /boot/cmdline.txt or /etc/shadow and /etc/passwd, as explained on pi 3 - Resseting password using init=/bin/sh - Keyboard doesn't work - Raspberry Pi Stack Exchange and the referenced ssh - Change/reset password WITHOUT monitor/keyboard - Raspberry Pi Stack Exchange)

Hey! New day, new tries!

I did as suggested and tweaked the SD card directly. I rebooted the server and even though I couldn’t access it locally, I was able to get in from ssh by accessing it via another server.
To summarize: I have root access again.

About the nsswitch.conf file, I had the same content, without the last line (which I added):
sudoers: files ldap

But my yunohost users don’t appear in my shadow file. I tried to add the admin manually, with a useradd and setting a password, but it didn’t give me access to the weblogin. It might be a group issue.

We’re not there yet, but it sure looks like progress.

Without access to the webadmin, you can run yunohost-commands from the CLI over SSH.

Once you got a root session, try:

yunohost user list # the users that Yunohost knows about
yunohost user group info admins # that returns a list of members, among other info
yunohost user group add admins von # to add user von to group admins 

Adding --help after any of the commands will give a short help and list of options.

My bad, sorry. LDAP provides the users. Without knowing the specifics, the web logins are checked against LDAP as well. If the yunohost user list returns an empty list (which would be peculiar…), you can run yunohost user create to add a new one. IF that is the case, I’d create a non-existing temporary user, to get regular access again, and then continue the troubleshoot.

Good luck!

I’m sorry I forgot to mention that I already tried those yunohost commands. But no matter which one, I always end up getting one of those errors:

Error: error during LDAP search operation with: base='ou=users,dc=yunohost,dc=org', filter='(&(objectclass=person)(!(uid=root))(!(uid=nobody)))', attrs={'mail', 'cn', 'mailuserquota', 'uid'} and exception {'msgtype': 101, 'msgid': 3, 'result': 32, 'desc': 'No such object', 'ctrls': []}

It doesn’t matter which yunohost command, I always get this kind of error, with some values being different. That’s why I think there’s something really broken with ldap here.

You mean, any Yunohost command, not just the yunohost user commands?

How about trying to re-run (as root) the last command in the migration log,

aptitude full-upgrade --show-why -o Dpkg::Options::='--force-confold'

Big chance it will error out again, but it’s worth a try.

It may be that the partial upgrade let parts of Yunohost and moulinette in non-compatible versions, thus rendering them unable to either start or communicate.

Does the service yunohost-api run?

service yunohost-api status

Yes, any yunohost command unfortunately.
Here’s the result for listing the apps for example:

root@cloud:~# yunohost app list
Error: Failed to read info for bookstack : error during LDAP search operation with: base='ou=permission,dc=yunohost,dc=org', filter='(objectclass=permissionYnh)', attrs=['cn', 'groupPermission', 'inheritPermission', 'URL', 'additionalUrls', 'authHeader', 'label', 'showTile', 'isProtected'] and exception {'msgtype': 101, 'msgid': 3, 'result': 32, 'desc': 'No such object', 'ctrls': []}
Error: Failed to read info for calibreweb : error during LDAP search operation with: base='ou=permission,dc=yunohost,dc=org', filter='(objectclass=permissionYnh)', attrs=['cn', 'groupPermission', 'inheritPermission', 'URL', 'additionalUrls', 'authHeader', 'label', 'showTile', 'isProtected'] and exception {'msgtype': 101, 'msgid': 4, 'result': 32, 'desc': 'No such object', 'ctrls': []}
Error: Failed to read info for cyberchef : error during LDAP search operation with: base='ou=permission,dc=yunohost,dc=org', filter='(objectclass=permissionYnh)', attrs=['cn', 'groupPermission', 'inheritPermission', 'URL', 'additionalUrls', 'authHeader', 'label', 'showTile', 'isProtected'] and exception {'msgtype': 101, 'msgid': 5, 'result': 32, 'desc': 'No such object', 'ctrls': []}
Error: Failed to read info for my_webapp : error during LDAP search operation with: base='ou=permission,dc=yunohost,dc=org', filter='(objectclass=permissionYnh)', attrs=['cn', 'groupPermission', 'inheritPermission', 'URL', 'additionalUrls', 'authHeader', 'label', 'showTile', 'isProtected'] and exception {'msgtype': 101, 'msgid': 6, 'result': 32, 'desc': 'No such object', 'ctrls': []}
Error: Failed to read info for my_webapp__2 : error during LDAP search operation with: base='ou=permission,dc=yunohost,dc=org', filter='(objectclass=permissionYnh)', attrs=['cn', 'groupPermission', 'inheritPermission', 'URL', 'additionalUrls', 'authHeader', 'label', 'showTile', 'isProtected'] and exception {'msgtype': 101, 'msgid': 7, 'result': 32, 'desc': 'No such object', 'ctrls': []}
Error: Failed to read info for navidrome : error during LDAP search operation with: base='ou=permission,dc=yunohost,dc=org', filter='(objectclass=permissionYnh)', attrs=['cn', 'groupPermission', 'inheritPermission', 'URL', 'additionalUrls', 'authHeader', 'label', 'showTile', 'isProtected'] and exception {'msgtype': 101, 'msgid': 8, 'result': 32, 'desc': 'No such object', 'ctrls': []}
Error: Failed to read info for privatebin : error during LDAP search operation with: base='ou=permission,dc=yunohost,dc=org', filter='(objectclass=permissionYnh)', attrs=['cn', 'groupPermission', 'inheritPermission', 'URL', 'additionalUrls', 'authHeader', 'label', 'showTile', 'isProtected'] and exception {'msgtype': 101, 'msgid': 9, 'result': 32, 'desc': 'No such object', 'ctrls': []}
Error: Failed to read info for redirect : error during LDAP search operation with: base='ou=permission,dc=yunohost,dc=org', filter='(objectclass=permissionYnh)', attrs=['cn', 'groupPermission', 'inheritPermission', 'URL', 'additionalUrls', 'authHeader', 'label', 'showTile', 'isProtected'] and exception {'msgtype': 101, 'msgid': 10, 'result': 32, 'desc': 'No such object', 'ctrls': []}
Error: Failed to read info for tinyfilemanager : error during LDAP search operation with: base='ou=permission,dc=yunohost,dc=org', filter='(objectclass=permissionYnh)', attrs=['cn', 'groupPermission', 'inheritPermission', 'URL', 'additionalUrls', 'authHeader', 'label', 'showTile', 'isProtected'] and exception {'msgtype': 101, 'msgid': 11, 'result': 32, 'desc': 'No such object', 'ctrls': []}
Error: Failed to read info for writefreely : error during LDAP search operation with: base='ou=permission,dc=yunohost,dc=org', filter='(objectclass=permissionYnh)', attrs=['cn', 'groupPermission', 'inheritPermission', 'URL', 'additionalUrls', 'authHeader', 'label', 'showTile', 'isProtected'] and exception {'msgtype': 101, 'msgid': 12, 'result': 32, 'desc': 'No such object', 'ctrls': []}
apps:
# nothing after

The result of the last command was the same as last time. Here’s the log: https://paste.yunohost.org/raw/zosugijoyo
And the log from the nsswitch error: https://paste.yunohost.org/raw/tohoyovozo

yunohost-api is definitely running:

root@cloud:~# service yunohost-api status
● yunohost-api.service - YunoHost API Server
     Loaded: loaded (/etc/systemd/system/yunohost-api.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-01-29 13:17:08 CET; 1 day 1h ago
   Main PID: 605 (yunohost-api)
      Tasks: 1 (limit: 2057)
        CPU: 2.335s
     CGroup: /system.slice/yunohost-api.service
             └─605 /usr/bin/python3 /usr/bin/yunohost-api

I really have no clue how to fix this ldap error. I tried adding some ou= values into the /etc/ldap/slapd.ldif file, but it didn’t do anything.

I also ran a apt full-upgrade just in case, which upgraded raspi-config rpi-eeprom

:frowning:

There have been some threads involving LDAP, of which one that found the culprit in /etc/hosts goes quite in-depth. It also mentions another thread in one of the first posts.

Could you have a look whether their troubleshooting shines any light on your situation? First of all, I’d check the hosts-file.

Ok! I managed to do stuff!

First, there was no problem on my hosts file, so I looked elsewhere. And based on what I found, here was my process:

# regen slapd
/usr/share/yunohost/hooks/conf_regen/06-slapd init
# check if a slapd process is already running
netstat -tulpn | grep 389
# kill the existing process
pkill slapd
# and restart slapd
systemctl restart slapd

Yunohost commands were working again!
But I couldn’t create a user due to domain error

# add main domain + create cert
yunohost domain add <DOMAINNAME>
# create new admin user
yunohost user create <USERNAME>
yunohost user group add admins <USERNAME>

Now I have access to the webadmin!

But even though I can add domains, none of my apps appear in the webadmin view, whereas I can see them via cli. Also I can’t access them via browser, but that should be another issue I guess.

I still can’t connect using a normal user I created, but at least stuff are coming back.

I’m just wondering about the migration. Should I try to run it again?

Anyway, thanks a lot for getting me here!

Edit: Yes, something is odd:

root@cloud:~# yunohost app setting app privatebin
Error: Could not find app in the list of installed apps: 
 * bookstack
 * calibreweb
 * cyberchef
 * my_webapp
 * my_webapp__2
 * navidrome
 * privatebin
 * redirect
 * tinyfilemanager
 * writefreely
1 Like

Ah, great, well done :slight_smile:

That’s what I would do, it is built safe for re-use.

I have great news: The migration went through all of it without errors. I’m the proud owner of an up to date Yunohost server thanks to you!

I just had to create new users, the former ones were obliterated from reality (but I got to keep their /home/).

I also had to recreate all my domains, and I have few apps that prevent me from login because the users are not existing anymore. So I might have to reinstall stuff from scratch and reconfigure.

The major issue I’m encountering is the 504 error when trying to log in to the weblogin. It’s preventing me to access some of the admin only app, and it’s a problem. But at least I have my yunohost running with the main apps.

Edit: My 504 error is exactly the same case depicted here: Erreur 504 via yunohost/sso portalapi (debian Bookworm) - #38 by valen
With all the commands returning the same. Such a shame there is no solution provided. Setting NGINX to “modern” didn’t solve it.

1 Like

I increased the timeout, following this post: After installation, I can't login as user

It worked! Even though it doesn’t seem long term fix :sweat_smile: