Failed to upgrade Gitlab

apprentiLAB_01 · October 19, 2021, 4:06pm

Hi everyone !

I tried to upgrade Gitlab because I noticed an issue which was solved in the following version.

My YunoHost server

Hardware: VM hosted by our university
YunoHost version: 4.2.8.3
I have access to my server : Through SSH | through the webadmin
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : I don’t know

Description of my issue

I have tried upgrading gitlab through the webadmin, it failed and uninstalled the app. I still have a pre-upgrade auto backup. I think it was v 13.2.3.

I have done this a few weeks ago (Oct. 4th) and cannot find the log file through the webadmin since there is no pagination. I tried finding it through ssh but I don’t know if it is in /var/log/ or elsewhere. I found one for that same day that is called user.log.1.

Should I just reinstall it and try again or do you have better advice ?

Thanks !!!

Aleks · October 19, 2021, 4:24pm

From the command line you should be able to display more logs and find the one corresponding to the failed gitlab upgrade using for example

sudo yunohost log list --limit 200 | grep gitlab -C3

then share it with:

sudo yunohost log display <the_log_id> --share

apprentiLAB_01 · October 20, 2021, 8:50am

Thanks, I found it !
Only when I try to share it, the command doesn’t return anything and gets stuck indefinitely.
I tried using log show and log share instead of display since I got warnings that it was going to be deprecated, but the same thing happens…

apprentiLAB_01 · October 20, 2021, 9:22am

So I tried log show without --share, the yunopaste server isn’t responding right now, here is my log :

description: Upgrade the 'gitlab' app
log_path: /var/log/yunohost/categories/operation/20211004-122347-app_upgrade-gitlab.log
logs:
  - 2021-10-04 14:42:15,802: DEBUG - 17154 + set +o xtrace
  - 2021-10-04 14:42:15,813: DEBUG - 17165 + local 'forbidden_path=     /var/www     /home/yunohost.app'
  - 2021-10-04 14:42:15,813: DEBUG - 17165 + '[' 1 -ge 2 ']'
  - 2021-10-04 14:42:15,813: DEBUG - 17165 + [[ -z /var/log/gitlab ]]
  - 2021-10-04 14:42:15,814: DEBUG - 17166 + [[      /var/www     /home/yunohost.app =~ /var/log/gitlab ]]
  - 2021-10-04 14:42:15,814: DEBUG - 17166 + [[ /var/log/gitlab =~ ^/[[:alnum:]]+$ ]]
  - 2021-10-04 14:42:15,814: DEBUG - 17166 + [[ b = \/ ]]
  - 2021-10-04 14:42:15,814: DEBUG - 17166 + '[' -e /var/log/gitlab ']'
  - 2021-10-04 14:42:15,814: DEBUG - 17166 + ynh_print_info '--message='\''/var/log/gitlab'\'' wasn'\''t deleted because it doesn'\''t exist.'
  - 2021-10-04 14:42:15,815: DEBUG - 17167 + local legacy_args=m
  - 2021-10-04 14:42:15,815: DEBUG - 17167 + args_array=([m]=message=)
  - 2021-10-04 14:42:15,815: DEBUG - 17169 + local -A args_array
  - 2021-10-04 14:42:15,816: DEBUG - 17170 + local message
  - 2021-10-04 14:42:15,817: DEBUG - 17170 + ynh_handle_getopts_args '--message='\''/var/log/gitlab'\'' wasn'\''t deleted because it doesn'\''t exist.'
  - 2021-10-04 14:42:15,817: DEBUG - 17171 + set +o xtrace
  - 2021-10-04 14:42:15,824: DEBUG - 17177 + echo ''\''/var/log/gitlab'\'' wasn'\''t deleted because it doesn'\''t exist.'
  - 2021-10-04 14:42:15,824: DEBUG - 17178 + ynh_del_swap
  - 2021-10-04 14:42:15,825: DEBUG - 17178 + '[' -e /swap_gitlab ']'
  - 2021-10-04 14:42:15,825: DEBUG - 17179 '/var/log/gitlab' wasn't deleted because it doesn't exist.
  - 2021-10-04 14:42:15,825: DEBUG - 17179 + ynh_script_progression '--message=Removal of gitlab completed' --last
  - 2021-10-04 14:42:15,826: DEBUG - 17179 + set +o xtrace
  - 2021-10-04 14:42:15,901: DEBUG - 17254 + set +o xtrace
  - 2021-10-04 14:42:15,919: DEBUG - 17272 + echo '! Helper used in legacy mode !'
  - 2021-10-04 14:42:15,920: DEBUG - 17273 + set +x
  - 2021-10-04 14:42:15,921: DEBUG - 17275 + echo '[####################] > Removal of gitlab completed'
  - 2021-10-04 14:42:15,921: DEBUG - 17275 [####################] > Removal of gitlab completed
  - 2021-10-04 14:42:15,922: DEBUG - 17275 + set -o xtrace
  - 2021-10-04 14:42:16,923: DEBUG - 18277 Pour voir le journal de cette opération 'Retirer 'gitlab' après un échec de restauration depuis une archive de sauvegarde', utilisez la commande 'yunohost log show 20211004-124211-remove_on_failed_restore-gitlab20211004-124211-remove_on_failed_restore-gitlab'
  - 2021-10-04 14:42:17,666: DEBUG - 19020 The permission database has been resynchronized
  - 2021-10-04 14:42:17,964: DEBUG - 19318 La configuration de SSOwat a été regénérée
  - 2021-10-04 14:42:17,977: DEBUG - 19330 Permission 'gitlab.main' supprimée
  - 2021-10-04 14:42:17,977: DEBUG - 19331 Pour voir le journal de cette opération 'Supprimer permission 'gitlab'', utilisez la commande 'yunohost log show 20211004-124217-permission_delete-gitlab20211004-124217-permission_delete-gitlab'
  - 2021-10-04 14:42:17,984: WARNING - 19338 L’opération 'Restaurer 'gitlab' depuis une sauvegarde' a échoué ! Pour obtenir de l’aide, merci de partager le journal de l’opération en utilisant la commande 'yunohost log share 20211004-124158-backup_restore_app-gitlab'
  - 2021-10-04 14:42:18,223: DEBUG - 19577 The permission database has been resynchronized
  - 2021-10-04 14:42:18,531: DEBUG - 19885 La configuration de SSOwat a été regénérée
  - 2021-10-04 14:42:18,545: DEBUG - 19899 action [19755.1] executed in 19.713s
  - 2021-10-04 14:42:18,546: WARNING - 19900 Rien n’a été restauré
  - 2021-10-04 14:42:18,609: DEBUG - + ynh_die '--message=The app was restored to the way it was before the failed upgrade.'
  - 2021-10-04 14:42:18,609: DEBUG - + local legacy_args=mc
  - 2021-10-04 14:42:18,609: DEBUG - + args_array=([m]=message= [c]=ret_code=)
  - 2021-10-04 14:42:18,609: DEBUG - + local -A args_array
  - 2021-10-04 14:42:18,610: DEBUG - + local message
  - 2021-10-04 14:42:18,610: DEBUG - + local ret_code
  - 2021-10-04 14:42:18,610: DEBUG - + ynh_handle_getopts_args '--message=The app was restored to the way it was before the failed upgrade.'
  - 2021-10-04 14:42:18,610: DEBUG - + set +o xtrace
  - 2021-10-04 14:42:18,636: DEBUG - + ret_code=1
  - 2021-10-04 14:42:18,636: DEBUG - + echo 'The app was restored to the way it was before the failed upgrade.'
  - 2021-10-04 14:42:18,637: WARNING - The app was restored to the way it was before the failed upgrade.
  - 2021-10-04 14:42:18,637: DEBUG - + exit 1
  - 2021-10-04 14:42:19,639: ERROR - Impossible de mettre à jour gitlab : Une erreur s’est produite durant l’exécution du script de mise à niveau de l’application
metadata:
  ended_at: 2021-10-04 14:42:19
  env:
    BASH_XTRACEFD: 7
    NO_BACKUP_UPGRADE: 0
    YNH_APP_CURRENT_VERSION: 13.2.3~ynh1
    YNH_APP_ID: gitlab
    YNH_APP_INSTANCE_NAME: gitlab
    YNH_APP_INSTANCE_NUMBER: 1
    YNH_APP_MANIFEST_VERSION: 14.3.0~ynh1
    YNH_APP_UPGRADE_TYPE: UPGRADE_APP
    YNH_CWD: /var/cache/yunohost/app_tmp_work_dirs/app_zdapglp8/scripts
    YNH_INTERFACE: api
    YNH_STDRETURN: /tmp/tmp4hzn2nt4/stdreturn
  error: Une erreur s’est produite durant l’exécution du script de mise à niveau de l’application
  interface: api
  operation: app_upgrade
  parent: None
  related_to:
    - app
    - gitlab
  started_at: 2021-10-04 14:23:47
  success: False
  yunohost_version: 4.2.8.3
metadata_path: /var/log/yunohost/categories/operation/20211004-122347-app_upgrade-gitlab.yml
name: 20211004-122347-app_upgrade-gitlab

ljf · October 20, 2021, 10:21am

Il manque le début du log. Le service paste.yunohost.org est opérationnel, je propose de réessayer avec --share

apprentiLAB_01 · October 20, 2021, 11:02am

J’ai tout recopié pourtant

Je viens de réessayer, ça ne répond toujours pas.

ljf · October 20, 2021, 1:51pm

J’ai identifié le problème directement sur le serveur. Le log est trop long et du coup le paste n’accepte pas le log visiblement.

Du coup, la partie intéressante du log c’est ça: hastebin . Apparemment un problème dans une migration qui convertit des int en bigint…

ljf · October 20, 2021, 1:56pm

Visiblement ça échoue lors de l’utilisation de dpkg qui lui même lance gitlab --reconfigure…

github.com

YunoHost-Apps/gitlab_ynh/blob/testing/scripts/upgrade#L292

    
      
          
          
		ynh_add_config --template="$YNH_APP_BASEDIR/conf/$architecture.src.default" --destination="$YNH_APP_BASEDIR/conf/$architecture.src"
          
          
		tempdir="$(mktemp -d)"
          
          
		ynh_setup_source --dest_dir=$tempdir --source_id=$architecture
          
          
		if ! ynh_exec_warn_less dpkg -i $tempdir/$gitlab_filename ;
          		then # This command will fail in lxc env
          			package_check_action # defined in upgrade.d/upgrade.X.sh
          			ynh_exec_warn_less dpkg --configure gitlab-ce
          		fi
          
          
		ynh_exec_warn_less ynh_secure_remove --file="$tempdir"
          
          
		current_version=$(grep gitlab-ce /opt/gitlab/version-manifest.txt | cut -d' ' -f2)
          
          
		# Sometimes we need to update the gitlab.rb configuration file in order to migrate to the next version.
          		if [ -e "$YNH_APP_BASEDIR/conf/gitlab.$current_major_version.rb" ]; then
          			if [ -e "$YNH_APP_BASEDIR/scripts/upgrade.d/upgrade.$current_major_version.last.sh" ]; then
          				source $YNH_APP_BASEDIR/scripts/upgrade.d/upgrade.$current_major_version.last.sh

Vu qu’il n’y avait qu’un repo, je propose de réinstaller gitlab pour voir si il y a le même soucis.

apprentiLAB_01 · October 20, 2021, 2:23pm

Merci beaucoup !

Tu veux dire ne pas réinstaller l’app sauvegardée, mais repartir de zéro ?

ljf · October 20, 2021, 2:33pm

Oui car de toutes façon sur gitlab il n’y avait pas eu besoin de beaucoup de réglages.

apprentiLAB_01 · October 20, 2021, 2:47pm

Ok, c’est pas grave pour le repo ?

ljf · October 20, 2021, 4:30pm

Il y a une copie du repo dans le répertoire des thèmes.

apprentiLAB_01 · October 21, 2021, 9:45am

C’est bon merci beaucoup.
Est-ce que tu penses que ça risque de se reproduire ?

mobil750 · October 29, 2021, 3:50pm

Hi,
I have the same problem but I’m unable to follow the French conversation. Could someone translate the relevant part regarding the “Solution”?

For me it is essential to use the update/upgrade path otherwise I will loose all my projects and work in the current GitLab environment - as far as I understand it.

Thanks a lot in advance
mobil750

Aleks · October 29, 2021, 3:56pm

The “solution” / workaround was to reinstall gitlab from scratch which ain’t great as the mid/long-term solution …

The actual error sounds pretty technical :

2021-10-04 14:41:23,956: DEBUG - rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embe
dded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
2021-10-04 14:41:23,956: DEBUG - ---- Begin output of “bash” “/tmp/chef-script20211004-15600-wuzuvh” ----
2021-10-04 14:41:23,956: DEBUG - STDOUT: rake aborted!
2021-10-04 14:41:23,956: DEBUG - StandardError: An error has occurred, all later migrations canceled:
2021-10-04 14:41:23,956: DEBUG -
2021-10-04 14:41:23,956: DEBUG - Expected batched background migration for the given configuration to be marked as ‘finished’, but it is ‘active’: {:job_class_name=>“CopyColumnUsingBackgroundMigrationJ
ob”, :table_name=>“events”, :column_name=>“id”, :job_arguments=>[[“id”], [“id_convert_to_bigint”]]}

Somehow gitlab did not manage to apply the migrations I guess But I don’t know enough about the actual logic of the app to debug further

Kayou · October 29, 2021, 4:08pm

Hi!

If you have the same error, could you please try to manually upgrade gitlab with the branch wait-migrations

You can do this in cli with:

sudo yunohost app upgrade gitlab -u https://github.com/YunoHost-Apps/gitlab_ynh/tree/wait-migrations --debug

If you accept to do this, I’ll look forward to your feedback, but if you are not comfortable with the idea of running such a command, you can either wait for another solution or use the workaround given by @Aleks , which is to uninstall/reinstall Gitlab

mobil750 · October 30, 2021, 8:25am

Hi,

I tried your proposed manual upgrade but it failed as well.
There have been several sequential upgrades but the upgrade from 14.0.11 to 14.4.0 finalliy failed.
I have uploaded the log via “yunohost log share 20211030-073907-app_upgrade-gitlab”. Hope you can see what went wrong.

The upload is not finished yet. I don’t know how long it will take.

Aleks · October 30, 2021, 12:54pm

Uuuuh yeah but we can’t see the log if you don’t share the url that this generated

Aleks · October 30, 2021, 12:55pm

Aah … I think we noticed the other day that gitlab logs are absolutely huge and take a shitload of time to upload (assuming that it succeeds at some point …)

mobil750 · October 30, 2021, 2:06pm

You are right. I got this one

root@yunohost:~# yunohost log share 20211030-073907-app_upgrade-gitlab

Error: Something wrong happened while trying to paste data on paste.yunohost.org : 413, <html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>