YunoHost troubleshooting guide

,

Hi,
I’ve noticed that many yunohost users have trouble to find the real cause of their problems so I’ve decided to write a guide to help troubleshooting yunohost.

You can consider this as some kind of administrator guide for troubleshooting or what to check when there is something wrong with your yunohost server. It’s not a step by step guide to make your yunohost work but more of a list of things to check to help diagnose the issue you may have. This can also be useful for debug when packaging a new application or trying to understand yunohost architecture.

Consider this as unofficial since I’m not part of yunohost team and in a general way don’t trust everything you read on internet without verification wink :wink: Feel free to improve it or correct my bad English in the comments, I will edit the post accordingly.

General notes

Don’t break yunohost

Best way to troubleshoot is to not have to troubleshoot because you have tested everything before deploying it on your server.
That means everytime you want to try a new thing (unofficial app, new specific config, package an app, …) try it on a development server not on your production. For this, use whatever way you think appropriate, you can use yunohost on a Digitalocean droplet for 1 cent/hour, locally with docker or a VM for example.
Also: https://wiki.debian.org/DontBreakDebian

Use unofficial app with caution

While it’s tempting to install every app on https://yunohost.org/#/apps_in_progress_en please don’t. Even if the app is marked as ready. Before testing an app you should read at least part of the package source code. install, remove and upgrade script should be present.
Keep in mind that when you install an app, you execute code on your server with administrator privileges, code that a stranger have put on internet.
From my experience some packaging are excellent, some can break some part of your yunohost with one install and some others are unmaintained. So before install check github issue, forum and yunohost support channel to see if other have problem with the app.

Check the official documentation

Answer to your question may be already in the doc: https://yunohost.org/#/sitemap_en

Check the command line help

Learn to use the yunohost command line like a pro: https://yunohost.org/#/moulinette_en

Upgrade

Problem often occurs after an upgrade. After a yunohost upgrade you may want to update your app according to documentation https://yunohost.org/#/app_update_en

Check if some processes are using old librairies

You’re probably familiar with:

sudo apt-get update && sudo apt-get dist-upgrade

Most of the time it’s enough. But in some cases it’s possible that some process are still using old versions of upgraded files (such as libraries), that can cause bug. In rare cases that can lead to security concern (ex: upgrade openssl because of a security bug, nginx will continue to use the version it has in memory). The utility checkrestart will help you to find and restart them.

sudo apt-get install debian-goodies
sudo checkrestart
Found 0 processes using old versions of upgraded files

If some process are running with old librairies versions checkrestart will tell you and propose you a way to restart them. In some cases checkrestart can’t find a way to restart them.

Easier way is to reboot if you can.

You can also use this script to automaticaly restart some services if needed. More info in French here.

Force an upgrade on a community app

/!\ Always check if there is a upgrade script and read it if you can /!\

sudo yunohost app upgrade
Warning: You must provide an URL to upgrade your custom app someapp
Error: No app to upgrade
sudo yunohost app upgrade -u https://github.com/user/someapp_ynh someapp

Services

Yunohost uses a bunch of software to accomplish its purpose. Most of theses software are registered as service in debian.
More info: https://yunohost.org/#/whatsyunohost_fr

Check services status

When something doesn’t work on your YunoHost, one of the first things to do is to check that all services used by yunohost are running.
Yunohost include a helper to see the status of all services used by yunohost:

 sudo yunohost service status

Sample result:

All services should be enabled and running. Except glances (optional) and tahoe-lafs. If some are not, try to restart them. Here is a short description of what they do:

  • amavis: anti-spam/virus/malwares, used when receiving email
  • avahi-daemon: system which facilitates service discovery on a local network via DNS
  • bind9: DNS server, you may or may not use it, I believe it’s no longer used by yunohost
  • dovecot: IMAP and POP3 email server, used to receive email
  • glances: optional, used by web administration to display server status
  • metronome: XMPP server, used by jappix and maybe some external client you have
  • mysql: database, used by some app
  • nginx: web server, used by all web app
  • php5-fpm: PHP service, used by all app written in PHP
  • postfix: SMTP email server, used to send email
  • postgrey: Greylisting Policy Server, if you use yunohost email, you should read this
  • slapd: LDAP server, used for authentification (sso and app)
  • ssh: Secure Shell, speaks for himself
  • tahoe-lafs: why is this thing here ? this should be disabled
  • yunohost-api: yunohost web administration

Others services installed by applications can also be present. In my example I have seafile-server wich serves Seafile app and uwsgi which is used to serve python app such as Searx.

Start or stop a service which is registered with yunohost:

sudo yunohost service start servicename
sudo yunohost service stop servicename

You can also use the generic debian command:

sudo service servicename start/stop/restart

After a launch attempt, always check that the service is running.

Note: Debian Jessie now uses Systemd instead of upstart, but, for now it’s fully compatible with Debian Wheezy way to handle services. Useful resource on systemd.

Logs

If a service doesn’t start you have to check the logs to see what’s wrong. There is no generic way for service to store their log, but most of the times they are in:

/var/log/

Here are the some useful for yunohost:
auth.log
Connection or attempt to connection to your server. That includes evry the time you connect with ssh, evry the time a cron job connects, and all the failed (hopefully) attempts to connect by potential intruders.

fail2ban.log
When someone tries to connect to your server and fails multiple times, fail2ban bans the ip to avoid bruteforce and (d)dos attacks. Here you can see if some ip have been banned.

mail.err, mail.info, mail.log, mail.warn
These are postfix (the mail server) logs, check theses if you have issues with email.

metronome/metronome.err, metronome/metronome.log
XMPP server logs.

mysql.err, mysql.log, mysql/error.log
Mysql database logs, these should be empty unless you have trouble with mysql.

php5-fpm.log
Genereic log for PHP apps.

yunohost.log
This is the log created by yunohost installation. If you have issue just after installing yunohost, check this one.

This list is not exhaustive. Also some app may put their log in /var/log too. Slapd logs are unfortunatly in /var/log/syslog

Memory

Sometime issues are caused because you don’t have enough free memory. To check it use the following command to check it:
free -m

5-10% of free memory is fine but it’s good to have margin (especially for upgrade). Since most of the time you can’t upgrade your physical ram, alternative it to use a swap file. Keep in mind that swap is memory but 100,000 times slower so you want to use it only when you have no other choice.

Create a swap file:
If free -m indicate that you have 0 total for swap line, you may want to add a swap file.

sudo install -o root -g root -m 0600 /dev/null /swapfile
dd if=/dev/zero of=/swapfile bs=1k count=512k
mkswap /swapfile
swapon /swapfile
echo "/swapfile       swap    swap    auto      0       0" | sudo tee -a /etc/fstab
sudo sysctl -w vm.swappiness=10
echo vm.swappiness = 10 | sudo tee -a /etc/sysctl.conf

Change 512 with the number of Mb of swap you want, 512Mb should be enough for yunohost. After that, check with free -m that you swap is activated.
Source with more explanation

Disk space

One other common issue on a server is lack of disk space. You can verify that your filesystem are not full with the command:

df -h

This will show you disk usage, if one of Filesystem is approching to 100% use you may have issue and want to take appropriate actions to liberate space or extend your filesystem.

Nginx

Nginx play a big part in yunohost since it serve all the web applications. Yunohost have a specific way to handle its configuration since there are multiple domain and multiple applications.

Configuration

General configuration structure

# Main nginx configuration, you don't want to touch this file
/etc/nginx/nginx.conf
# Directory where all yunohost, domain and app config are located
/etc/nginx/conf.d/
# Configuration of web administration
/etc/nginx/conf.d/yunohost_admin.conf
# Per domain configuration (one per domain)
/etc/nginx/conf.d/example.com.conf

Application configuration

For each application on a given domain there is a nginx conf file located in:

/etc/nginx/conf.d/example.com.d/appname.conf

Application configuration file usually follow this type of pattern

location YNH_WWW_PATH { # path to access the app in browser
  alias YNH_WWW_ALIAS ; # Path to source, usually /var/www/appname

  # Specific configuration for the application according to its programming language and deployment option
  ...
  ...

  # Include SSOWAT user panel in bottom right
  include conf.d/yunohost_panel.conf.inc;
}

Logs

Nginx log files are located in this directory:

/var/log/nginx/

Generic logs:
access.log
Generic access log, you will find here all the access to the yunohost administration and sometimes intrusion tentative.

error.log
Should be empty with a correct nginx config. If nginx doesn’t start, error are probably located in this log.

For each domain:
example.com-access.log
All access to the domain, including all apps.

example.com-error.log
All error regarding app installed on the domain.

Sometime application may have their log located here too.

SSOwat

SSowat is the software that connect the web server nginx to the LDAP server. His purpose is to authentificate users for the yunohost portal and all the web apps. More info: https://github.com/Kloadut/SSOwat

Configuration

You can view (don’t edit it, it will be often overwritten) your current ssowat config in the file:

/etc/ssowat/conf.json

Which is generated by:

sudo yunohost app ssowatconf

Protip: if you want to add a personalized rule for ssowat, do it in this file:

/etc/ssowat/conf.json.persistent

Logs

There is no specific log for SSOwat, logs are located in nginx logs files. If you see some line with "lua’ in it, it’s probably ssowat.

Yunohost

Configuration

Yunohost config is located in:

/etc/yunohost/

If you want to keep a custom service configuration use this file:

/etc/yunohost/yunohost.conf

For all the service you pass “yes”, yunohost will not upgrade the config of the specified service. Do this only if you know what you’re doing.

All apps configurations are located in:

/etc/yunohost/apps/appname/

For each app you will find in this directory:

  • manifest.json: manifest of the app
  • scripts/: directory with install, upgrade and remove script with which the application has been installed
  • settings.yml: config of the app, also accessible via sudo yunohost app setting appname settingname

Logs

As far as I know there is no logfile for application installation so when you install an app, keep the log. There is some log about the command line utilisation in:

/var/log/yunohost/

Applications

This part is more for packager or to understand the link between nginx and webapps. For packaging a new app you should read this first: https://yunohost.org/#/packaging_apps_en

When troubleshooting an application issue can occur at several levels. There is a wide variety of applications and their deployement to yunohost will differ depending the programation langage of the app. I will deal here with the most common case.
Applications configurations are not treated because it completely differs according to the application.

Oversimplified schema
your browser -> nginx <- (web server) <- runtime (php, python, node.js,…) <- app

App is interpretated by the runtime, runtime provide or not a webserver, if not a webserver can be added, webserver or runtime communicate with nginx, nginx serve webpage to your browser.

The purpose of this configuration is to have multiple application on one server but with only the https port (443) open to the world.

PHP app

Deployment option
PHP works natively with nginx.

Communication with nginx
The php interpreter communicate with nginx throught PHP-FPM wich is a FastCGI implementation.

Logs

/var/log/php5-fpm.log

Yunohost app example
Owncloud: https://github.com/Kloadut/owncloud_ynh

Python app

Deployment option
A python webapp should run with it’s own python interpreter with it’s own dependancies, for this, the tool virtualenv is used.
Usually, a light web server will be used to serve the app behind nginx. Uwsgi is a good example.

Communication with nginx
Nginx can talk too python server via 3 ways:

Logs
Specific to the app and/or the webserver used, for example uwsgi:

/var/log/uwsgi/

Ressources
good resource in french on python virtualenv: http://sametmax.com/les-environnement-virtuels-python-virtualenv-et-virtualenvwrapper/
Yunohost app example
Searx: https://github.com/abeudin/searx_ynh

Node.js app

Deployment option
A Node.js app have it’s own web server integrated in node runtime, usually Node will expose the app to a tcp port.

Communication with nginx
The http endpoint will be passed locally to nginx via proxy_pass.

Logs
This will be app specific.

Ressources

Yunohost app example
Etherpad-Lite (unofficial but good package) https://github.com/abeudin/etherpadlite_ynh

note: node runtime can use “a lot” of memory comparated to other app, make sure you have enough.

Other (go binary, java, …)

Webapp can take multiple form an can be deployed in many different manner. Go app have usually an integrated webserver, java can be deployed with Tomcat or equivalent,… There is no way I can be exhaustive here, but most of the time your deployement option will expose an http endpoint that you can pass to nginx via proxy_pass.

Note on Apache: never install apache or a package that have apache as dependancy, this will probably break yunohost.

Note on https: sometimes the webserver integrated with the app is capable to serve https on its own, it’s ok to use this when you have one application on one server and you don’t want to install nginx for example. But in yunohost case, it’s nginx who serve https and this should be only nginx (mostly to simplify configuration). So when passing the local url to nginx via proxy_pass, use http, nginx will https the thing to the open world later.

TODO

  • Email
  • Firewall
  • LDAP
  • DNS
  • Authentification in app (HTTP basic auth / LDAP / SSOwat)
  • You tell me :smiley:
8 Likes

Thanks tostaki for that documentation.

Thanks a lot for this guide, @tostaki. I have messed up my own Yunohost quite a lot and a few times, so I am reading very thoroughly your documentation…

Your English may be as bad as mine, but I spotted a few miss-steacks; I’ll list corrections down here, as you suggested.

Tout cela se veut bien entendu constructif, si d’aventure vous compreniez la langue de Molière.

without verification wink Feel free to improve it or correct my bad english in the comment, I will edit the post in consequences.

=>

without verification wink . Feel free to improve it or correct my bad English in the comments, I will edit the post accordingly.

That’s mean everytime

=>

That means everytime

Also: https://wiki.debian.org/DontBreakDebian3

:smile: Very good advices in this link! I especially liked this one:
https://wiki.debian.org/DontBreakDebian#Don.27t_make_a_FrankenDebian
I ran (often successfully!), a number of FrankenDebians in the past… À la Monsieur Jourdain.

with administrator priviledge, code that
=>
with administrator privileges, code that

that some process are still
=>
that some processes are still

nginx will continue to use the version it have in memory
=>
nginx will continue to use the version it has in memory

In some case checkrestart
=>
In some cases, checkrestart

More info in french here
=>
More info in French here

Yunohost use a bunch of software
=>
Yunohost uses a bunch of software

one of the first thing to do
=>
one of the first things to do

Yunohost include a helper
=>
Yunohost includes a helper

ssh: speak for himself
=>
ssh: Secure Shell, speaks for himself

wich serve Seafile app and uwsgi wich is used to serve
=>
wich serves Seafile app and uwsgi which is used to serve

Debian Jessie now use Systemd insted of upstart
=>
Debian Jessie now uses Systemd instead of upstart

If a service don’t start
=>
If a service doesn’t start

That include all the time you connect with ssh, all the time a cron job connect, and all the failed (hopefully) attempt to connect by potential intruders.
=>
That includes every time you connect with ssh, every time a cron job connects, and all the failed (hopefully) attempts to connect by potential intruders.

When someone try to connect to your server and fail multiple times, fail2ban ban the ip
=>
When someone tries to connect to your server and fails multiple times, fail2ban bans the ip

Sometime issue are
=>
Sometime issues are

Hope it helps. I must go, I’ll come back and edit this post.

Thanks @pchg, I have corrected the guide :wink:

Thanks to both of you! Do you think it will be a good idea to have this sort of guide in the official documentation?

Yep, I’ll add it. Thanks again.

Added to the documentation: https://yunohost.org/#/troubleshooting_guide_en.
@tostaki, If you want actively contribute to documentation, you are welcome. We could give you easily editing access.

Thank you all for this troubleshooting guide.

Thanks, this is awesome !

Thx a lot tostaky ! Great Shot :wink:

How can I restart yunohost manually? It isn’t loading by default on reboot for some reason

@ark
may be you speak about yunohost -api ?

service yunohost-api restart

Well on my side it seems to “work”, in that I get redirected to your nohost.me domain, most probably because there’s no app installed on the root of the domain (or no default app) so it wants me to connect to the sso, which is on your nohost.me domain.

But basically this :

$ curl https://mothnet.xyz/ -k
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<center><h1>302 Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

shows that the server answers correctly (modulo the -k to ignore the self-signed certificate)

Soooooo can you take a screenshot of that “Closed connection” error ? Is there no more detail on the exact error happening ?

On my side it works, after accepting the certificate exception, I end up on the yunohost portal login form …

hmm. that’s weird. it doesn’t work on my end and i know its not hair pinning because it doesn’t work on my phone either [mobile data]… i thought it was DNS but i guess not as it works for you… plus the domain name has been operational with my older server hosting my website for about a year now. obviously wouldn’t be propagation issues. which leads me to believe it might be my SSL cache? only thing that makes sense being my phone is on mobile data and not the local wifi… and both that and my laptop which is on local wifi don’t work. none of my devices do. so even if it still doesn’t work on my end, thank you so much for helping narrow it down. i’ll see if chrome on my android has a “delete ssl cache” button or something. i’ve never had this issue before and i’ve changed my server OS a few times. this is the first time i’ve had the issue which is super weird. i thought it’d try to load the new certs or something automatically if it sensed it changed. maybe it’ll fix when i get certbot working because then they’ll expire and all that… but i can’t do that if i can’t get to the login screen. [even though it works i can’t get to the front end login screen so YunoHost has the lets encrypt button greyed out saying “http was unable to connect to mothnet.xyz”] so, yeah. it MUST be the ssl cache on my local devices [that i login to my website every day with to do work and such… so makes sense that they all wouldn’t work, if that was the actual issue].

UPDATE: my friend brought their MacBook over and it works, confirming its not hairpinning or anything and that the issue does exist solely on my devices. i’ll set up the website and such on his computer and get let’s encrypt going and hopefully it’ll just work itself out… if not i guess resetting my browser should work but then i lose all my tabs i need for setting everything up. or at least i can look at them while i use my friends laptop then reset my browser after its all done. either way, you helped a lot and either way, the website seems to be up and running… just gave me a panic attack there for a moment. asked a few other friends if they could visit the website and they could as well. so completely 100% localized to my devices it seems. both a good thing and a bad thing. heh.

thanks for writing this guide Moul and hope you are keeping safe.
I noticed that this was done in 2015, and may be this guide needs updating…I found the guide while looking for a solution where yunohost went for a back-up and never returned. The admin login page just shows white. Website is working fine and all the services seem to be working fine.
I also see that this documentation link is now showing 404…i don’t know if thats by design