Reverse proxies and redirection loops

jellium · February 4, 2019, 7:12pm

Dear community and developers,

I have a fresh YunoHost 3.4.2.2 running inside an LXC container hosted on a fresh Debian 9.7 install (x64 VPS OVH).
I downloaded and installed YunoHost from and on the LXC container (using the curl |bash way) and performed the post-installation from my web browser. For this last step, the Host (gateway address 10.5.0.1) acts as a VPN server allowing to access directly to the LXC container (address 10.5.0.2), from the browser using the public IP of the Host.

Below, please find the diagnosis as obtained from the admin panel Diagnostic tool:

{
    "host": "Debian 9.7",
    "kernel": "4.9.0-8-amd64",
    "packages": {
        "yunohost": {
            "repo": "stable",
            "version": "3.4.2.2"
        },
        "yunohost-admin": {
            "repo": "stable",
            "version": "3.4.2"
        },
        "moulinette": {
            "repo": "stable",
            "version": "3.4.2"
        },
        "ssowat": {
            "repo": "stable",
            "version": "3.4.2"
        }
    },
    "backports": [],
    "system": {
        "disks": {
            "sda1": "Mounted on /, 39.4GiB (36.3GiB free)"
        },
        "memory": {
            "ram": "3.8GiB (3.2GiB free)",
            "swap": "0B (0B free)"
        }
    },
    "nginx": [
        "nginx: the configuration file /etc/nginx/nginx.conf syntax is ok",
        "nginx: configuration file /etc/nginx/nginx.conf test is successful"
    ],
    "services": {
        "glances": "running (enabled)",
        "nslcd": "running (enabled)",
        "metronome": "running (enabled)",
        "postfix": "exited (enabled)",
        "rspamd": "running (enabled)",
        "yunohost-firewall": "exited (enabled)",
        "nginx": "running (enabled)",
        "php7.0-fpm": "running (enabled)",
        "dnsmasq": "running (enabled)",
        "fail2ban": "running (enabled)",
        "yunohost-api": "running (enabled)",
        "mysql": "running (enabled)",
        "avahi-daemon": "running (enabled)",
        "dovecot": "running (enabled)",
        "redis-server": "running (enabled)",
        "slapd": "running (enabled)",
        "ssh": "running (enabled)"
    },
    "applications": {},
    "security": {
        "CVE-2017-5754": {
            "name": "meltdown",
            "vulnerable": false
        }
    }
}

I have two issues that I cannot manage to find a solution for.

First, in the admin panel interface, I obtain the following error very frequently, while the page stops loading (the error comes with a red ribbon on the top of the page):

The server closed the connection instead of answering it, has nginx been restarted by error? (Error code/message: 0 error)

From the logs (as far as I know, for instance in /var/log/nginx/* or /var/log/syslog or monitoring systemctl status nginx), I cannot see that nginx is actually restarted. Any idea? It can be related to the fact that the YunoHost instance is running inside an LXC container, would you advice any checkups or tests to see if everything is running OK ?

Secondly, I setup a domain to access to my YunoHost instance, with the following various steps:

I set up the domain name DNS A record to the public address of the VPS (LXC Host)
I created the proper LetsEncrypt certificate for this domain with Certbot
I installed and set up an nginx web server on the LXC Host with the following conf file for the domain to redirect to the YunoHost LXC container (reverse proxy):

server {
        listen 80;
        listen [::]:80 ipv6only=on;

        server_name domain.tld www.domain.tld;

        root /var/www/html;
        index index.nginx-debian.html;

        location / {
                proxy_set_header        Host $host;
                proxy_set_header        X-Real-IP $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header        X-Forwarded-Proto $scheme;

                proxy_pass      http://10.5.0.2:80/;
                proxy_redirect  http://10.5.0.2:80 http://domain.tld;
        }
}


server {
        listen 443;
        listen [::]:443 ipv6only=on;

        server_name domain.tld www.domain.tld;

        ssl_certificate           /etc/letsencrypt/live/domain.tld/cert.pem;
        ssl_certificate_key       /etc/letsencrypt/live/domain.tld/privkey.pem;

        ssl on;
        ssl_session_cache  builtin:1000  shared:SSL:10m;
        ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
        ssl_prefer_server_ciphers on;

        access_log            /var/log/nginx/domain.tld.log;

        location / {
                proxy_set_header        Host $host;
                proxy_set_header        X-Real-IP $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header        X-Forwarded-Proto $scheme;

                proxy_pass         http://10.5.0.2:80/;
                proxy_redirect      http://10.5.0.2:80 https://domain.tld;
        }
}

(please ask me for the actual – but obvious – domain name by private message if needed )

I then added the domain to the YunoHost configuration, from the admin panel, which in turn generated the proper LetsEncrypt certificate inside the YunoHost LXC container.
I also set up the proper firewall and iptables rules to allow connection to the LXC container from the domain.

Now, when I try to access to the YunoHost LXC container with http(s)://(www.)domain.tld (any of the possibilities implied by the brackets), I end up in a sort of redirect loop, which ends up with the following browser error:

too many HTTP redirects

Or something like this in Firefox:

La page n’est pas redirigée correctement

Which translates into the following nginx logs:

For the Host nginx /var/log/nginx/access.log file:

[04/Feb/2019:19:56:47 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:47 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:47 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:47 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:47 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:47 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:47 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:47 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:47 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:47 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:47 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:48 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:48 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:48 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:48 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:48 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:48 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:48 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:48 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:48 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
[04/Feb/2019:19:56:48 +0100] "GET /yunohost/admin HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"

For the YunoHost LXC container /var/log/nginx/access.log file:

10.5.0.1 - - [04/Feb/2019:18:56:47 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:47 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:47 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:47 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:47 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:47 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:47 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:47 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:47 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:47 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:47 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:48 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:48 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:48 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:48 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:48 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:48 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:48 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:48 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"
10.5.0.1 - - [04/Feb/2019:18:56:48 +0000] "GET /yunohost/admin HTTP/1.0" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0"

I suspect a conflit between the nginx servers and https redirections and certificates that could lead to a sort of 301 redirect ping pong between the Host and the LXC container. I am not sure at all. But, in fact it does not look consistent to have two LetsEncrypt certificates for the same domain, one on the Host and one on the YunoHost LXC container which manages domains itself, but before breaking everything up I wanted to call for some help. Any idea how to correct things?

Thanks for your help! (dites-moi si une version français de mon post peut être utile)

Aleks · February 4, 2019, 8:03pm

Edit: thank you for taking the time to explain the problem with context and details by the way

Hellow,

don’t know much about the first issue, but my guess is that it’s related to the second issue and fixing it would also fix the first issue…

So about certificates : so far from my own experiment, what I did was to copy the certificate from the guest to the host each time it changed (so every 3 months) so that there’s no weird inconsistencies…

Note that you should not reverse-proxy to http … That’s because basically yunohost redirects everything to https, which is probably creating the redirection loop.

Otherwise, I encourage you to think about if you really need this whole reverse proxy setup, instead of simply putting yunohost in front directly (otherwise that makes everything much more complicated and is kinda out of the scope of yunohost)

jellium · February 5, 2019, 11:19pm

Thanks for your reply.

OK! I have copied the certificate from YunoHost (guest lxc container) to the host. I am not sure whether I made it properly, but at least there are no nginx errors… (I copied crt.pem, ca.pem and key.pem [guest, yunohost-letsencrypt] onto cert.pem, chain.pem and privkey.com [host, letsencrypt])
By itself, this doesn’t solve the redirect loop problem (but again, I have suspicions that I have not copied the certificates from YunoHost (guest) to Debian (host) properly).

EDIT: In fact, there might be a glimpse of improvement as reaching domain.tld leads to /yunohost/sso… with a warning (from Firefox) saying that the certificate might not be good, warning that is passed by “accepting the risk”, and only then the redirect loop occurs. Not sure I witnessed things correctly though!

Could you help me further on that, please?
I am not sure I fully understand your point.
I have tried using port 8080 instead of 80 on the YunoHost LXC container (after opening that port on the YunoHost firewall), but the loop remains.

Sorry, I am again not totally sure I understand what you mean: do you think that YunoHost in an LXC container is not a good idea because it implies difficulties which YunoHost was not design to fight against in the first place?
The idea is to benefit from YunoHost on a VPS where I can safely work on other stuff without mixing anything up. From my small skills and way of doing things, I find myself more comfortable with Debian and LXC containers to install very different apps rather than with YunoHost-and-its-many-different-apps-but-all-dedicated-to-self-hosting along with other different apps that I would rather not use with the risk of breaking the YunoHost functionalities.

Thanks for your help!

Aleks · February 5, 2019, 11:27pm

What i mean is that your proxy_pass should proxy stuff to https, not http. Also the proxy_redirect should be set to off i believe…

Here’s a config snippet I use in some context :

    location / {
        proxy_pass         https://the.domain.tld;
        proxy_redirect     off;
        proxy_read_timeout 60s;
        
        proxy_set_header          Host            the.domain.tld;
        proxy_set_header          X-Real-IP       $remote_addr;
        proxy_set_header          X-Forwarded-For $proxy_add_x_forwarded_for;
    }

YunoHost in an LXC is fine I think, but the whole reverse proxy thing makes everything more complicated (except if you manage to get it working properly …) But in practice, what happens is that we (yunohost devs) regularly end up teaching and debugging reverse proxy configuration which don’t really have anything to do with yunohost, all of this “for free” while not making yunohost progress …

So instead, something that you might want to do (if that’s still relevant for your use case) is just to have YunoHost as “front machine”, and then configure reverse-proxy and other stuff, possibly using redirect_ynh, to your “experiment machine(s)”

jellium · February 7, 2019, 10:10pm

My bad! I did not understand precisely your point at first (actually after you explaining a second time ). Port 8080 is not used for https. Even I have switched to proxy_pass https://10.5.0.2; as you showed in your example.

Below is a bit off-topic, hope it won’t bother.

I understand completely. You guys are doing (and have been doing) an amazing job. It may appear a bit selfish but I have learned extensively from this forum and from YunoHost as I have been using it for self-hosting for years now. Not only YunoHost has been useful as a working tool (associated with apps and services such as handling data or mail servers), also I have learned a lot on Debian and the GNU world. My point here (besides telling you guys how grateful I am for what you do for people like me) is that I thought I would use Debian as my VPS OS rather than YunoHost-on-Debian, for everything which is totally not related to the data of the self-hosting solution I use.

You must be right and I may not be looking at how to handle properly the VPS I use for the different things I want to explore. But I feel like I won’t be able to manage properly how to install and test apps that are far away from YunoHost, without breaking or changing confs within YunoHost.

Anyway thanks for your time and help.