All services hosted by yunohost VM return 502 / connection refused

Support
1

My YunoHost configuration

Hardware: vm in computer
Internet access: in a datacenter
YunoHost version:
yunohost: 3.4.2.1
yunohost-admin: 3.4.2
moulinette: 3.4.2
ssowat: 3.4.2
Have you personalized your yunohost with some specifics configurations or do you use only the yunohost cli/webadmin tool ?
no personalization

Description of my problem

I discovered today that all the services in my yunohost VM are not accessible through the browser. I didn’t make any change, didn’t touch the server in a while. The forwarding from the main machine didn’t change. I can’t really understand where the problem is coming from.
From the main machine, I checked logs when I make a request to a service that the yunohost VM serves.
It looks like this
07/Mar/2019:22:22:22 +0100 [ERROR 502 /] dial tcp 192.168.xxx.xxx:443: connect: connection refused

Mails are still working if using a mail client.
I still can ssh to the yunohost VM, and from the inside I can’t find anything looking wrong.
Seems to be an http problem. I don’t know.
I tried restarting nginx with service nginx restart and systemctl restart nginx, and also restarting the yunohost vm. No sucess.
I also ran yunohost tools diagnosis nothing looks strange.

Any idea where this could be coming from?

2

Meeeh, I dunno, I guess that could be something in the reverseproxy setup …

But let’s try this : if you go inside the server using SSH and try to curl a page either in HTTP or HTTPS, does it works ?

3

I tried those two that are both normally valid accessible services:

root@yunohost:~# curl someservice.mydomain.tld
<a href="https://someservice.mydomain.tld/">Moved Permanently</a>.
root@yunohost:~# curl yunohost.mydomain.tld
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<center><h1>302 Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
4

And from the host machine, what about if you curl https://192.168.x.y ?

5 
curl https://192.168.x.y
curl: (7) Failed to connect to https://192.168.x.y port 443: Connection refused
6

Well uh, I dunno, is there a firewall or something that is preventing access to port 443 ? Either in the host, or in the VM …

Edit: also, can you check in the VM that fail2ban did not ban the host’s IP for example …

1 Like
7

There is something that looks like this in /var/log/fail2ban.log
2019-03-07 19:19:19,132 fail2ban.actions [1554]: NOTICE [recidive] Ban 192.168.122.1
Could be that everything went down around seven, because that’s a bit after that that I realized there was a problem.
It’s probable that 192.168.122.1 is th IP of the host no ?

8

Yup, would look like it …

To be checked what the other entries in the log say, but it could be that too many failed login would trigger this, and the IP of the host gets banned because all requests appears to be coming from the host …

The reverse-proxy could be configured to send the original IP though

2 Likes
9

Ok cool.
So if I understand well from this page

Running iptables -L -n, I find this:

Chain f2b-recidive (1 references)
target     prot opt source               destination         
REJECT     all  --  192.168.122.1        0.0.0.0/0            reject-with icmp-port-unreachable
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Then to unban it I should run:

fail2ban-client set recidive unbanip 192.168.122.1

And then I should properly configure my reverse-proxy in the host so it forwards the IP, in order not to have this problem again if someone fails to login to many times.

1 Like
10

Yup :+1:

11

Yay, it worked, thank you :slight_smile: :+1::+1::+1:

fail2ban-client set recidive unbanip 192.168.x.x solved it

1 Like
12

I looked into host’s Caddy config, who is proxying requests to the vm to do what you said, and realized it was already done.
The thing is that the original IP is put in the request’s header, and it seems that nginx in the vm is not taking that into consideration (maybe to do it, it needs this plugin??)

Does that mean I have to tweak nginx in the yunohost vm to achieve that?
If yes, is that a good idea, is it likely to create compatibility problems with some ynh things, and will it not create issues or disappear on upgrades?
Sorry if those are stupid questions, but that’s how little I know about this subject ;/

13

To me it’s not 100% clear either, I don’t know so much about reverse proxy except what I learned from tweaking around :confused: … Naively I’d think it’s the job of the host to send the real IP and that should be transparent to the guest but yup, maybe the guest in fact needs special configuration …

14

I guess that’s what I have to do, tweak nginx config so that it logs the IP that is in the request’s header.
From what I understand, fail2ban checks IPs from nginx logs, so configuring nginx properly should be enough to prevent the initial issue from happening again.

15

So using ngx_http_realip_module plugin that I mentioned before was actually really easy, didn’t have to install it in the ynh vm, it was already usable.
Then I put the following config in /etc/nginx/nginx.conf:

set_real_ip_from 192.168.xx.xx;
real_ip_header X-Forwarded-For;

I checked the config was good with: nginx -t and ran service nginx reload.
Now all nginx logs have the appropriate request’s IP, not the one of the host. So I assume fail2ban will work properly now :slight_smile: