How do I determine what port an Application is using?

[Twizzay]

My YunoHost server

Hardware: Old laptop or computer
YunoHost version: 11.2.8.2 (stable)
I have access to my server : SSH | webadmin | direct access |
Are you in a special context? : I don’t think so.

Problem Apps:

• Bludit 3.15.0~ynh2
• SearxNG 2023.11.29.00.14.42~ynh1

Description of the Issue

I am trying to set up my Yunohost instance through Cloudflare Zero Trust. To do this, I need to know the local address that the aforementioned apps are listening on (Something like localhost:8001).

To find this out, I have checked my nginx configs, and nearly all the files in the locations at /var/www/searxng/searxng-src/.

I have also used netstat -tunlp to check every one of the ports that my machine is listening on with curl http://localhost:<portnumber>.

The Dockerfile of SearxNG seems to indicate that it is listening on Port 8080.

I have tried setting this along with every port listed in the output from netstat in curl and nothing is responding as expected.

Context

Here is the output from netstat

Steps to reproduce

On a machine running this package, run curl localhost:<portnumber> on any of the port numbers listed by netstat.

Expected behavior

curl is expected to return the HTML of the front page for SearxNG when using localhost or 127.0.0.1. With one of the listening ports.
It is returning anything but that.

curl is expected to return the HTML of the front page for Bludit when using localhost or 127.0.0.1. With one of the listening ports.
It is returning anything but that.

TIA

I have been working on this for several days, trying to learn as much as I can. I have read many blogs, including the confusing articles in Cloudflare’s docs. I realize this is a skill issue. There is probably something very basic that I am missing.

Furthermore, I am not a great troubleshooter. Any help is greatly appreciated. Anything you can offer will help me take advantage of self-hosting, this awesome service.

Thank you to all who help maintain this great resource that allows us to take back our data.

FWIW

I was able to find the ports of:

• Pleroma
• Gitea
• Standard Notes
• Filebrowser

Using the netstat + curl method described earlier. I have successfully set them up, and they are accessible through CF’s Zero Trust.

[wbk]

Hi Twizzay,

@Twizzay

I am trying to set up my Yunohost instance through Cloudflare Zero Trust. To do this, I need to know the local address that the aforementioned apps are listening on (Something like localhost:8001).

I have never used Cloudflare (other than while visiting a website that ran through them), and I have no idea what “Zero Trust” is (other than in plain English) so I’m quite clueless here.

Don’t these services run though nginx directly?

tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      824008/nginx: maste
tcp6       0      0 :::443                  :::*                    LISTEN      824008/nginx: maste

Or do you put zero trust between actual service and nginx?

[Twizzay]

@wbk I have never used Cloudflare (other than while visiting a website that ran through them), and I have no idea what “Zero Trust” is (other than in plain English) so I’m quite clueless here.

Cloudflare Zero Trust works by installing a proprietary VPN on your machine, which is then used to route traffic into your LAN without opening any ports to the WWW.

So, eventually, I will close port 443 and the VPN (cloudflared) will connect through local ports on my LAN to route DNS traffic to a local address in my home.

The advantages are that I can securely access any device on my network while outside my house. Zero Trust makes it possible to set rules or a login screen for certain applications that I don’t want to be widely accessible.

The disadvantages are that I am now running the proprietary cloudflared VPN out of my house, and I am therefore passing all of my personal data through Cloudflare. Which kinda sucks.

This is necessary for me because opening port 443 is no longer feasible. My personal IP has been discovered and is being accessed even though I am using Cloudflare. So, DDoS/DOS attacks are now a threat.

[Twizzay]

I discovered yesterday that I can set .local and .test DNS records using dnsmasq and yunomdns so I can access certain services on my LAN exclusively.

This is really cool, but creates a separate issue.

For some reason yunomdns and dnsmasq will only resolve .local addresses for about 15 minutes. Then both services need to be restarted. After which they will resolve queries for another 15 minutes.

This is really annoying. I have thought about making a cronjob to just brute force it into doing what I want, but I decided it’s better to fix this issue for real rather than band-aid it.

Anyway, it doesn’t appear that this is a viable solution because using curl searxng.local or curl bludit.local do not resolve even when both yunomdns and dnsmasq are working.

I have taken the extra step and connected them to cloudflare and am unable to access them this way outside of my house too.

For reference, here is what the zero trust tunneling dashboard looks like. Maybe it will help provide some context to the issue.

[wbk]

I see, thank you for enlightening me

@Twizzay

passing all of my personal data through Cloudflare. Which kinda sucks.

Yes, I feel your pain It is TLS-encrypted before entering the tunnel though, I assume?

@Twizzay

My personal IP has been discovered and is being accessed even though I am using Cloudflare. So, DDoS/DOS attacks are now a threat.

What do you mean by ‘discovered’? As in, “people don’t like my (personal) posts, and they know where my server lives”?

All ports on your personal internet connection are regularly under constant attack of course, even if you’d run your ‘regular’ internet access traffic over (another) VPN.

[Twizzay]

@wbk

What do you mean by ‘discovered’? As in, “people don’t like my (personal) posts, and they know where my server lives”?

Yeah basically that.

Maybe this isn’t an issue with that knowledge in mind. But, I think I would perhaps still like to determine whether this is a possible configuration for yunohost.

It would be very valuable to people who can’t open ports or don’t have NAT.

[wbk]

@Twizzay

I have checked my nginx configs, and nearly all the files in the locations at /var/www/searxng/searxng-src/.

Via a somewhat similar question on Github I found a piece of documentation for searX; it seems to suggest a default lister is on port 4004.

By the way, I had searX running in the (vain) hope to be able to exclude sites such as Pinterest from the search results.
Searching apart from that was fine still though, but it started to misbehave at some point, and for some reason it stopped working all together. How is your experience running searX?

[Twizzay]

@wbk

Via a somewhat similar question on Github I found a piece of documentation for searX; it seems to suggest a default lister is on port 4004.

That may have been true for SearX but doesn’t appear to be the case for SearxNG

@wbk

By the way, I had searX running in the (vain) hope to be able to exclude sites such as Pinterest from the search results.
Searching apart from that was fine still though, but it started to misbehave at some point, and for some reason it stopped working all together. How is your experience running searX?

Well, I am running SearxNG. Searx is actually no longer maintained. Searx has been great. I had a similar concern as yours but there is a tool that allows you to rewrite domains that are pulled in through search.

So, I have rewritten domains like twitter, youtube and reddit to use privacy friendly front end s like nitter, invidious, and old.reddit.
If one wanted to leave a certain domain out, they would just rewrite it to an empty string and this would filter it out of the results.

There are actuall some github gists out there that offer pretty comprehensive rewrite configurations, fwiw.

So far, I have loved SearxNG. Highly configurable, well maintained and does exactly what I need.

I just wish I knew how to access it on my LAN so that I didn’t have to open it up to the whole world.

[wbk]

@Twizzay

I am running SearxNG. Searx is actually no longer maintained.

Hahaha, I actually stopped reading your post, uninstalled searX and installed searXNG before reading on! It works

@Twizzay

I just wish I knew how to access it on my LAN

I had a go at skimming the code at Github, which you might have done earlier.

I do see the code that deals with http (link to one of the files), but I don’t know how it actually connects to the network.

Could it be a socket connection to a generic Python process that then presents the results? In that case, port 6787 in your netstat could be searXNG

Can the access restrictions in “Groups and permissions” help you out?

Edit: I thought group permissions could help shield those apps from not logged-in users; it won’t help your DNS still pointing to your home IP. You could take another (sub)domain for the single app, to not have everything under a domain point to the same IP.

Another workaround, not a usable workaround for just one or two apps, but depending on traffic volume and (perceived) threat, a low cost VPS with Wireguard could perform the non-distributed equivalent of Cloudflare (many come with dDoS-protection).

[wbk]

@wbk

@Twizzay

passing all of my personal data through Cloudflare. Which kinda sucks.

Yes, I feel your pain It is TLS-encrypted before entering the tunnel though, I assume?

I did write ‘assume’ on purpose, but had not expected to find the acknowledgement of the saying so quickly:

@https://github.com/searxng/searxng/issues/1680

Cloudflare can read everything (because they MITM your https connection)

I had until now not paused to think, until just now reading a bit more about it, thad CDN actually is a Cached, Decrypted Nightmare

[mh4ckt3mh4ckt1c4s]

Hi @Twizzay,

I’m the packager / maintainer of SearXNG so I will be able to answer you precisely for that, and give you some hints for your other apps (I never heard about Bludit, though I’ll check after posting this message out of curiosity).

For SearXNG :

SearXNG is not using HTTP(S) behind nginx, but is forwarding the contents directly to a socket managed by uwsgi. uwsgi then proceeds to give the contents of the HTTP request to the Python searXNG app, get the response and send it back through nginx using the same socket. That’s why you were not able to find the port used by SearXNG in localhost : a socket is used instead. For the record, it took me several weeks to understand this whole nginx ↔ uwsgi socket ↔ python app interactions and adapt it to YNH when I first packaged the app.

You may find more information about how to configure uwsgi for SearXNG in this documentation. The uwsgi configuration file used for the SearXNG YNH app is available here, and the installation script uses a systemd service available in this file (this one is mostly not my code, but the one I reused from the SearX YNH app).

On a deployed app, that means you should find the socket at /var/run/searxng/app.socket, and the uwsgi config file at /etc/uwsgi/apps-available/searxng.ini. I don’t know anything about Cloudflare Zero Trust, but you should be able to use these files to either configure Cloudflare to use a socket, or change the uwsgi config to make it use HTTP instead of a socket.

General case :

As far as I know, most applications (at least webapps) are using the nginx proxy to communicate with the apps. The apps not using nginx are for specific use cases, e.g. Adguard encrypted DNS or a Tor relay. This means that you can read the nginx configuration of your apps to find where the incoming connections are forwarded to.

All of nginx configurations files for the apps are located at /etc/nginx/conf.d/my.domain.tld.d/app.conf. For example my configuration file for searXNG is stored at /etc/nginx/conf.d/search.mydomain.tld.d/searxng.conf.

Hope that helps you configure what you want.

Another solution for you ?

From the last post of wbk I understand that the thing you are trying to setup will allow Cloudflare to see all your YNH traffic. Imo this is not a good solution and generally against the values of selfhosting your own services and “take back your data” as you said in your first post. I don’t know what is your exact problem, but another kinda similar solution would be to install the Wireguard YNH app and use it to access securely your services.

With a Wireguard VPN you will be able to close your port 443 and only allow access your services through the VPN. This should solve your issue with DoS threats. Also, it has the following advantages :

• Wireguard is a well-known, open-source VPN solution. You will be able to find lots of support, dive through docs, source code and the code of the YNH app to learn more and configure things exactly the way you want.
• As it is a self-hosted solution, nobody will be able to access your data. An encrypted VPN tunnel will be directly created between your client and your YNH server, and that’s it. No more reliance on a big company.
• VPNing on your YNH server should allow you to interact directly with the nginx proxy, thus saving you the tweaks you are trying to put in place for exotic apps. The Wireguard VPN should allow you to use YNH exactly as normal, but allowing you to remove its exposition from the outside world.

[Twizzay]

Fantastic Answer. Thanks so much for all your work. I will have to find some time to contribute to this project in some way.

Ultimately, after much thought, I decided to do exactly what you prescribe. I have a VPN running out of my house that allows me to route the private data that I want access externally.

This is not a solution for everyone though. For example, I am also hosting Pleroma a decentralized microblogging platform which is a part of the larger Federated Universe. Pleroma, Bludit and any other service that would face the public would need to be accessed without a VPN and would therefore require a port to be open.

There is still a solution, though. In the grand effort to get away from cloudflare, possibly the best solution for self-hosters is to route traffic through a VPN like wireguard to a reverse proxy on a VPS that can handle DDOS attacks. Something like hetzner could do this.

This way you retain security. The drawbacks are that you don’t get all the residual benefits that cloudflare offers for free. Hetzner costs money, after all.

[wbk]

@Twizzay

possibly the best solution for self-hosters is to route traffic through a VPN like wireguard to a reverse proxy on a VPS that can handle DDOS attacks
… [but] Hetzner costs money, after all.

Hetzner is well-known, but not cheapest. I keep an eye on VPS offers on lowendspirit.com. There is a members-only exclusive offers section, but the general offers section now has an offer for quite a fast server at 2,50 euro/month. I try to pay less than that, often in the 1-1,50 range for slightly lower specs.

Depending on from where you expect your visitiors: if you can get by on IPv6-only availability for your services, prices drop considerably (IPv4 costs 1-2 euro per address per month at most places)