How to repel crawlers, bots and malicious attempts

Discuss
, , ,
1

My YunoHost server

Hardware: Old laptop Lenovo T400
YunoHost version: 11.2.10.3 (stable)
I have access to my server : Through SSH, through the webadmin & direct access via keyboard / screen
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no, however
If yes, please explain:
I placed a blocklist for amazon servers from ASN Info
in /etc/nginx/conf.d/block_amazon_AS16509.conf

Description of my issue

Since a couple of days my internet at home is unusably slow.
It turns out, that I have tons of connection attempts and crawlers (especially from Amazon) connecting to my yunohost machine.

Somehow I cannot find good explanations on how to deal with this and protect my computer.
I attempted to put a blocklist in nginx, I know that the blocklist is being read, however it does not help the situation.

I hope that I can gather some more information on this, however for now I simply shut down the server with a little tear in my eye and give up on self hosting until I can resolve this somehow. It feels especially sad that it appears that most of the traffic comes from known Amazon crawlers, and somehow I cannot stop them from entering my network.

I’d be superhappy if someone can point me in the right direction what to do in these situations with fail2ban/nginx/iptables/etc.

THANK YOU!

2

How you knew it?

3

valid question!

Sorry for the late reply.

I noticed a general low internet connection at home, and checked the traffic on the yunohost server with tcptrack and iftop.
The traffic was caused by ip adresses that are linked to amazon services according to ASN Info.
A search in the internet lead to other people with the same symptoms identifying these as crawlers.
So that’s how I came to the conclusions above.

The traffic stopped after around two weeks (that’s a couple of days ago).

For the future I’d be very curious how I could react better in a case like this.
My hunch is to add blacklists to nginx, but I didn’t want to mess up something (like yunohost updates) by doing it manually.

4

You can add custom configurations in /etc/nginx/conf.d by creating a new file. Then check the nginx config is OK : sudo nginx -t and reload it.

1 Like
5

adguard home would not help in that case?

1 Like
6

yeah, that’s what I tried. on ASN Info you can download an nginx conf file.
I put it in /etc/nginx/confd and saw in the logs that it was loaded, but somehow it had no effect.

I’ll double check if I did something wrong, thanks.

7

ha, I didn’t know about adguard home. I’ll try that, thanks!

1 Like