Prevent LLM scrapers/trawlers?

Thanks for starting this discussion, I had also been thinking about this.

I’m sympathetic to the bot tarpit approach, but would be happy just to add them to fail2ban, and not let them waste any additional cpu cycles.

I while back I had installed darkvisitors to my website, and dismayed but not surprised to see that 90% of the traffic was coming from these llm/scraper bots.

I think a project like this would be great to add somehow to yunohost, and from a glance seems compatible: