I have been running a YunoHost server for ~5 years now, but there is one question I have never been able to reply to: how loaded is my server?
I am a terrible administrator, I have no idea how to determine:
if my server is running smoothly
if the server is under stress, and why
what applications are the heaviest
if there is the possibility of installing more apps
when peaks of stress are happening and what is causing them
In general, I would like to understand the fundamentals of server monitoring: what are the most critical metrics and what do they mean? What parameters do I have to keep an eye on?
I installed Prometheus and Grafana, but then I realized I have absolutely no idea what to do next… Do you have any suggestions?
I thought about watching some video tutorials, but I would not really know how they would relate to YunoHost installations.
Feel free to send me any resource that comes to your mind! Once I will have learned the basics, I would be very happy to write some pointers about this in the documentation, or an essential YunoHost Monitoring tutorial.
Hi,
same here, i don’t really know how to monitor my little server. What i do is using htop, there you can see the load and ram usige. And sometimes not all services are running so i check yunohost diagnosis show --issues . On my old yunohost VPS with Debian11 i got email, when an issue was there.
My problem is not really what is not working, because indeed YunoHost’s own diagnosis works quite well. It’s more about what’s stressing/loading the hardware…
I use Monitorix for that purpose. It displays lots of graphs and allows you to select the time period. I haven’t yet configured it to send me emails about critical issues. That would be the next step.
I generally have phpsysinfo, htop, btop, iotop, watchdog installed on my servers.
I still didn’t find an easy way for log analysis and viewing logs on web interface. Some options are available on github but I didn’t find time to look at them.
@tommi I finally got around to thinking about this myself the other day and tried out a few things that have been suggested. I started with Prometheus and Grafana - all you have to do is run the official node_exporter metrics exporter on your machine and add the Grafana node_exporter dashboard. This is the most powerful and complete system, which you can also wire up alerts for, but I found it to be overkill for my needs. Then I tried some of the other tools folks have already mentioned:
Monitorix - provides a nice grab-bag of graphs out-of-the-box, but has an unusable UI, IMO
PHPSysinfo - provides a nice/simple page with basic stats
I also tried Glances, but I didn’t see much benefit to that over shelling into my machine and running top, etc. NetData was a no-go because of the proprietary front-end.
Finally I realized what I really wanted was just basic alerting for memory and disk usage - which the YNH Diagnosis tool already provices While it’s true this only provides memory stress alerting at the time the diagnosis is run and doesn’t account for memory spikes, CPU or disk/network I/O, I’ve found the only thing that’s really caused issues with my YNH instance and apps over the past 6 years is memory and disk space.
That said, if you need more granularity for some reason, then I think Prometheus/Grafana is probably your best option - mostly because of the alerting component. Although I didn’t know you could configure email alerts for Monitorix - might take a second look at it, now that I know about that.
UPDATE: Meh, took a look at the Monitorix alerts - the docs are like the graphs, IMO - all over the place. Lots of noise that I think most people don’t need, and if I’m going to be configuring scripts for each built-in graph to send alerts, then I may as well configure Prometheus Alertmanager or Grafana alerting. If anyone knows any other monitoring tools with a simple and sensible alerting setup, I’d love to know.
As a newbie I may be missing the point here but I use the Cockpit app to see how much memory is being used, how much swap is available, what process load is like and which apps are soaking up both. It seems like a simple easy to use tool to me. What more do you need or importantly what am i missing?
@tzb6js Cockpit looks interesting, love that it’s written in Python - but one thing it at least doesn’t seem to cover is alerting, which I find pretty critical for monitoring. Not sure if that’s a priority for the OP, though. (Oh, and it looks like Cockpit wont run out of the box on YNH if you’re also running Jitsi because they compete for port 9090)
I use Beszel for monitoring and alerting (alerts are being send to me via Telegram). Also I use Uptime Kuma to have some kind of healthcheck (Uptime Kuma also sends alerts via Telegram).
Important note here is that I host Beszel Server and Uptime Kuma on external VPS which is connected with my Yunohost box via Tailscale (Uptime Kuma just pings the web services, but Beszel needs beszel-agent installed on Yunohost box).
I use Webadmin, this seems to provide all the info that I needed on the health of my server. It also provides me with UI to navigate my Yunohost server internal file structure and installed apps.
One thing I’m concerned about is to figure out what is the impact of AI (and other) crawler bots on my server’s resources.
How can I tell what is legitimate traffic and what is hammering I want to block/ban. How can I block/ban that after I’ve found it? Is Fail2Ban enough here?
As @tommi implied: what good are numbers and graphs without the context knowledge that a less-experienced – e.g. I’ve been self-hosting for decades now, but I’m still not much of a SysAdmin IMHO – YunoHost admin would simply not have (yet).
Some noobie-friendly guides would be more than welcome. And thank you, @tommi , for both bringing this up and volunteering to kick-start the documentation effort. I’m happy to contribute in what little I can.
I know about Anubis and its ilk. I’m just unsure if I need it. And since it does make everyone’s lives a bit harder (and wastes their resources a bit), I would much prefer not deploying it preemptively.
Also Anubis probably does not protect me from an angry script kiddie or a different (D)DoS.
hi
I use Server box, user-friendly with no extension on server.
Serverbox is a versatile, open-source application designed for Android devices, offering users a powerful tool to manage and monitor servers directly from their mobile devices. This innovative software provides a comprehensive suite of features tailored for system administrators, developers, and IT professionals who require on-the-go access to their server infrastructure.
Key Features:
Remote Server Management: Serverbox allows users to connect to and manage remote servers securely. Whether you need to execute commands, monitor server performance, or manage files, Serverbox provides an intuitive interface to handle these tasks efficiently.
Multi-Protocol Support: The application supports various protocols, including SSH, FTP, and SFTP, ensuring compatibility with a wide range of server configurations. This multi-protocol support enables seamless integration with existing systems and workflows.
User-Friendly Interface: Designed with usability in mind, Serverbox features a clean and intuitive interface that simplifies complex server management tasks. The user-friendly design ensures that both novice and experienced users can navigate the application with ease.
Security: Security is a top priority for Serverbox. The application employs robust encryption and authentication mechanisms to protect sensitive data and ensure secure communications between the mobile device and the server.
Customization and Extensibility: Serverbox offers extensive customization options, allowing users to tailor the application to their specific needs. Additionally, its open-source nature encourages community contributions, fostering continuous improvement and innovation.
Real-Time Monitoring: With Serverbox, users can monitor server performance in real-time, receiving alerts and notifications for critical events. This feature enables proactive management and quick response to potential issues, minimizing downtime and enhancing reliability.
Serverbox is an invaluable tool for anyone involved in server management, providing the flexibility and functionality needed to maintain optimal server performance from anywhere, at any time. Its open-source license ensures that users can benefit from a transparent, community-driven development process, making it a trusted choice for mobile server management.
Sources: F-droid
Thanks @tommi for starting this thread. I was looking into this subject at this very time and all responses given here have been extremely useful to me to weight different options. Also thanks in particular @yekibud for your comments, I found them very useful.
I’ll summarize here what I’ve found, maybe it can be of some use for others, please note that I haven’t tested most of those things, I only gathered them from this thread and other sources:
Nice tools for a quick, extensive and readable instantaneous screenshot of your server:
They are convenient display of what’s happening at that moment, but not over time.
Very comprehensive monitoring tool, but requires most probably quite a bit of time to setup, to connect it with other elements like prometheus or others. If one wants to use netdata as well, maybe the yunohost package works out of the box though.
Server admin interface, with a really impressive amount of features. Very useful to learn about many settings one has to handle in a server. But that’s giving lots of powers to a webui, and the secuity risks that goes with it.
Very comprehensive monitoring tool. Works out of the box with the yunohost package, and very easy to configure graph reports and alerts on some stats through the yunohost admin panel.
Very comprehensive monitoring tool: “Designed to be very plug and play, a default installation provides a lot of graphs with almost no work.” Unfortunately the yunohost package is not maintained anymore.
Their website is very corporate, but it seems like probably it can do everything one can hope to do with monitoring. Don’t know how easy for small setups like yunohost.
Like their website say: “a fully featured network monitoring system that provides a wealth of features”. Testing it with their demo site, it seems the interface is quite complex though for beginners, so will require a bit of time to learn to use it.
Conclusions
In the end I decided to try using monitorix. Hope it’ll be a good choice over time. But to answer a bit to @yekibud’s critics to it, the yunohost package makes a great job at having it working out of the box, and configuring it for reports.
The monitorix yunohost admin panel also makes it easy to configure alerts for a few parameters without having to script anything. And to have more out-of-the-box alerting available for all yunohost users requires just one person to take the time to create those config scripts and propose them to the monitorix_ynh repo.
Thanks everyone for your suggestions. Please feel welcome to correct me for anything I’ve gotten wrong.
And @tommi or others, if you want to use this as a basis for the documentation page on monitoring you were thinking of, feel free to do so
EDIT1: Added goaccess, zabbix and LibreNMS after discovering this old similar thread.
EDIT2: Added cronicle.
While it’s true this only provides memory stress alerting at the time the diagnosis is run and doesn’t account for memory spikes, CPU or disk/network I/O, I’ve found the only thing that’s really caused issues with my YNH instance and apps over the past 6 years is memory and disk space.