Get Grafana alerts over ntfy

[u9000]

What app is this about, and its version: Grafana, ntfy
What YunoHost version are you running: 12.1.39
What type of hardware are you using: Other(?)

Describe your issue

i got grafana to work with ntfy, but it was a pain in the ass to figure out so here’s how to do it if you want:

Setting up the alert contact point

1. In Grafana, go to Home > Alerting > Contact points and press “create contact point”.
2. Set the integration to “webhook” and the URL to “https://ntfy.sh/” (or your self hosted version). Importantly, you want to link to the root URL, not the specific channel you’re trying to send to.
3. In “Optional webhook settings”, set “HTTP method” to POST and paste your ntfy access token in “Authorization Header - Credentials”.
4. Scroll down to “Custom Payload”, press “add”, press “edit payload template”, and then press “Enter custom payload template”.
5. Enter the below JSON into the custom payload template (but update it for whatever event you’re trying to send an alert about)

{
    "topic": "ntfy-topic-name",
    "message": "RAM usage is above 80%!",
    "title": "80% RAM Usage",
    "tags": ["warning"],
    "priority": 4
}

Setting up the alert condition

For this example we’re going to assume you want to alert when RAM usage is at 80%. Also, it is tremendously helpful to have a dashboard like Prometheus Node Exporter Full already setup because then you can just copy the queries.

6. In Grafana, go to Home > Alerting > Alert rules and press “New alert rule”
7. In a new tab, go to the dashboard with the metric you want to copy, and press “edit dashboard”.
8. When you look at the code for the RAM Used metric, we see it’s the following:

clamp_min((1 - (node_memory_MemAvailable_bytes{instance="$node", job="$job"} / node_memory_MemTotal_bytes{instance="$node", job="$job"})) * 100, 0)

9. Back in the alert creation tab, select node_memory_MemAvailable_bytes as the first metric.
10. We know that the filters we need are instance="$node", job="$job", so set them as such. You’ll need the actual values of the variables though, which are at the top of the dashboard page.
11. Now add a second query for node_memory_MemTotal_bytes.
12. Then in the “Expressions” section of the page, delete the threshold expression.
13. Add a math expression. The original formula was node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100. We’re going to use the same formula here, but the variable syntax is different, so it will actually need to be $A / $B * 100
14. Finally, add a threshold expression. Set “Input” to C (which is the math expression we just made), and “is above” to 80.
15. Press “Set ‘D’ as alert condition”.
16. Set up sections three and four on that page however you like.
17. In “5. Configure notifications”, set the contact point to the one you created earlier.
18. Press save

Share relevant logs or error messages

[u9000]

[reply needed to mark topic as solved]

[ewilly]

Thanks
I add it to a wiki page in the package: Get Grafana alerts over ntfy · YunoHost-Apps/grafana_ynh Wiki · GitHub