Cleaning GitLab Prometheus data

[yunohoster]

My YunoHost server

Hardware: Hetzner VPS, 40 GB
YunoHost version: 11.0.10.2
I have access to my server : Through SSH
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no

This is a continuation of my quest to increase available storage on my yunohost install. You can find part 1 at Reducing Docker directory size (spoiler: the culprit was GitLab there as well).

Description of my issue

I was taking a stroll through the output of sudo ncdu / and decided to focus my attention on /var/opt/gitlab/prometheus/data/, which was clocking at 9.3 GiB.

If I understand correctly, Prometheus being a monitoring tool, such a large size of its data dir would be either indicative of intense levels of activity, or simply of a complete lack of cleaning of obsolete history.

The size couldn’t be due to activity, given that my GitLab only has 1 user (me), there are only a few repos hosted on there, and the intervals between commits are measured in months.

In theory, GitLab’s Prometheus is set to only keep 15d of history by default. For some reason, this seems to not have been taken into account by my install (I might be wrong here, but that’s the simplest explanation I can think of, and that’s why I’m adding this to the YunoHost forum instead of some GitLab-specific resource).

Solution

GitLab provides instructions on how to configure Prometheus to one’s liking here.

There are 2 possible solutions:

1. Depending on what you want to do, add the following config to /etc/gitlab/gitlab-persistent.rb (because gitlab.rb gets rewritten at each upgrade)
   • Reduce the time and size of retained data:

     prometheus['flags'] = {
       'storage.tsdb.retention.time' => "7d",
       'storage.tsdb.retention.size' => "2GB"
     }
     

   • Disable prometheus entirely:

     prometheus_monitoring['enable'] = false
     

2. Take the new config into account by running:

   sudo gitlab-ctl reconfigure
   

This made my /var/opt/gitlab/prometheus/data/ dir size go from 9+ GiB to 70 MiB.