now before we will go deeper into analyzing,
please navigate to:
/var/log/mysql
and check if you have any error log in present?
Here are the last lines of error.log : hastebin
there some issues there allow me a moment
1 Like
oom killer is because of the overun stuck, (out of memory)
can you please share the whole error log? i want to check previous events
It’s weird, I share you the logs through the “service” menu in yunohost admin panel but the log file in /var/log/mysql is empty…
that’s probably because yunohost did changed the native mysql log location to another path, which i have no idea where is it,
do you think its the whole log? or partially?
i am little bit in hesitation cause i think if we will not see the whole log we will work harder that might can lead us to faster to the problem
? full log?
i search an answer in the forum… but I don’t find anything
For example :
journalctl -u mysql
-- Journal begins at Wed 2022-08-10 15:15:39 CEST, ends at Wed 2022-12-07 17:20:58 CET. --
-- No entries --
okay never mind its fine, we will go for the things we should go from here,
in general there are some php configurations we can made to avoid crash but that is a complete wrong way to my opinion,
its like if some has a headache instead of finding if he dry and need to drink more water we will give him a medicine to shut hip up or better shoot him,
so we will start with the best way to do it but it will not be something we can do in one shot,
it will take us hours or even days to done,
and you need to be aware to that your server will not be fully available for its purposes and be sure you’re alright with this?
Ok, if the server is not available, I have some announces to make for several people who use my services. So, can you tell me when we will make something like that ?
i have no idea cause what we are trying to do is this:
we need to wait several hours for RAM consumption just more than what is it and be will much more than it is now under some different conditions,
which slowly will indicate what process bring us to this situation,
in general oom killer get active if the sql server is lck’s buffer more than it needs due to processes or else table structure or else web server tolerance misconfigurations,
for now what we need to do the best in concerning about your server availability is to reverse things in beginning from less important things to the most important things,
we need to neutralize some functions and follow the glances,
for so example is “mattermost” is the most important we will go for the things that we don’t really need them and vise the process that way will be easier to understand what’s going on
@pcet
oh so by the way what is your most important app on your server to be available for your users?
Ok. So Freshrss is the most important, follow by Mattermost, Wallabag and dokuwiki.
I will monitoring ram and send you Glances results time after time. I will reinstall Netdata because he sends me mails when RAM is almost full
no need to send me glances,
do this please:
first disable any of your external websites:
we need to make sure no php requests comes from them whatsoever
my_webapp : just a ftp serveur where I test html/css pages
multi custom webapp : just a hugo static blog
multi custom webapp : phplist for sending newsletters
uninstall these apps:
SnappyMail
Kanboard
Shaarli x 3
OpenSondage
Bludit x3
after you done to do so, get into glances get a screenshot for yourself, better in some folder and give it a name like 01 or something, make a not to yourself what date/time exactly you took a screenshot of it,
we will wait several hours maximum 12, and will check if things go bad,
the most important thing to do, while we are waiting, is,
please try in some how to stretch requests and writing sql queries as much as you can by
inserting things
for example if your users can help you by log into mattermost and paste dummy content as much as they can that will be great, you also can do it yourself
RSS or whatever the way it works, so you anyway you getting me i think,
we need to create a situation that the server is writing and executing as in an usual situation of many hours in a shorter time
1 Like
Ok. So, I make this in a few days because Shaarli and kanboard are used everydays by some users so I have to prevent them.
So I will wait until the RAM is almost full and prevent my users today so that they can be organized.
Thanks a lot for your help !!
sure,
remember that you can also, create pressure by yourself even without deleting the scripts,
and watch online,
for example Shaarli, instead of to deleting it,
stretch it like as much as you, upload things or whatever changes you can made,
according to your descriptions its look like to me some of the process’s queries whatever is it are very brutal,
so its means, that doesn’t have to get long time, once you will touch the right thing it will be very brutal with RAM consumption, you supposed to see a very quick response of consumption at the moment,
cause its just taking days to increase ram in an ordinary situation when a user does something rarely in hours,
but if you will do that intentionally app by app you will see radical changes on the ram
in that moment you should notice something unusual,
that’s another method,
it depends on you how much you’re going to invest on this
@pcet
i think don’t block your users just leave them in to help you stretch the server but you need to do changes as much as you can on one app and check the ram,
after you will do that, if nothing helps that will bring us to a serious problem which we will investigate some cron jobs, but i don’t think we will get into it
Ok, but I cannot really tell my users to make anything on there services because this is private or public usages and they don’t really can make this stuff.
So I prevent them, and I will think about the procedure
you dont need ask them to do anything, whatever they does they does,
but again what you can do is this,
login as a fake / temp user and start upload as much photos as you can,
also, you can create fake users, open a browser, and login as as one by one of the fake users, upload and do things as a usual user will do, just make yourself a folder with a copied x 200 jpg files, 5 fake users with different names such as temp1 temp2 with the same password,
if you does changes on Shaarli and you don’t see the RAM moves much move on, to the next app,
that’s the idea, if i was doing that i will find very quick that way which process doesn’t release the sql tables / connections, and that’s the only way, cause the app what does it,
doesn’t produce errors cause its actually are not errors, and mysql does produce errors? well we couldn’t get any normal error log so i don’t know,
this method is very effective, i could do it in better way, but the problem is your server is involved with people and so on,
i think most of us don’t use yunohost as a production server, most of people use yunohost for private purposes you know like just friends and family, maybe some i dont know, but anyway in your situation things become little bit harder and that a good way to do it,
as i said in the beginning, just by looking on a performance indicator will not bring you anywhere, but if you stretch the server and watching at moment that can lead you somewhere,
or instead of stretching, in an situation when you don’t mind to take the server down,
we goes by reducing processes and investigate just few and less and less apps till we get the problem cause, which are not so possible currently
Ok, I understand. I will do all these tests after 19 december because I don’t have time until then. I’ll keep you informed.
Thanks a lot for the time you spent for me 
1 Like
@izakis , thanks for the in-depth troubleshoot!
1 Like
J’ai déinstallé les applis Rainloop, Bludit, Kanboard, My Webapp et Multi Custom Webapp. J’en ai transféré certaines sur un hébergement mutualisé.
Quelques jours plus tard, le serveur a replanté pour les mêmes raisons (mysql crash à cause du manque de ram).
J’ai constaté aussi que le processus PERFCTL que j’avais supprimé est revenu : Processus étrange : PERFCTL - #3 by pcet
J’ai donc créé hier un autre VPS, installé les mêmes applis et j’attends une semaine pour voir si le processus apparaît ou non. S’il n’apparaît pas, je ferai une fresh install de mon serveur qui est peut-être infecté.