Investivate 'breach' and Pentesting/Hardening of Yunohost

Discuss
,
1

Yesterday, my ISP informed me that my ‘computer was attacking people’. I’m not sure quite what that means (it was a Chinese message, translated by my wife). Sadly, I did not save the message. It mentioned something about NSLOOKUP etc.

  1. how do I investigate what happened? I know it was between the hours of 11:44-13:44 11/10 local time. What logs should I be looking at?

  2. Pentesting/hardening of the server. Any suggestions of steps to take to ‘harden’ the server? good pentesting tools or strategies?

I found https://cisofy.com/lynis/ lynis and it looks good to start with. I only get a 67% hardening score though.

2

Just start by this:

  • Are you sure it’s not a spam ?
  • Check if you have yunohost account with weak password
  • Have you some login in /var/log/auth.log around this hour ?
  • Check if you send spam through postfix /var/log/mail.log ?
  • Check the diagnosis tool of yunohost
  • Count the number of apps you have installed (the more you have, the moreit’s risky)
  • Have you wordpress ? Or some apps not up to date ?
  • Have you some apps that turns without dedicated user to this app ?
  • Have you open and forward the 53 port inside your router ? If yes it could be DNS issues ?
  • Is there some strange process visible (with ps aux --forest) ? Especially, check process who are run by specific app users.
  • Have you a backup to restore on a fresh install ?

If you tell us which apps you have installed, we could found some possible explanations.

4

Are you sure it’s not a spam ?

Looks like it is. See below.

Check if you have yunohost account with weak password

Deleted a few unused accounts.

Have you some login in /var/log/auth.log around this hour ?

Looks not good. Lots of random people trying to break in.

    Nov 10 13:14:34 arkadi sshd[14461]: Connection from 149.56.44.47 port 51856 on 192.168.15.228 port 22
Nov 10 13:14:36 arkadi sshd[14461]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=149.56.44.47  user=root
Nov 10 13:14:38 arkadi sshd[14461]: Failed password for root from 149.56.44.47 port 51856 ssh2
Nov 10 13:14:42 arkadi sshd[14461]: Failed password for root from 149.56.44.47 port 51856 ssh2
Nov 10 13:14:46 arkadi sshd[14461]: Failed password for root from 149.56.44.47 port 51856 ssh2
Nov 10 13:14:49 arkadi sshd[14461]: Failed password for root from 149.56.44.47 port 51856 ssh2
Nov 10 13:14:51 arkadi sshd[14461]: Failed password for root from 149.56.44.47 port 51856 ssh2
Nov 10 13:14:54 arkadi sshd[14461]: Failed password for root from 149.56.44.47 port 51856 ssh2
Nov 10 13:14:54 arkadi sshd[14461]: error: maximum authentication attempts exceeded for root from 149.56.44.47 port 51856 ssh2 [preauth]
Nov 10 13:14:54 arkadi sshd[14461]: Disconnecting authenticating user root 149.56.44.47 port 51856: Too many authentication failures [preauth]
Nov 10 13:14:54 arkadi sshd[14461]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=149.56.44.47  user=root
Nov 10 13:14:54 arkadi sshd[14461]: PAM service(sshd) ignoring max retries; 6 > 3
Nov 10 13:14:55 arkadi sshd[14479]: Connection from 198.251.89.99 port 50636 on 192.168.15.228 port 22
Nov 10 13:15:01 arkadi CRON[14482]: pam_unix(cron:session): session opened for user nextcloud by (uid=0)
Nov 10 13:15:04 arkadi sshd[14479]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=198.251.89.99  user=root
Nov 10 13:15:06 arkadi sshd[14479]: Failed password for root from 198.251.89.99 port 50636 ssh2
Nov 10 13:15:08 arkadi CRON[14482]: pam_unix(cron:session): session closed for user nextcloud
Nov 10 13:15:09 arkadi sshd[14479]: Failed password for root from 198.251.89.99 port 50636 ssh2
Nov 10 13:15:13 arkadi sshd[14479]: Failed password for root from 198.251.89.99 port 50636 ssh2
Nov 10 13:15:18 arkadi sshd[14479]: Failed password for root from 198.251.89.99 port 50636 ssh2
Nov 10 13:15:22 arkadi sshd[14479]: Failed password for root from 198.251.89.99 port 50636 ssh2
Nov 10 13:15:27 arkadi sshd[14479]: Failed password for root from 198.251.89.99 port 50636 ssh2
Nov 10 13:15:29 arkadi sshd[14479]: error: maximum authentication attempts exceeded for root from 198.251.89.99 port 50636 ssh2 [preauth]
Nov 10 13:15:29 arkadi sshd[14479]: Disconnecting authenticating user root 198.251.89.99 port 50636: Too many authentication failures [preauth]
Nov 10 13:15:29 arkadi sshd[14479]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=198.251.89.99  user=root
Nov 10 13:15:29 arkadi sshd[14479]: PAM service(sshd) ignoring max retries; 6 > 3

Shoot. Looks like they got in.

    Nov 10 13:37:38 arkadi sshd[14611]: Connection from 106.55.146.113 port 50364 on 192.168.15.228 port 22
Nov 10 13:37:39 arkadi sshd[14611]: Connection closed by 106.55.146.113 port 50364 [preauth]
Nov 10 13:39:01 arkadi CRON[14727]: pam_unix(cron:session): session opened for user root by (uid=0)
Nov 10 13:39:01 arkadi dbus-daemon[458]: [system] Rejected send message, 4 matched rules; type="method_call", sender=":1.6294" (uid=109 pid=14730 comm="worker_nscd 0 ") interface="org.freedesktop.systemd1.Manager" member="GetDynamicUsers" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/sbin/init ")
Nov 10 13:39:01 arkadi CRON[14727]: pam_unix(cron:session): session closed for user root

Since this scare I have 100% moved to SSH key authentication.

Also this log has lots of stuff like this:

Nov  8 15:49:09 arkadi dbus-daemon[458]: [system] Rejected send message, 4 matched rules; type="method_call", sender=":1.5851" (uid=109 pid=10983 comm="worker_nscd 0 ") interface="org.freedesktop.systemd1.Manager" member="GetDynamicUsers" error name="(unset)" requested_reply="0" destination="org.freedesktop.systemd1" (uid=0 pid=1 comm="/sbin/init ")

Check if you send spam through postfix /var/log/mail.log ?

Looks like my machine was being used as a spam server :frowning:

Nov  8 13:30:54 arkadi postfix/smtpd[9761]: disconnect from mone183.secundiarourous.com[141.98.10.183] ehlo=1 auth=0/1 quit=1 commands=2/3
Nov  8 13:34:14 arkadi postfix/anvil[9753]: statistics: max connection rate 1/60s for (smtp:45.125.65.105) at Nov  8 13:29:04
Nov  8 13:34:14 arkadi postfix/anvil[9753]: statistics: max connection count 1 for (smtp:45.125.65.105) at Nov  8 13:29:04
Nov  8 13:34:14 arkadi postfix/anvil[9753]: statistics: max cache size 1 at Nov  8 13:29:04
Nov  8 13:34:41 arkadi postfix/smtpd[9779]: connect from unknown[141.98.10.143]
Nov  8 13:34:42 arkadi postfix/smtpd[9779]: disconnect from unknown[141.98.10.143] ehlo=1 auth=0/1 quit=1 commands=2/3
Nov  8 13:35:52 arkadi postfix/smtpd[9779]: connect from unknown[103.253.42.54]
Nov  8 13:35:53 arkadi postfix/smtpd[9779]: disconnect from unknown[103.253.42.54] ehlo=1 auth=0/1 quit=1 commands=2/3
Nov  8 13:36:34 arkadi postfix/smtpd[9779]: connect from unknown[185.36.81.33]
Nov  8 13:36:35 arkadi postfix/smtpd[9779]: disconnect from unknown[185.36.81.33] ehlo=1 auth=0/1 quit=1 commands=2/3
Nov  8 13:39:39 arkadi postfix/smtpd[9917]: connect from unknown[141.98.10.136]
Nov  8 13:39:40 arkadi postfix/smtpd[9917]: disconnect from unknown[141.98.10.136] ehlo=1 auth=0/1 quit=1 commands=2/3
Nov  8 13:40:32 arkadi postfix/smtpd[9917]: connect from unknown[45.125.65.39]
Nov  8 13:40:32 arkadi postfix/smtpd[9917]: disconnect from unknown[45.125.65.39] ehlo=1 auth=0/1 quit=1 commands=2/3
Nov  8 13:43:53 arkadi postfix/anvil[9781]: statistics: max connection rate 1/60s for (smtp:141.98.10.143) at Nov  8 13:34:41
Nov  8 13:43:53 arkadi postfix/anvil[9781]: statistics: max connection count 1 for (smtp:141.98.10.143) at Nov  8 13:34:41
Nov  8 13:43:53 arkadi postfix/anvil[9781]: statistics: max cache size 2 at Nov  8 13:36:34
Nov  8 14:08:20 arkadi postfix/smtpd[10036]: connect from mone183.secundiarourous.com[141.98.10.183]
Nov  8 14:08:21 arkadi postfix/smtpd[10036]: disconnect from mone183.secundiarourous.com[141.98.10.183] ehlo=1 auth=0/1 quit=1 commands=2/3
Nov  8 14:11:41 arkadi postfix/anvil[10041]: statistics: max connection rate 1/60s for (smtp:141.98.10.183) at Nov  8 14:08:20
Nov  8 14:11:41 arkadi postfix/anvil[10041]: statistics: max connection count 1 for (smtp:141.98.10.183) at Nov  8 14:08:20
Nov  8 14:11:41 arkadi postfix/anvil[10041]: statistics: max cache size 1 at Nov  8 14:08:20
Nov  8 14:19:15 arkadi postfix/smtpd[10215]: connect from unknown[45.125.65.105]

Now it makes sense why my IP is blocked on some email spam things. It looks like the moral of the story is USE KEY BASED AUTHENTICATION!

  • I changed SSH port to non-standard one
  • Key authentication only
  • No root login to SSH

Thank you for the suggestions! I learned a lot about logs and stuff hunting this down.

5

It’s not a problem people try to break in. It’s a problem if you have weak password and no fail2ban.

Strong password = more than 15 chars with special chars, numbers, and upper/lowercase OR 4 words (in lower case)

IMPORTANT: If you change your ssh port, you need to change it in fail2ban rules too.

If you are sure some one is enter and it’s not you, you should consider to reinstall your server properly. Indeed, if someone access as root, the person/robot can install some invisible rootkit.

No if you was sending spam through postfix, you will see “send” word in logs

About SSH config, YunoHost SSH config should be ok, but some server have their own SSH config (you have been asked for this at install or migrations)

6

In Yunohost diagnosis I’ve had this for long time and could never figure out. I never use email on Yunohost.

2: 
        details: 
          - The blacklist reason is: "https://matrix.spfbl.net/180.218.211.159"
          - After identifying why you are listed and fixed it, feel free to ask for your IP or domaine to be removed on https://spfbl.net/en/dnsbl/
        status: ERROR
        summary: Your IP or domain 180.XXX.XXX.XXX is blacklisted on SPFBL.net RBL

But the people didn’t get in. I couldn’t find the phrase “pam_unix(sshd:session): session opened for user root” or “pam_unix(sshd:session): session opened for user admin” in the logs for that time.

So I’m still very confused about: what my ISP was talking about? and why my IP address is blacklisted for email?

I’ll just have to wait for my ISP to bother me again if it happens.

  1. About password strength, my current admin password is 3 random diceware words, total 14 characters. My root password is the same, 3 random words, 15 characters. I guess I should switch to random characters now that I use KEY authentication.
7

Bonjour,

Nous avions abordé avec @Minifab la question du durcissement d’un serveur Yunohost l’an dernier : Question sur le durcissement de la sécurité / Security hardening questions

Et dans cette exploration, l’utilisation de l’outil Lynis (qui permet de faire un premier audite, certes pas parfait et exhaustif, mais qui donne une idée) peut aider à corriger quelques problèmes potentiels.

Pour indication, l’hardening index sur mes Yunohost est de 80 avec cet outil (sans omettre de test).

Si le sujet vous intéresse, je peux détailler la manière dont j’obtiens ce score.

Sangokuss.

1 Like
8

Salut,

perso ça m’intéresse :+1: A mon avis c’est cool si on discute de ça dans une issue github ( https://github.com/yunohost/issues/issues ) pour ne pas que ça reste perdu et enterré dans un thread du forum

1 Like
9

Pas de souci, cela me convient.

Je commence cela dans la soirée (en français ? Cela m’évitera de mal me faire comprendre et il sera plus simple de traduire ensuite)

10

Oui en français si tu es plus à l’aise - dans l’absolument il faut mieux que ce soit documenté en japonais que pas documenté du tout

1 Like
11

Voici un premier jet :

Je complèterai avec du plus “technique” si vous trouvez cela pertinent :wink:

2 Likes
12

If I am right, jail.conf use port = ssh for the SSH jail, so you don’t need to modify the port in fail2ban.

13

Yes you do, because the meaning of the “ssh” you are referring to is defined in /etc/services which 99.9999% of the time is port 22. Fail2ban will not parse your conf files to try to guess which port you’re effectively using, it’s not its job (you could say “oh but it could !” - but it would need to do this for many different kind of services, and in exotic setups you could have several ssh servers running on different ports so things would get complicated - or your could have some funky network rules and so on)

14

Where do you modify it ?

I only changed my ssh port in /etc/ssh/sshd_config, reloaded the ssh service, and fail2ban banned me as intended after I failed to log n times on the new port.

15

By replacing port = ssh by port = 1234 (… the relevant port number …) in the corresponding fail2ban config

16

On va repasser en français, arkadi utilisera un traducteur en ligne (de toute façon il s’est déjà fait hijack son topic :stuck_out_tongue:)

Histoire que ça soit bien clair : quel est l’intérêt d’aller spécifier le port ssh à fail2ban s’il fonctionne déjà nativement avec la clef = ssh, et ce quel que soit le port attribué à SSH ? Meilleures perfs ? Simple question de bonnes pratiques ?

17

Parce que “ssh” veut dire 22, et que si tu changes le port dans la conf ssh (/etc/ssh/sshd_config) il faut propager la valeur de ton port custom sur la conf fail2ban… Sinon fail2ban détectera des tentatives de bruteforce dans le log d’authentification, mais ne bloquera pas le bon port que les attaquants utilisent…

18

C’est là que je ne saisie pas : sur ma config, j’ai changé le port ssh dans /etc/ssh/sshd_config sans toucher à la config F2B.

Déconnexion, tentatives erronées de me connecter en SSH (évidemment sur le nouveau port) => je me retrouve mis dehors, mon IP est bien présente dans la prison sshd.

19

Certes, cool, mais “sshd” c’est juste le nom de la prison, et derrière ça correspond à des régle iptables sur un numéro de port précis. Tu peux le voir par exemple en faisant /usr/sbin/iptables-save | grep sshd | grep port