So, I have found a solution.
I removed the local user with userdel joris. The LDAP user has not been touched by that.
See linux - How to convert local user to authenticate against LDAP directory? - Unix & Linux Stack Exchange
I run into some problem with the group 1007. Like in Premier utilisateur partiellement admins, ssh impossible
I also had to reboot.
Now I can log with my user, which is in admins group and I removed the admin user.
Thanks for the help!