This server got hacked. It was used as a scanner for a couple days, attempting to break into other servers to test for vulnerabilities. I only discovered this because I attempted to access this server by the .onion domain that I have for it and it didn’t connect.
I’ve had a home server hacked before due to a user account with a I set up for my son with “an easy” password. I thought I learned that lesson. Apparently I did not with the server I have exposed to the interwebs.
Time to forensically breakdown the timeline of events.
Stopping the attack
sshing into the host and viewing the process list I saw there was some processes running in a screen under a user. instead of terminating the processes I logged in as that user, screen -R‘d and saw the scanner running. One CTRL-C later and CTRL-D the screen session and I searched for files but didn’t really see any. I remember I was dropped to a root shell and that caught me odd. Even history didn’t really have anything. I disabled the users login and checked last and verified that the user, in fact, logged in remotely.
Auditing logged-in user
The user that was logged in in no way should’ve been able to sudo su to root. I double-checked visudo, verified what groups the user was in, and didn’t see anything out of the ordinary. Something was amiss.
How did a user get root?
Checking /var/log/secure gave me a bit of insight:
Oct 6 11:17:42 derp pkexec[13768]: unliterate: The value for the SHELL variable was not found the /etc/shells file [USER=root] [TTY=/dev/pts/0] [CWD=/tmp/1] [COMMAND=GCONV_PATH=./.pkexec PATH=GCONV_PATH=. CHARSET=pkexec SHELL=pkexec]
Oct 6 11:18:06 derp pkexec[13904]: unliterate: The value for the SHELL variable was not found the /etc/shells file [USER=root] [TTY=/dev/pts/0] [CWD=/tmp/1/1] [COMMAND=GCONV_PATH=./pwnkit.so:. PATH=GCONV_PATH=. SHELL=/lol/i/do/not/exists CHARSET=PWNKIT GIO_USE_VFS=local]I seemed to be hit by CVE-2021-4034 (blog.qualys.com), which starts with me being an idiot and having a standard user account with an easy password. What was even worse is that the users on my system were being run as if they matched the domain names, which implies they pulled a user list from my webservers SSL Certificates SAN.
This CVE was discovered after I had last upgraded this server on Server Upgrade Time!, and if I kept my practice of awesome passwords for users it wouldn’t have been a problem.
How to fix
I’ve already planned to update this server since pricing is going up and I feel I can get my bang for my buck with less resources. In the meantime I’ve updated passwords for all users again.
Lesson
If you’re paying money for it you might as well take care of it.