Unauthorized access, data deleted

Tanaka's picture

Forums: 

Hi everybody,

My Etch Debian server, kernel 2.6.18, running on VmWare virtual machine, has been accessed without authorization and important files and directories have been deleted (all www folder, possibly others too). Unfortunately, there is no recent backup for the deleted files. The server is hosted at the provider site and I have only remote access to it. I also have remote acces to the host machine, besides the virtual one.

I could use some help in trying to secure the system and recover as many deleted information as possible...

Thanks in advance!

Re: Unauthorized access, data deleted

IntnsRed's picture

Ugh, sorry to hear that.

Numerous questions come to mind, the first of all is: Have you examined all of the logs and/or do you know how they got in?

If not, if it were me I'd take a bottom-up approach and would wipe the hard drive and reinstall the entire OS. Etch is old (still updated for security?), so a reinstall may be in order.

It's easy to get into lazy security habits with a remote server. Among common things are installing too much software (apt-get and various web apps make it easy!). Getting lazy about using long and high-secure passwords -- and changing them once in a while -- is another thing.

The backup thing is just your bad. It's easy to run a shell script to tar up things and to download them. For example, I use this script to grab the entire /var/www tree every so often:

#!/bin/sh
#
# Routine to backup essential system and website files
#
SERVERNAME=`cat /etc/hostname`
# No trailing slash below!
BACKUPDIR=/var/backups2download
TIMESTAMP=`date +%y%m%d`

# Backup databases:

# Backup the full /etc subdirectory tree
nice -n 17 tar -cjf $BACKUPDIR/$SERVERNAME-etc-$TIMESTAMP.tar.bz2 /etc

# Backup the entire /var/www subdirectory tree but don't backup files named BackupData*.tar.bz2
nice -n 17 tar -cjf $BACKUPDIR/$SERVERNAME-var-www-$TIMESTAMP.tar.bz2 /var/www --exclude=BackupData*.tar.bz2

# Backup root's subdirectory
nice -n 17 tar -cjf $BACKUPDIR/$SERVERNAME-etc-$TIMESTAMP.tar.bz2 /root

Running such a script will leave the created tarballs in /var/backups2download (aka "BACKUPDIR") for you to download from the server with scp (or however) at your leisure.

You may have to backup MySQL databases too. This can be done with something like:

#!/bin/sh

# Dump all databases on the SQL server:

USER=root
PASSWORD=YourMySQLRootPassword
HOST=localhost
HOSTNAME=`cat /etc/hostname`
# No trailing slash below!
BACKUPDIR=/var/backups2download
TIMESTAMP=`date +%y%m%d`

for EachDatabase in $(echo 'SHOW DATABASES;' | mysql -u$USER -p$PASSWORD -h$HOST|grep -v '^Database$');
   do
      echo "Starting work on $EachDatabase..."
      nice -n 10 mysqldump \
         -u$USER -p$PASSWORD -h$HOST \
         -Q -c -C --add-drop-table --add-locks --quick --lock-tables \
         $EachDatabase > $BACKUPDIR/$HOSTNAME-$EachDatabase-$TIMESTAMP.sql
      echo "BZipping up $EachDatabase..."
      nice -n 17 bzip2 $BACKUPDIR/$HOSTNAME-$EachDatabase-$TIMESTAMP.sql
      echo "All done with $EachDatabase."
done;

There are pros and cons of doing MySQL backups different ways, but this is certainly better than nothing. Smile

If you don't need to back up often, scripts like this and the resulting downloads can be done manually. Of course, they could be put in /etc/cron.daily to be done daily since the files generated are time-stamped. While such a strategy is nowhere near as slick as using rdist or one of the many other backup solutions, having crude tarballs of everything is better than nothing.

If a reinstall is in order, have you thought about using some sort of web hosting control panel? Not only does such a tool make managing web sites easier, but they're pretty good about dealing with semi-obscure security issues and configuring the web server securely. FWIW, ISPConfig is a free software control panel, and their guide to setting up Debian (they call it setting up the "perfect server") is a pretty good tutorial whether you use ISPConfig or not.

Reinstall

Tanaka's picture

Thanks a lot for being so kind and supplying such usefull information, and so quickly too! Truth is, I have been quite negligent with the backups and security checks, a reinstall is in order of course but not right now, first I have to absolutely get that website running as quickly and as close to how it has been prior to the attack, while closing the door to future attacks. Unfortunately the server in case it is a production server, even if the deleted website is only a semi-critical service.
I have looked to syslog, auth.log, vsftpd.log and found nothing relevant so far, also stopped services like www, mysql, vsftpd and kept only qmail and tinydns running (as these are really critical services for us).

The virtual machine has a single "real" partition and a virtual one (I suspect, gathered from free RAM). Would un-mounting and re-mounting read-only the system partition help with the eventual recovery process? Is it possible that this could limit my capability to log into the server afterwards, since getting physical access to the server is no so easy and straightfoward?

Any further help deeply appreciated!

Ok

Tanaka's picture

I'm on it, thanks...

As I said, anything of real help is appreciated!

PS. Indeed, next week, as you said... Right now I should find the back-door and close it while trying to recover as much as possible from the deleted directories (aprox 7 GB from an ext3 partiton of 19 GB). Any suggestions welcome...

Connection

Tanaka's picture

Finally, got to something interesting...
It seems that apache2 or a process with the same name spawns periodically connecting to 2 different IP addresses, one from France and one from Germany. If I kill the process it respawns shortly trying to connect to the other address, at the https (443) port.

After adding a firewall rule which blocks outbound connections to port 443, the apache2 process communication remains in SYN_SENT state. I tried to block the process by renaming the apache2 executable but to no avail. The apache2 process is still starting regularly and trying to connect. Have to stop here for now and continue in the morning. It will be a long day...:(

Re: Connection

IntnsRed's picture

Yes, that is interesting. What userID are the processes running as? I'm guessing they may not be root/superuser and are "just" running as the Apache user, www-data.

Since you can't wipe the entire machine, can you yank(purge) all of Apache and then reinstall and then reconfigure Apache? (A simple apt-get "--reinstall" of Apache may not do anything if the process is some Perl or Python thing that is loaded via an Apache or PHP configuration/module option.)

If it's a web/Apache process, that strategy should allow you to eliminate whatever is spawning that process. If successful, that means you just have to figure out what/where that process was/is, and to ID how they got into the system in the first place.

Thankfully...

Tanaka's picture

...with a little external support I have been able to identify that rogue process (it was running as root, btw, and it was named apache2, same as the authentic apache2 process) and some other two other "supporting" it and deactivate them all. Right now the server runs only sshd and tinydns (which I have mirrored already to another server) and the priority is now to recover as much of the deleted data as possible... Not much expertise here either, so I'm off to reading what I can find online about it... sigh...
Probably will try to unmount/remount the filesystem read-only, while booting from the original CD/DVD ISO mounted image lying on the host machine, and try my luck (as I was dumb enough not to do that in the first place) with some data recovery software like scalpel, foremost, photorec or extundelete...

Thanks for the response and for your interest in helping me! Any further valid advice is welcome...