How to rescue a server with no free inodes (from DDOS)

by Reed   Last Updated April 26, 2017 15:00 PM

One of my web server was DDOS attacked. All is well except there are millions of PHP session files used up 100% inodes of the partition. There is only one partition for the entire /

Tried several solutions, but only worked to some extend.

After freed up 8% of inodes, the disk become extremely slow to delete anything more.

rm -f filename* 

rsync -a --delete empty_dir/    yourdirectory/

perl -e 'for(<*>){((stat)[9]<(unlink))}'

And the disk look like this now

Filesystem      Inodes   IUsed  IFree IUse% Mounted on
/dev/vda1      2621440 2385895 235545   92% /
tmpfs           128789       1 128788    1% /dev/shm

There are still 6million+ small files in a dir. The above methods delete at about 2 files/sec

I read about b-tree re-balancing. But how do I diagnose/solve the slow delete problem?


Answers 2

@HBruijn's comment is spot on.

After moved the large dir, I found out a new round of DDOS attack was coming and creating hundreds of files per sec.

That is the cause of slow io performance. Had to stop the webserver to clean up files.

April 26, 2017 14:54 PM

A quick thing to do is to move/rename your current /tmp directory and create a new one so that normal usage of /tmp isn't impacted anymore.

mkdir /newtmp
chmod 1777 /newtmp
mv /tmp /tmp-old && mv /newtmp /tmp 

and maybe you need to do the same for /var/tmp as well.

April 26, 2017 14:54 PM

Related Questions

How to clean up an unprocessed orphan inode list?

Updated September 14, 2018 03:00 AM

Understanding XFS inode limits

Updated August 08, 2018 06:00 AM

File recovery through inodes

Updated June 15, 2017 18:00 PM

What happens to orphaned hard links?

Updated November 14, 2015 06:00 AM