So today one of our servers decided to run a process to the ground

We got an alert that our server was playing up.. This was odd, as we have everything running as it should, i might as well check the server anyway..

Here we go, username... password.. lala waiting for the server to lo... WHAT?!

ERROR: NO DISK SPACE.

This is a problem.. lets have a look,

SSH >

[____@______ ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1               10G  10G      0 100% /
/dev/md2              914G  100G  769G  12% /var
/dev/shm              4.0G     0  4.0G   0% /dev/shm

this is upsetting..

[____@______ ~]# du -sh */

Whoa! What is going on here..

PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
40   0 15264  12m 1004 R  100  0.2 106:55.66 perl

What is perl doing?!

Worried that we had been exploited i quickly jumped into our Skyblue-Batmobile Type car, which proved a useless addition to the troubleshooting session. After putting the car away we got down to some hardcore investigation..

After investigation it appears perl was stuck in a loop.. Our weekly server backup decided to keep backing up, using up 10G of space in the tmp folder and causing server errors..

All fixed now.. well done plesk!!

Note: We always keep full backups of our clients web projects - there is never any danger of our clients losing their projects or longterm downtime.

And if any of you were effected by the short server problems, we apologize!

To show how sorry we are, we've attached a short video for you.