Where or how to find process that suddently and repeatedly consume all memory

gdhia's picture
AttachmentSize
Image icon log.png21.22 KB

Forums: 

Hi

Since two days, my Debian (with Parallels Plesk v12.0.18_build1200140606.15 os_Debian 6.0) dedicated server, becomes unreacheable through http, pleks admin, ftp, and ssh. But it remains accepting PING. This occurs each few hours, and i have to restart it through hosting web pannel.

This server is hosting 5 websites, with a 6th one based on Drupal COD distribution, that i recently installed (4 or 5 days ago)

Today, Support team of the hosting company told me that my server run:

"out of memory"

. I installed "Health Monitor" on plesk, and i got this graph: http://fr.tinypic.com/view.php?pic=zjv6yv&s=8

Now, how can I find which process is suddenty taking all this memory ? is there a log file on which I can find this information and if so, which entry should I search for?

If not, is there any other way to know, on that period (between 16h30 and 19h00) what happened on my server?

Thank you all

Piggish program detection

IntnsRed's picture

> Now, how can I find which process is suddenty taking all this memory ?

Interactively, you can log in and run the "top" command and press "M" to sort the results by memory amount/usage. This will give you an updated, real-time snapshot of what's using how much memory on a running computer.

Your problem is that you need to check things during the wee hours of the night, or some other odd time.

For something like this, we can use the "ps" command and string it together using some other *nix tools. PS gives tons of information on various running processes. Running a "ps auxw" will give you an example. Reading the manual page ("man ps") is informative but it's a long man page.

So a "ps aux" will be our basic command (read the man page to figure out what those switches/options make the program do). But we want the output sorted by memory usage. And even more, ps's output is too long. We don't want to know about the memory usage of all processes, just the piggish ones. So to accomplish all that, we'll do this:

ps aux | sort -nrk +4 | head

This runs "ps" and pipes its output into the "sort" command to sort things for memory usage (a "man sort" will tell you what the switches mean/do). We'll then take that output and pipe it again into "head". By default head will show us only the first 10 lines -- cool, we don't need to know about everything, just the big pigs. Voila! We have a snapshot of the computers top 10 piggish memory using programs. (Read the ps man page to understand what the various columns mean.)

Of course, we still have to figure out a way to do this during the wee hours of the night (or whenever). For that we can use the system "cron" scheduler to run the task.

What I'd do is to have the system send me an e-mail with the output. I can do this by making sure Debian's "mailx" package is installed, and then creating a small shell script.

So I can create a shell script called something like "CheckForPiggishPrograms.sh". That shell script could include something like this:

#!/bin/sh
ps aux | sort -nrk +4 | head | mail webmaster@debianhelp.org -s "Top 10 Memory Pig Programs Snapshot"

(Note: The site's anti-spam features mangle that e-mail address.)

This takes the output of the above ps command and adds another pipe, piping that output into the "mail" program to send myself an e-mail with an appropriate mail subject.

A "chmod +x CheckForPiggishPrograms.sh" would make the file executable so I can run it.

At that point, I can stick that "CheckForPiggishPrograms.sh" shell script file into /etc/cron.hourly and I'll get an e-mail every hour with the results of the ps command which will list out the top 10 memory piggish programs. Or I can set up a specific cron job if I want to have the script run more frequently and/or only between certain hours.

This is a classic illustration of Unix's modular tool philosophy: Create small, well-written tools. The small tools are easier to write, debug and maintain. If you need a more complex task, simply string together (integrate) some of the various tools to accomplish that more complex task.

P.S. My guess as to what's going wrong without knowing anything? You're being hammered by spammers who want to post crap to push up their Google ranking. Google has (unwittingly?!) created a huge incentive for spammers to do this to push up their PageRank.

Many companies and web developers sell "services" and brag to their clients that they can improve their ranking on Google searches. Many do this by spamming sites with links to their clients. It functions as a corrupt, greed-based system that creates problems for countless thousands of web sites. But all this benefits Google -- Google has entire companies and many experts which rely on their knowledge of Google's search engine details. And when you have many companies and people dependent on you, that is power that can be translated into cash somehow. It's very similar to Microsoft's relationship with anti-virus software companies.

Thus, your machine is getting bogged down and Apache or your web server is eating up the RAM trying to process huge numbers of connections by the spammers. Just a guess...