disk i/o performance problem

On Thu, May 31, 2007 at 11:02:06PM -0400, WHIRLYCOTT wrote:
> Hi, I'm trying to track down a pretty apparent problem that I'm
> having with disk i/o on an Etch box. Basically, I have two identical
> machines. One is running Etch and one is running CentOS. The Etch
> machine is really slow. I'm trying to figure out why. Details are
> as follows:
>
> Etch: (output of hdparm)
>
> /dev/sda:
> Timing cached reads: 1588 MB in 2.00 seconds = 794.41 MB/sec
> Timing buffered disk reads: 176 MB in 3.03 seconds = 58.15 MB/sec
>
> /dev/sda:
> IO_support = 0 (default 16-bit)
> readonly = 0 (off)
> readahead = 256 (on)
> geometry = 30401/255/63, sectors = 488397168, start = 0
>
> Linux etch 2.6.18-4-k7 #1 SMP Wed May 9 23:42:01 UTC 2007 i686 GNU/
> Linux
>
> CentOS: (output of hdparm)
>
> /dev/sda:
> Timing cached reads: 4480 MB in 2.00 seconds = 2242.86 MB/sec
> Timing buffered disk reads: 176 MB in 3.03 seconds = 58.16 MB/sec
>
> /dev/sda:
> IO_support = 0 (default 16-bit)
> readonly = 0 (off)
> readahead = 256 (on)
> geometry = 30401/255/63, sectors = 488397168, start = 0
>
> Linux centos 2.6.18-8.1.3.el5PAE #1 SMP Mon Apr 30 20:26:37 EDT 2007
> i686 athlon i386 GNU/Linux
>
> Please note that I'm specifically not trying to spark a centos/debian
> war here. I just provide the CentOS numbers because it's the same
> hardware and there's a pretty obvious difference between the two
> otherwise identical machines.

these are cache reads, try running 10 in a row and taking a look at the
averages.

the other thing i notice is that you have PAE kernel for centos - does this
give it access to a large kernel memory space ?

>
> I'm not sure where to look next. The fact that the hdparm settings
> are identical indicate that the problem (or setting?) may be
> elsewhere. But where?
>
> Does anybody have an idea of where to look?
>
> phil.
>
> --
> Whirlycott
> Philip Jacob
>
> http://www.whirlycott.com/phil/
>
>
>
> --

On Thu, May 31, 2007 at 11:02:06PM -0400, WHIRLYCOTT wrote:
> Etch: (output of hdparm)
>
> /dev/sda:
> Timing cached reads: 1588 MB in 2.00 seconds = 794.41 MB/sec
> Timing buffered disk reads: 176 MB in 3.03 seconds = 58.15 MB/sec

> CentOS: (output of hdparm)
>
> /dev/sda:
> Timing cached reads: 4480 MB in 2.00 seconds = 2242.86 MB/sec
> Timing buffered disk reads: 176 MB in 3.03 seconds = 58.16 MB/sec

The buffered disk reads are the same since that's the transfer rate from
the drive's buffer. The problem is the cache reads which is the time
from you OS's cache.

What does vmstat show as cache size with both OSs? Does one OS have
more processes going than the other? What happens if you boot up each
in init=/bin/sh, mount whatever filesystem holds hdparm ro, and run the
timing tests that way with only those few proceses running?

Finally, benchmarks are all well and good but how does this impact you
directly? What do you do that relies on fast cache disk IO for which
you end up waiting?

Doug.

--