Troubleshoot performance - part 6. Disk issues with iostat

Iostat for pinpointing bottleneck devices

If problem is on IO side, iostat can be used to check “hot” devices. These figures should be compared to the theoretical capacity of the devices.

There are too limiting factors for the disk speed: channel (bus or cable) and the disks itself. The channel (IDE, SATE, USB etc.) is seldom the limiting factor in personal or workstations (perhaps if you would put an external SSD to an oldest possible USB channel, there could theoretically be some impact). Not even if there are multiple disks in the same channel. Also in external disk arrays (EMC and the likes.) the connections are usually optic and the system design is such that there won’t be a bottleneck there. You can look at different channel bit rates here: http://en.wikipedia.org/wiki/List_of_device_bit_rates

Usually the disk itself is the limiting factor. Manufacturers publish performance numbers but in practical terms they are often not directly comparable. A common way to tell performance is to indicate IOPS (Input/Output Operations per Second). A disk may for example do 200 IOPS and if one IO is 32kB then the speed is 102 MB/sec. But naturally it is not so simple, manufacturers do not tell normally on what type of load they have calculated the performance numbers. It is quite different whether we are talking about random access to disk or sequential writes/reads. As rule of thumb you may usually assume the manufacturer figures are from sequential writes/reads. In addition caches (both operating system and the disk controller have caches) do affect. Luckily the net is full of instructions and programs that allow you to estimate the speed of the disk if you want to do more research on the topic and your specific environment.

The crude first estimate is to compare the iostat numbers with the speed as published by the manufacturer. If you are over that figure, the disk is having a bad day. If you have a good safety margin and still mpstat reports a lot of io-waits, the there could be some program that is doing random access to disk like there is no tomorrow.

[root@soaserver ~]# iostat 2 2
Linux 2.6.9-55.0.0.0.2.ELsmp (soaserver) 10/14/2012
avg-cpu: %user %nice %sys %iowait %idle
1.64 0.00 0.65 0.07 97.64
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 4.83 13.98 150.48 42224192 454375484
sda1 0.00 0.00 0.00 948 4
sda2 19.10 13.98 150.48 42222140 454375416
dm-0 18.95 13.63 149.67 41147754 451931448
dm-1 0.15 0.36 0.81 1073808 2444032

avg-cpu: %user %nice %sys %iowait %idle
1.00 0.00 1.25 0.00 97.75

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 4.02 0.00 144.72 0 288
sda1 0.00 0.00 0.00 0 0
sda2 18.09 0.00 144.72 0 288
dm-0 18.09 0.00 144.72 0 288
dm-1 0.00 0.00 0.00 0 0