Troubleshoot performance part 3 - Using mpstat for CPU analysis

Mpstat for CPU
If the problem is at CPU, mpstat (multiple processor statistics) gives a cleaner report. Especially mpstat reports separately information about each CPU separately so you can see if a single CPU is overloaded while others are free. The syntax for mpstat is:

mpstat <interval> <count>

Interval is the time in seconds between printing out a line of statistics. Count is the number of lines of output you want.

The report generated by the mpstat command has the following format:

• CPU: Processor number, starts with 0. The keyword all indicates that statistics are calculated as averages among all processors.
• %user: Percentage of CPU utilization that occurred while executing at the user level (application).
• %nice: Percentage of CPU utilization that occurred while executing at the user level with nice priority.
• %sys: Percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time spent servicing interrupts or softirqs.
• %iowait: Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
• %irq: Percentage of time spent by the CPU or CPUs to service interrupts.
• %soft: Percentage of time spent by the CPU or CPUs to service softirqs. A softirq (software interrupt) is one of up to 32 enumerated software interrupts which can run on multiple CPUs at once.
• %steal: Percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
• %idle: Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
• intr/s: Total number of interrupts received per second by the CPU or CPUs.

[root@soaserver ~]# mpstat 2 3
Linux 2.6.9-55.0.0.0.2.ELsmp (soaserver) 10/14/2012
04:03:03 PM CPU %user %nice %system %iowait %irq %soft %idle intr/s
04:03:05 PM all 0.25 0.00 0.25 0.00 0.25 0.00 99.25 1129.65
04:03:07 PM all 0.25 0.00 0.00 0.00 0.00 0.00 99.75 1114.50
04:03:09 PM all 3.75 0.00 1.25 0.25 0.00 0.00 94.75 1124.00
Average: all 1.42 0.00 0.50 0.08 0.08 0.00 97.92 1122.70

On SMP machines a processor that does not have any activity at all is a disabled (offline) processor.

Below are some general tips, which you can use while interpreting the output -

If %user is very high then your application is consuming the CPUs and it is being overburdened.

If %sys is high then your server is burnened by the system (kernel) calls.

If %iowait constantly a non-zero number, then you may have some disk I/O contention. It is recommended to check the “Time spent waiting for IO (wa)” of vmstat to see whether there is any waiting on disk storage subsystem. You should also consult the iostat output.

Links:

http://en.wikipedia.org/wiki/Mpstat
http://linuxcommand.org/man_pages/mpstat1.html