收到IO占用高告警
系统信息：Linux version 2.6.32-696.18.7.1.el6.ucloud.x86_64 (root@59c188f3c79d)(gcc version 4.4.6 0305
(Red Hat 4.4.6-5) (GCC) ) #1 SMP Fri Jan 5 16:48:58 CST
1、到机器上看到io使用率忽高，同时iowait也高
$ iostat -d -x 1
Linux 2.6.32-696.18.7.1.el6.ucloud.x86_64 (bd-prod-acc-web0) 07/08/ _x86_64_ (2 CPU)
Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 1.17 0.00 0.88 0.06 16.40 18.58 0.01 16.88 0.90 16.92 4.06 0.36
vdb 0.00 0.86 0.00 1.03 0.00 15.14 14.68 0.01 11.48 1.79 11.48 4.56 0.47
Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 0.00 0.00 1.00 0.00 72.00 72.00 1.00 2387.00 0.00 2387.001000.00 100.00
vdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.06 0.00 0.00 0.00 0.00 100.00
Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00100.00
vdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00 0.00100.00
Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 0.00 0.00 1.00 0.00 8.00 8.00 0.51 2258.00 0.00 2258.00 513.00 51.30
vdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00 0.00100.00
2、找到占用高的进程
$ sudo iotop
Total DISK READ: 0.00 B/s | Total DISK WRITE: 51.21 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
810 be/3 root 0.00 B/s 0.00 B/s 0.00 %99.99 % [jbd2/vdb-8] #这里看到是系统进程
2614 be/4 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % sadc -F -L -S DISK -S DISK 1 1 -
412 be/3 root 0.00 B/s 3.94 K/s 0.00 % 99.99 % [jbd2/vda1-8]
3、jdb2进程占用IO资源较高，jdb2进程是文件系统ext4的写日志进程，占用高是内核一个bug
4、（1）升级内核
$ sudo yum update kernel
或
(2)通过修改文件系统的提交次数来降低
commit=60
或（3）重启
Wait IO problem can be caused by several reasons, the basic road-map to find out is which process is "eating" your CPU first and then determine why. The main cause are those background processes with "D" status code which means "Uninterruptiable sleep". But the those processes with "D+" which means "Uninterruptible sleep foreground process" will generally not cause the serious problem as those background processes. In this example, the cause of Wait IO is the File System Journal, so the configuration of file system is the cause of the problem.
ByQuentin Sherman Xue- CEO (/en/interesting-articles/126-linux-wait-io-problem)
系统io问题的一般思路：
查看系统总体IO性能------->找到占用高的进程-------->查看进程对应的IO操作-------->结合场景分析，根据场景优化
