Linux??????????60 ??
?????Brendan Gregg ???????[ 2016/9/12 10:53:02 ] ??????????????? Linux
??????????????????? Brendan Gregg?? ???? Oracle/Linux????????????????????
??????????????????????????Linux??????????????????????????????
??????Netflix?????????????EC2 Linux????????з?????????????????????????????????????а????????????Atlas??????????????????Vector???????Щ??????????????????????????????????????Linux??????????Щ?????Linux????????????????
???????????????Netflix Performance Engineering??????t???????Linux????????й???????????????????60?????????￡?
?????????60??……
????????????10?????????????60????????????????????????????????д?????????????????????????????????????????????????????“????”???????????????????????????????????е???????????????????
????uptime
????dmesg | tail
????vmstat 1
????mpstat -P ALL 1
????pidstat 1
????iostat -xz 1
????free -m
????sar -n DEV 1
????sar -n TCP??ETCP 1
????top
??????Щ??????????sysstat????????????mpstat?? pidstat?? iostat??sar???e????????????sysstat????? ??Щ?????????????????????????USE??????????????????λ??????????????????????????????????CPU????棬???????????????????????????????????????????????μ?????????????????????????????????????????????????????USE??????????????????????????utilization?????????saturation????????error????
?????????????????????????????????????Щ?????????????????????????????????man page??
????1. uptime
????$ uptime
????23:51:26 up 21:31?? 1 user?? load average: 30.02?? 26.43?? 19.02
????????????????????е???????????????????????????????????????????Linux?????Щ??????????????????????CPU??????????????????uninterruptible I/O??????????I/O???????????????????????????????????????????????????????д??????????Э????????????????????????檔
???????????????????1?????5?????15?????????????????????????????????????????????????????????????????????????????????????????????1??????????????15?????????????????????????????
????????????????У???????????????????????1????????30????15?????????19????????????????????????п??????CPU???????????е??3?????4??????——vmstat??mpstat——???????????????????????????
????2. dmesg | tail
????$ dmesg | tail
????[1880957.563150] perl invoked oom-killer: gfp_mask=0x280da?? order=0?? oom_score_adj=0
????[...]
????[1880957.563400] Out of memory: Kill process 18694 (perl) score 246 or sacrifice child
????[1880957.563408] Killed process 18694 (perl) total-vm:1972392kB?? anon-rss:1953348kB?? file-rss:0kB
????[2320864.954447] TCP: Possible SYN flooding on port 7001. Dropping request. Check SNMP counters.
?????????????????μ?10?????????????е????????????????????????????????????????????????????????????????о??????ж???TCP???????檔
????????????????????dmesg?????????
????3. vmstat 1
????$ vmstat 1
????procs ---------memory---------- ---swap-- -----io---- -system-- ------cpu-----
????r b swpd free buff cache si so bi bo in cs us sy id wa st
????34 0 0 200889792 73708 591828 0 0 0 5 6 10 96 1 3 0 0
????32 0 0 200889920 73708 591860 0 0 0 592 13284 4282 98 1 1 0 0
????32 0 0 200890112 73708 591860 0 0 0 0 9501 2154 99 1 0 0 0
????32 0 0 200889568 73712 591856 0 0 0 48 11900 2459 99 0 0 0 0
????32 0 0 200890208 73712 591860 0 0 0 0 15898 4840 98 1 1 0 0
????^C
????vmstat(8)????“virtual memory stat”????????????????????BSD?????У??????????????????????????????????????????????????????
??????????1???vmstat???????????????????????????????????????????????vmstat?????????????????????????????????????????????????????????????????????У????????漸?е??????
?????????????У?
????r?????CPU?????????????????????ж?CPU?????????????????I/O?????????r?????CPU?????ζ??????
????free?????е????????????????????????ж???λ???????????????????????????????7??????free -m???????????????????????????
????si??so??Swap-ins??Swap-outs???????????????ζ??????????????????y?????????????
????us??sy??id??wa??st????????????CPU????e????????????user time??system time?????????????????idle??wait I/O??steal time?????????????????????????Xen?????豸??????isolated driver domain???????????????
??????????us??sy??????????????CPU??????μ????????????????wait I/O??ζ??????????????????????????????CPU????У???????????????I/O???????????wait I/O????CPU???е??????????????????????CPU???е???????
????I/O?????????????????????????20%????????????????y?????????????????I/O?????????
????????????????У?CPU????????仨?????????棬??ζ?????????????????????????????CPU??????????90%?????????????????????“r”?У?????????????
????4. mpstat -P ALL 1
????$ mpstat -P ALL 1
????Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
????07:38:49 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
????07:38:50 PM all 98.47 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 0.78
????07:38:50 PM 0 96.04 0.00 2.97 0.00 0.00 0.00 0.00 0.00 0.00 0.99
????07:38:50 PM 1 97.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00
????07:38:50 PM 2 98.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00
????07:38:50 PM 3 96.97 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.03
????[...]
?????????????????CPU???????e??????????????????CPU???????????????????????μ??CPU??????ζ???????????е?????????????
????5. pidstat 1
????$ pidstat 1
????Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
????07:41:02 PM UID PID %usr %system %guest %CPU CPU Command
????07:41:03 PM 0 9 0.00 0.94 0.00 0.94 1 rcuos/0
????07:41:03 PM 0 4214 5.66 5.66 0.00 11.32 15 mesos-slave
????07:41:03 PM 0 4354 0.94 0.94 0.00 1.89 8 java
????07:41:03 PM 0 6521 1596.23 1.89 0.00 1598.11 27 java
????07:41:03 PM 0 6564 1571.70 7.55 0.00 1579.25 28 java
????07:41:03 PM 60004 60154 0.94 4.72 0.00 5.66 9 pidstat
????07:41:03 PM UID PID %usr %system %guest %CPU CPU Command
????07:41:04 PM 0 4214 6.00 2.00 0.00 8.00 15 mesos-slave
????07:41:04 PM 0 6521 1590.00 1.00 0.00 1591.00 27 java
????07:41:04 PM 0 6564 1573.00 10.00 0.00 1583.00 28 java
????07:41:04 PM 108 6718 1.00 0.00 0.00 1.00 0 snmp-pass
????07:41:04 PM 60004 60154 1.00 4.00 0.00 5.00 9 pidstat
????^C
????pidstat???????top??????top?????????????????????pidstat???????????????????????檔?????????????????????????????????????????????????????鱨???С?
??????????????????CPU?????????????java???????%CPU?????????CPU??????????????1591%??ζ??java?????????????16??CPU??
????6. iostat -xz 1
????$ iostat -xz 1
????Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
????avg-cpu: %user %nice %system %iowait %steal %idle
????73.96 0.00 3.73 0.03 0.06 22.21
????Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
????xvda 0.00 0.23 0.21 0.18 4.52 2.08 34.37 0.00 9.98 13.80 5.42 2.44 0.09
????xvdb 0.01 0.00 1.02 8.94 127.97 598.53 145.79 0.00 0.43 1.78 0.28 0.25 0.25
????xvdc 0.01 0.00 1.02 8.86 127.79 595.94 146.50 0.00 0.45 1.82 0.30 0.27 0.26
????dm-0 0.00 0.00 0.69 2.32 10.47 31.69 28.01 0.01 3.23 0.71 3.98 0.13 0.04
????dm-1 0.00 0.00 0.00 0.94 0.01 3.78 8.00 0.33 345.84 0.04 346.81 0.01 0.00
????dm-2 0.00 0.00 0.09 0.07 1.35 0.36 22.50 0.00 2.55 0.23 5.62 1.78 0.03
????[...]
????^C
??????????????ū????豸?????????????????????????????????????????????
????r/s??w/s??rkB/s??wkB/s???????????豸????????д??????????KB????д??KB???????????????????????????????????????????????????????
????await??I/O???????????????λ???????????I/O???????????????????????м??????????????????????????????????????ζ???豸????????????豸????????
????avgqu-sz????????豸?????????????????1????豸??????????????????Щ?豸??????д????????????????????????????豸??
????%util???豸????????????????豸????????????????????????λ??????60%?????????????????????await?п??????????????????豸???????????????ζ??????
???????????洢?豸?????????????????????????豸?????????????????ζ??I/O???
???????μ??????disk I/O????????????????????I/O????????????????????read-ahead????д???壨buffering for writes??????????????????????????????
????7. free -m
????$ free -m
????total used free shared buffers cached
????Mem: 245998 24545 221453 83 59 541
????-/+ buffers/cache: 23944 222053
????Swap: 0 0 0
?????????????????
????buffers????????豸I/O???????????
????cached??????????????????
???????????????0?????????????????I/O?????????iostat???????????????????????????????????????ж??кü?M???
???????????У?-/+ buffers/cache???????????????????Щ??Linux???????ò??????????????棬?????????????????????·?????????????????????????????????????????棬?????????????????????????????????????? ???????????????????? linuxatemyram??
???????????Linux??????ZFS?????????????Щ??????????????????????????????ZFS???????????????治????free -m?????????????????ж????????????????????涼????ZFS??????????
????8. sar -n DEV 1
????$ sar -n DEV 1
????Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
????12:16:48 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
????12:16:49 AM eth0 18763.00 5032.00 20686.42 478.30 0.00 0.00 0.00 0.00
????12:16:49 AM lo 14.00 14.00 1.36 1.36 0.00 0.00 0.00 0.00
????12:16:49 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
????12:16:49 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
????12:16:50 AM eth0 19763.00 5101.00 21999.10 482.56 0.00 0.00 0.00 0.00
????12:16:50 AM lo 20.00 20.00 3.25 3.25 0.00 0.00 0.00 0.00
????12:16:50 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
????^C
??????????????????????????????????????rxkB/s??txkB/s???????????????????????????У?eth0???????????22Mbytes/s?????176Mbits/sec???????1Gbit/sec??
???????????汾?л?????%ifutil????豸???????????????????е?????????????????????Brendan??nicstat?????????????nicstat??sar?????????????????????????????????????????????0.00????
????9. sar -n TCP??ETCP 1
????$ sar -n TCP??ETCP 1
????Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
????12:17:19 AM active/s passive/s iseg/s oseg/s
????12:17:20 AM 1.00 0.00 10233.00 18846.00
????12:17:19 AM atmptf/s estres/s retrans/s isegerr/s orsts/s
????12:17:20 AM 0.00 0.00 0.00 0.00 0.00
????12:17:20 AM active/s passive/s iseg/s oseg/s
????12:17:21 AM 1.00 0.00 8359.00 6039.00
????12:17:20 AM atmptf/s estres/s retrans/s isegerr/s orsts/s
????12:17:21 AM 0.00 0.00 0.00 0.00 0.00
????^C
???????????????Щ???TCP???????????а?????
????active/s?????????????TCP????????????concept()???????
????passive/s????????????TCP????????????accept()???????
????retrans/s?????TCP???????
????????????????active???????????????passive?????????????????????????????????????????????????????????????????????????????????????????????????磬?????localhost??localhost???????
?????????????????????????????????????????????磨???繫?????????????п???????????????????????????????????????У?????????????μ?TCP?????
????10. top
????$ top
????top - 00:15:40 up 21:56?? 1 user?? load average: 31.09?? 29.87?? 29.92
????Tasks: 871 total?? 1 running?? 868 sleeping?? 0 stopped?? 2 zombie
????%Cpu(s): 96.8 us?? 0.4 sy?? 0.0 ni?? 2.7 id?? 0.1 wa?? 0.0 hi?? 0.0 si?? 0.0 st
????KiB Mem: 25190241+total?? 24921688 used?? 22698073+free?? 60448 buffers
????KiB Swap: 0 total?? 0 used?? 0 free. 554208 cached Mem
????PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
????20248 root 20 0 0.227t 0.012t 18748 S 3090 5.2 29812:58 java
????4213 root 20 0 2722544 64640 44232 S 23.5 0.0 233:35.37 mesos-slave
????66128 titancl+ 20 0 24344 2332 1172 R 1.0 0.0 0:00.07 top
????5235 root 20 0 38.227g 547004 49996 S 0.7 0.2 2:02.74 java
????4299 root 20 0 20.015g 2.682g 16836 S 0.3 1.1 33:14.42 java
????1 root 20 0 33620 2920 1496 S 0.0 0.0 0:03.82 init
????2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
????3 root 20 0 0 0 0 S 0.0 0.0 0:05.35 ksoftirqd/0
????5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
????6 root 20 0 0 0 0 S 0.0 0.0 0:06.94 kworker/u256:0
????8 root 20 0 0 0 0 S 0.0 0.0 2:38.05 rcu_sched
????top????????????????????????????????????????????????????????????????????Щ????
????????????????????????????????????top???????????????vmstat??pidstat????????????????????????????????????????????????????????????top???????Ctrl-s?????Ctrl-q????????????Щ????????????????μ?????С?
???????????…
?????к??????????????????????????????Brendan??2015??Velocity??????Linux Performance Tools tutorial???г?????40????????????????????????????????????????????profile??????????tracing????????檔
??????
???·???
??????????????????
2023/3/23 14:23:39???д?ò??????????
2023/3/22 16:17:39????????????????????Щ??
2022/6/14 16:14:27??????????????????????????
2021/10/18 15:37:44???????????????
2021/9/17 15:19:29???·???????·
2021/9/14 15:42:25?????????????
2021/5/28 17:25:47??????APP??????????
2021/5/8 17:01:11
sales@spasvo.com