[Done] Linux System Performance Monitor

[ Monitor - Ali Cloud ]

https://cms.console.aliyun.com/#/cloud/dashboard

[ Monitor - top ]

eric@eric-pc:~$ top

H: toggle between process view and thread view.
1: toggle to display usage (%) of each CPU.

Z,B: change color and bold
x,y: Toggle highlights: 'x' sort field; 'y' running tasks

f: change display fields
<,>: to change the sort column
c: show/not-show the command line

d: set update interval

w: save the config
i: not show idle processes
k: to kill process

[ Monitor - iotop ]

# iotop

[ Key System Elements ]

* CPU, Memory, Storage, I/O, Network

[ KPI -  Internet]

* ping : http://ping.chinaz.com/

* HTTP GET: http://ping.chinaz.com

* HTTP Upload

* TCP Connection

[ KPI -  Load Average ]

eric@eric-pc:~$ uptime
10:02:46 up 26 min,  1 user,  load average: 0.28, 0.25, 0.34

* The most important KPI

* How average, during a period of time (1/5/15 minutes), how many processes are using CPU or waiting CPU or doing I/O. 

* Ideally, the load average should <NumOfCPU*70%.

[ KPI -  CPU Utilization ]

eric@eric-pc:~$ mpstat -P ALL 1

Linux 4.4.0-36-generic (eric-pc) 2016年09月18日 _x86_64_ (4 CPU)

12时07分02秒 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
12时07分02秒 all 2.14 0.07 0.65 1.97 0.00 0.01 0.00 0.00 0.00 95.17
12时07分02秒 0 2.31 0.14 0.61 1.74 0.00 0.00 0.00 0.00 0.00 95.20
12时07分02秒 1 2.26 0.07 0.62 1.65 0.00 0.01 0.00 0.00 0.00 95.38
12时07分02秒 2 1.99 0.02 0.87 2.30 0.00 0.00 0.00 0.00 0.00 94.82
12时07分02秒 3 2.01 0.04 0.49 2.18 0.00 0.01 0.00 0.00 0.00 95.27

...

eric@eric-pc:~$ vmstat -t -S M 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CST
0 0 0 9654 41 992 0 0 40 30 101 502 2 1 95 2 0 2016-09-18 12:03:12

...

r: The number of runnable processes (running or waiting for run time).
b: The number of processes in uninterruptible (normally I/O) sleep.

To Investigate Details

/usr/src$ mpstat -A

[ KPI -  Memory ]

eric@eric-pc:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:          11930        1211        8793         305        1925       10067
Swap:          7811           0        7811

To free up memory (like cache/buffer/inactive)

root@eric-pc:~# sync && echo 3 > /proc/sys/vm/drop_caches

Detailed Info About Memory Usage

root@eric-pc:~# cat /proc/meminfo
eric@eric-pc:~$ vmstat -s

[ KPI -  Storage ]

eric@eric-pc:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            5.9G     0  5.9G   0% /dev
tmpfs           1.2G  9.4M  1.2G   1% /run
/dev/sda9       227G   44G  172G  21% /
tmpfs           5.9G   13M  5.9G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           5.9G     0  5.9G   0% /sys/fs/cgroup
tmpfs           1.2G   80K  1.2G   1% /run/user/1000

eric@eric-pc:~$ df -hi
Filesystem     Inodes IUsed IFree IUse% Mounted on
udev             1.5M   549  1.5M    1% /dev
tmpfs            1.5M   772  1.5M    1% /run
/dev/sda9         15M  625K   14M    5% /
tmpfs            1.5M    38  1.5M    1% /dev/shm
tmpfs            1.5M     5  1.5M    1% /run/lock
tmpfs            1.5M    16  1.5M    1% /sys/fs/cgroup
tmpfs            1.5M    36  1.5M    1% /run/user/1000

[ KPI -  I/O ]

eric@eric-pc:~$ iostat -xzp 1
Linux 4.4.0-36-generic (eric-pc) 2016年09月18日 _x86_64_ (4 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
2.10 0.06 0.64 1.81 0.00 95.40

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 3.93 4.75 4.88 2.54 137.68 103.29 64.94 0.20 27.41 21.72 38.36 4.40 3.26
sda1 0.00 0.00 0.01 0.00 0.20 0.00 77.63 0.00 72.30 72.30 0.00 64.07 0.03
sda2 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.00 170.00 170.00 0.00 170.00 0.00
sda5 0.00 0.00 0.00 0.00 0.20 0.00 86.33 0.00 78.69 78.69 0.00 67.27 0.03
sda6 0.00 0.00 0.01 0.00 0.21 0.00 82.59 0.00 75.78 75.78 0.00 61.11 0.03
sda7 0.00 0.00 0.00 0.00 0.20 0.00 86.33 0.00 75.10 75.10 0.00 66.69 0.03
sda8 0.00 0.00 0.00 0.00 0.20 0.00 91.04 0.00 74.78 74.78 0.00 67.57 0.03
sda9 3.93 4.75 4.84 2.44 136.37 103.29 65.82 0.20 27.20 21.46 38.60 4.07 2.96

eric@eric-pc:~$ vmstat -d

[ KPI -  Network ]

eric@eric-pc:~$ sar [-qu] -n DEV,EDEV 1
Linux 4.4.0-36-generic (eric-pc) 	2016年09月18日 	_x86_64_	(4 CPU)

12时34分03秒     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
12时34分04秒    enp2s0      7.00      2.00      0.56      0.14      0.00      0.00      3.00      0.00
12时34分04秒        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12时34分04秒   docker0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

12时35分31秒 IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s

12时35分32秒 enp2s0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12时35分32秒 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12时35分32秒 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

...

eric@eric-pc:~$ sar [-qu] -n SOCK 1

eric@eric-pc:~$ sar [-qu] -n TCP,ETCP 1

eric@eric-pc:~$ sar [-qu] -n UDP 1

eric@eric-pc:~$ sar [-qu] -n IP,EIP 1

eric@eric-pc:~$ sar [-qu] -n ICPM,EICMP 1

root@eric-pc:~# netstat -tanp : to list all the TCP sockets
root@eric-pc:~# netstat -tlnp : to list all the TCP sockets which are in listening state
root@eric-pc:~# netstat -tnp : to list all the TCP sockets which are in connected state
root@eric-pc:~# netstat -uanp : to list all the UDP sockets
root@eric-pc:~# netstat -ulnp : to list all the UDP sockets which are in listening state
root@eric-pc:~# netstat -unp : to list all the UDP sockets which are in connected state
root@eric-pc:~# netstat -wanp : to list all the RAW sockets
root@eric-pc:~# netstat -wlnp : to list all the RAW sockets which are in lisening state
root@eric-pc:~# netstat -wnp : to list all the RAW sockets which are in connected state

root@eric-pc:~# netstat -xanp : to list all the Unix sockets
root@eric-pc:~# netstat -xlnp : to list all the Unix sockets which are in lisening state
root@eric-pc:~# netstat -xnp : to list all the Unix sockets which are in connected state

[ KPI -  Process Status ]

eric@eric-pc:~$ pidstat -t 1
Linux 4.4.0-36-generic (eric-pc) 	2016年09月18日 	_x86_64_	(4 CPU)

13时50分00秒   UID       PID    %usr %system  %guest    %CPU   CPU  Command
13时50分01秒     0     10755    0.00    1.00    0.00    1.00     1  kworker/u8:0

eric@eric-pc:~$ pidstat -t -d 1
Linux 4.4.0-36-generic (eric-pc) 	2016年09月18日 	_x86_64_	(4 CPU)

13时50分15秒   UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
13时50分16秒  1000      4528      0.00      4.00      0.00       0  chrome

eric@eric-pc:~$ ps -elF
-e : to show all the processes
-l and -F : to show detailed information
--sort -c : short by cpu usage
--sort -time : short by total used cpu time
-H : to show it in the process tree style
ps -e -o pid,pri,ni,nlwp,s,%cpu,time,%mem,vsz,rss,wchan,args --sort -c: to show all processes with selected fields.

PID PPID: PID and Parent PID
PRI NI: priority and nice value
S: process status (sleep, running, etc.)
WCHAN: the name of the kernel function in which the process is sleeping on.
C: the CPU time used divided by the time the process has been running.
TIME: the total CPU time a process has used in its lifetime.
RSS: the physical memory a process has used. Unit KB.
VSZ: the virual memory a process is using. Unit KB.

[ KPI - Logs ]

* dmesg, service logs, etc.

eric@eric-pc:/usr/src$ dmesg | tail
[   46.167219] audit_printk_skb: 39 callbacks suppressed
[   46.167221] audit: type=1400 audit(1474162598.925:25): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="docker-default" pid=3140 comm="apparmor_parser"
[   46.265077] aufs 4.x-rcN-20160111
[   47.315328] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[   47.316634] Bridge firewalling registered
[   47.405546] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   47.872659] ip_tables: (C) 2000-2006 Netfilter Core Team
[   48.918204] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[ 5705.903969] bash (7560): drop_caches: 3
[ 5727.048471] bash (7560): drop_caches: 3