网络监测介绍
网络性能监测有如下几点：
* 检查并确保所有网卡运行在适当的速率下。
* 检查每个网卡的总吞吐量并确保其符合网速。
* 监测网络流量类型以确保适当的流量有适当的优先级。
网络是所有子系统中最难监测的一个，因为网络比较抽象，在监测时有很多在系统可控制之外的因素如延迟，冲突，拥塞和丢包等对监测产生影响。下面将讨论的是以太网、IP、TCP的性能监测。
8.1 以太网配置设定
除非有明确的设定，所有以太网的速度都是自动协商的，这很大程度上是由于历史原因造成的，早些时候一个网络里经常有不同网速和双工模式的网络设备。
大多数企业以太网是100BaseTX或1000BaseTX，可以使用ethtool工具来判断一个系统的网速。
下面的示例中一个拥有100BaseTX网卡的机器工作在10BaseTX下：
# ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised auto-negotiation: Yes
Speed: 10Mb/s
Duplex: Half
Port: MII
PHYAD: 32
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: d
Current message level: 0x00000007 (7)
Link detected: yes
下面将其强制设定为100BaseTX模式：
# ethtool -s eth0 speed 100 duplex full autoneg off
# ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised auto-negotiation: No
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 32
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: pumbg
Wake-on: d
Current message level: 0x00000007 (7)
Link detected: yes
8.2 网络吞吐量监测
监测网络吞吐量最好的办法是在两个系统之间发送流量并统计其延迟和速度。
8.2.0 使用iptraf监测本地吞吐量
iptraf工具可提供以太网卡的吞吐量情况：
# iptraf -d eth0
上面的数据显示被测试系统正以61mbps（7.65M）频率发送数据，相比于100mbps网络这有点低。
8.2.1 使用netperf监测远端吞吐量
与iptraf的动态监测不一样的是netperf使用可控方式测试网络，这一点对测试一个客户端到一个高负载服务器之间的吞吐量很有帮助，netperf工具是以C/S模式运行。
首先需要在服务器上运行netperf服务端：
server# netserver
Starting netserver at port 12865
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC
netperf可以执行多种测试，最基本的是标准测试：
client# netperf -H 192.168.1.215 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.230 (192.168.1.230) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 30.02 89.46
输出显示吞吐量在89mbps左右，服务器和客户端在同一网段。
从一个10跳的54G无线网进行测试只能达到14mbps左右：
client# netperf -H 192.168.1.215 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.215 (192.168.1.215) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 30.10 14.09
从50跳距离局域网测试：
# netperf -H 192.168.1.215 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.215 (192.168.1.215) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 30.64 5.05
从外网测试：
# netperf -H litemail.org -p 1500 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
litemail.org (72.249.104.148) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 31.58 0.93
通过VPN测试：
# netperf -H 10.0.1.129 -l 30
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
10.0.1.129 (10.0.1.129) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 31.99 0.51
另外一个有用的测试模式是测试TCP请求应答速率，其原理是建立一个TCP连接并发送多个请求：
client# netperf -t TCP_RR -H 192.168.1.230 -l 30
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
to 192.168.1.230 (192.168.1.230) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec
16384 87380 1 1 30.00 4453.80
16384 87380
数据显示网络可支持4453左右的psh/ack 每秒，因为发送包的大小只有1k。下面使用2k的请求和32k的应答：
client# netperf -t TCP_RR -H 192.168.1.230 -l 30 -- -r 2048,32768
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
192.168.1.230 (192.168.1.230) port 0 AF_INET
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec
16384 87380 2048 32768 30.00 222.37
16384 87380
可以看到速率已经降到222左右。
8.2.3 使用iperf测试网络性能
iperf与netperf相似在两台机器间进行测试，不同之处在于iperf测试更深入，它只有一个可执行文件既可用于服务端又可用于客户端，默认通信端口是5001.
启动服务端（192.168.1.215）：
server# iperf -s -D
Running Iperf Server as a daemon
The Iperf daemon process ID : 3655
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
一个满负荷无线网中的客户端进行测试：
client# iperf -c 192.168.1.215 -t 60 -i 5
------------------------------------------------------------
Client connecting to 192.168.1.215, TCP port 5001
TCP window size: 25.6 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.224.150 port 51978 connected with
192.168.1.215 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 5.0 sec 6.22 MBytes 10.4 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 5.0-10.0 sec 6.05 MBytes 10.1 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 10.0-15.0 sec 5.55 MBytes 9.32 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 15.0-20.0 sec 5.19 MBytes 8.70 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 20.0-25.0 sec 4.95 MBytes 8.30 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 25.0-30.0 sec 5.21 MBytes 8.74 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 30.0-35.0 sec 2.55 MBytes 4.29 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 35.0-40.0 sec 5.87 MBytes 9.84 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 40.0-45.0 sec 5.69 MBytes 9.54 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 45.0-50.0 sec 5.64 MBytes 9.46 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 50.0-55.0 sec 4.55 MBytes 7.64 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 55.0-60.0 sec 4.47 MBytes 7.50 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 61.9 MBytes 8.66 Mbits/sec
下面是进行UDP丢包率测试：
# iperf -c 192.168.1.215 -b 10M
WARNING: option -b implies udp testing
------------------------------------------------------------
Client connecting to 192.168.1.215, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 107 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.224.150 port 33589 connected with 192.168.1.215 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 11.8 MBytes 9.90 Mbits/sec
[ 3] Sent 8420 datagrams
[ 3] Server Report:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.0 sec 6.50 MBytes 5.45 Mbits/sec 0.480 ms 3784/ 8419 (45%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
10M的数据只传输了一部分，丢包率在45%左右。
8.3 用tcptrace进行特定连接测试
tcptrace工具提供了基于TCP的详细测试信息，基于libpcap文件，它可以监测出有时不太容易抓到的TCP流：
* TCP中继——需要被重新发送的包数量和数据大小
* TCP窗口大小——使用很小的window sizes识别低速度连接
* 连接的总吞吐量
* 连接持续性
8.3.1 案例分析
tcptrace把libpcap文件作为输入，不加任何选项执行此工具可显示出包里所有信息，下面的示例中tcptrace处理一个叫bigstuff的包文件：
# tcptrace bigstuff
1 arg remaining, starting with 'bigstuff'
Ostermann's tcptrace -- version 6.6.7 -- Thu Nov 4, 2004
146108 packets seen, 145992 TCP packets traced
elapsed wallclock time: 0:00:01.634065, 89413 pkts/sec analyzed
trace file elapsed time: 0:09:20.358860
TCP connection info:
1: 192.168.1.60:pcanywherestat - 192.168.1.102:2571 (a2b) 404> 450<
2: 192.168.1.60:3356 - ftp.strongmail.net:21 (c2d) 35> 21<
3: 192.168.1.60:3825 - ftp.strongmail.net:65023 (e2f) 5> 4<
(complete)
4: 192.168.1.102:1339 - 205.188.8.194:5190 (g2h) 6> 6<
5: 192.168.1.102:1490 - cs127.msg.mud.yahoo.com:5050 (i2j) 5> 5<
6: py-in-f111.google.com:993 - 192.168.1.102:3785 (k2l) 13> 14<
。。。 。。。
在上面输出的数据中可以看到每条信息前面都有一个编号，tcptrace最常用的选项是-l和-o，用来指定某个具体的连接。下面的示例：
# tcptrace -l -o1 bigstuff
1 arg remaining, starting with 'bigstuff'
Ostermann's tcptrace -- version 6.6.7 -- Thu Nov 4, 2004
146108 packets seen, 145992 TCP packets traced
elapsed wallclock time: 0:00:00.529361, 276008 pkts/sec analyzed
trace file elapsed time: 0:09:20.358860
TCP connection info:
32 TCP connections traced:
TCP connection 1:
host a: 192.168.1.60:pcanywherestat
host b: 192.168.1.102:2571
complete conn: no (SYNs: 0) (FINs: 0)
first packet: Sun Jul 20 15:58:05.472983 2008
last packet: Sun Jul 20 16:00:04.564716 2008
elapsed time: 0:01:59.091733
total packets: 854
filename: bigstuff
a->b: b->a:
total packets: 404 total packets: 450
ack pkts sent: 404 ack pkts sent: 450
pure acks sent: 13 pure acks sent: 320
sack pkts sent: 0 sack pkts sent: 0
dsack pkts sent: 0 dsack pkts sent: 0
max sack blks/ack: 0 max sack blks/ack: 0
unique bytes sent: 52608 unique bytes sent: 10624
actual data pkts: 391 actual data pkts: 130
actual data bytes: 52608 actual data bytes: 10624
rexmt data pkts: 0 rexmt data pkts: 0
rexmt data bytes: 0 rexmt data bytes: 0
zwnd probe pkts: 0 zwnd probe pkts: 0
zwnd probe bytes: 0 zwnd probe bytes: 0
outoforder pkts: 0 outoforder pkts: 0
pushed data pkts: 391 pushed data pkts: 130
SYN/FIN pkts sent: 0/0 SYN/FIN pkts sent: 0/0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 0 bytes mss requested: 0 bytes
max segm size: 560 bytes max segm size: 176 bytes
min segm size: 48 bytes min segm size: 80 bytes
avg segm size: 134 bytes avg segm size: 81 bytes
max win adv: 19584 bytes max win adv: 65535 bytes
min win adv: 19584 bytes min win adv: 64287 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 19584 bytes avg win adv: 64949 bytes
initial window: 160 bytes initial window: 0 bytes
initial window: 2 pkts initial window: 0 pkts
ttl stream length: NA ttl stream length: NA
missed data: NA missed data: NA
truncated data: 36186 bytes truncated data: 5164 bytes
truncated packets: 391 pkts truncated packets: 130 pkts
data xmit time: 119.092 secs data xmit time: 116.954 secs
idletime max: 441267.1 ms idletime max: 441506.3 ms
throughput: 442 Bps throughput: 89 Bps
8.3.2 案例分析——计算转发百分比
# tcptrace -f'rexmit_segs>100' bigstuff
Output filter: ((c_rexmit_segs>100)OR(s_rexmit_segs>100))
1 arg remaining, starting with 'bigstuff'
Ostermann's tcptrace -- version 6.6.7 -- Thu Nov 4, 2004
146108 packets seen, 145992 TCP packets traced
elapsed wallclock time: 0:00:00.687788, 212431 pkts/sec analyzed
trace file elapsed time: 0:09:20.358860
TCP connection info:
16: ftp.strongmail.net:65014 - 192.168.1.60:2158 (ae2af) 18695> 9817<
# tcptrace -l -o16 bigstuff
arg remaining, starting with 'bigstuff'
Ostermann's tcptrace -- version 6.6.7 -- Thu Nov 4, 2004
146108 packets seen, 145992 TCP packets traced
elapsed wallclock time: 0:00:01.355964, 107752 pkts/sec analyzed
trace file elapsed time: 0:09:20.358860
TCP connection info:
32 TCP connections traced:
================================
TCP connection 16:
host ae: ftp.strongmail.net:65014
host af: 192.168.1.60:2158
complete conn: no (SYNs: 0) (FINs: 1)
first packet: Sun Jul 20 16:04:33.257606 2008
last packet: Sun Jul 20 16:07:22.317987 2008
elapsed time: 0:02:49.060381
total packets: 28512
filename: bigstuff
ae->af: af->ae:
.... ....
unique bytes sent: 25534744 unique bytes sent: 0
actual data pkts: 18695 actual data pkts: 0
actual data bytes: 25556632 actual data bytes: 0
rexmt data pkts: 1605 rexmt data pkts: 0
rexmt data bytes: 2188780 rexmt data bytes: 0
计算方式：
1605/18695* 100 = 8.5%
8.2.3 案例分析——通过时间计算转发
tcptrace工具提供了可以根据不同标尺（协议，端口，时间等）显示数据的模块，其中slice模块可以根据运行时间查看TCP性能。你可以准确地测量出何时发生了大量的包转发并与其他性能数据相结合来判断性能瓶颈之所在。
下面的示例展示了如何使用slice模块：
# tcptrace –xslice bigfile
命令会在当前目录生成一个叫slice.dat的文件，这个文件包含了15秒为间隔的转发信息：
# ls -l slice.dat
-rw-r--r-- 1 root root 3430 Jul 10 22:50 slice.dat
# more slice.dat
date segs bytes rexsegs rexbytes new active
--------------- -------- -------- -------- -------- -------- --------
22:19:41.913288 46 5672 0 0 1 1
22:19:56.913288 131 25688 0 0 0 1
22:20:11.913288 0 0 0 0 0 0
22:20:26.913288 5975 4871128 0 0 0 1
22:20:41.913288 31049 25307256 0 0 0 1
22:20:56.913288 23077 19123956 40 59452 0 1
22:21:11.913288 26357 21624373 5 7500 0 1
22:21:26.913288 20975 17248491 3 4500 12 13
22:21:41.913288 24234 19849503 10 15000 3 5
22:21:56.913288 27090 22269230 36 53999 0 2
22:22:11.913288 22295 18315923 9 12856 0 2
22:22:26.913288 8858 7304603 3 4500 0 1
