perf trace 指令实现监控系统用和系统事件
perf trace 介绍
perf trace 指令的用途主要是用于对Linux系统性能进行分析和调试。它基于Linux性能计数器(Performance Counters for Linux,PCL),能够监控和记录系统调用和其他系统事件,提供关于硬件事件和软件事件的信息。
perf trace 可以执行与 strace 工具类似的功能,即监控指定线程或进程使用的系统调用,以及该应用收到的所有信号。这使得开发者能够深入了解进程与操作系统的交互过程,从而诊断性能问题或调试应用程序。
perf trace 还可以与其他 perf 子命令结合使用,如 perf record 和 perf report,以收集和分析性能数据。例如,可以先使用 perf record 命令记录性能数据,然后使用 perf trace 命令对记录的数据进行追踪和分析,最后使用 perf report 命令生成详细的性能报告。
常用参数:
-a 从所有 CPU 上做性能分析
-p 针对指定的 PID 做性能分析,多个 PID 之间使用逗号分隔
-t 针对指令的线程多性能分析
-e 采样指定的事件,支持使用 *,如果事件前面有 ! 表示除了该时间之外的事件。
-i 指定 perf trace 分析指定的 perf.data (perf record 指令生成)文件
-T 默认情况下,按照相对时间 展示结果;可以切换为按照绝对时间展示结果
--failure 仅仅展示失败的系统调用,既系统调用的返回结果 <0
--duration 指令结果仅展示事件运行周期大于 N.M 毫秒的系统调用。N.M 是参数值;可用于分析
实例:查看某个进程中运行较慢的系统调用
perf trace -p $pid --duration 50
命令输出效果:
#perf trace -p 120884 --duration 1000
0.127 (1001.022 ms): gunicorn/120884 epoll_wait(epfd: 6<anon_inode:[eventpoll]>, events: 0x7fc7d4372db0, maxevents: 8, timeout: 1000) = 0
1001.291 (1000.513 ms): gunicorn/120884 epoll_wait(epfd: 6<anon_inode:[eventpoll]>, events: 0x7fc7d4372db0, maxevents: 8, timeout: 1000) = 0
2994.120 (1001.025 ms): gunicorn/120884 epoll_wait(epfd: 6<anon_inode:[eventpoll]>, events: 0x7fc7d46807b0, maxevents: 5, timeout: 1000) = 0
3995.361 (1000.444 ms): gunicorn/120884 epoll_wait(epfd: 6<anon_inode:[eventpoll]>, events: 0x7fc7d460bb10, maxevents: 8, timeout: 1000) = 0
4995.949 (1000.256 ms): gunicorn/120884 epoll_wait(epfd: 6<anon_inode:[eventpoll]>, events: 0x7fc7d460bb10, maxevents: 8, timeout: 1000) = 0
5996.405 (1001.021 ms): gunicorn/120884 epoll_wait(epfd: 6<anon_inode:[eventpoll]>, events: 0x7fc7d460bb10, maxevents: 8, timeout: 1000) = 0
2994.457 (4935.005 ms): gunicorn/121886 futex(uaddr: 0x558536292a50, op: WAIT_BITSET|PRIV|CLKRT, val3: MATCH_ANY) = 0
2999.874 (4929.782 ms): gunicorn/122002 futex(uaddr: 0x558536292a50, op: WAIT_BITSET|PRIV|CLKRT, val3: MATCH_ANY) = 0
3000.428 (4941.717 ms): gunicorn/121887 futex(uaddr: 0x558536292a50, op: WAIT_BITSET|PRIV|CLKRT, val3: MATCH_ANY) = 0
3000.485 (4941.763 ms): gunicorn/121812 futex(uaddr: 0x558536292a50, op: WAIT_BITSET|PRIV|CLKRT, val3: MATCH_ANY) = 0
3500.575 (4999.996 ms): gunicorn/121269 futex(uaddr: 0x7fc7e18efeac, op: WAIT|PRIV ) = 0
7999.688 (1001.020 ms): gunicorn/120884 epoll_wait(epfd: 6<anon_inode:[eventpoll]>, events: 0x7fc7d43fb818, maxevents: 7, timeout: 1000) = 0
9000.915 (1001.023 ms): gunicorn/120884 epoll_wait(epfd: 6<anon_inode:[eventpoll]>, events: 0x7fc7d4372b10, maxevents: 8, timeout: 1000) = 0
10002.090 (1001.024 ms): gunicorn/120884 epoll_wait(epfd: 6<anon_inode:[eventpoll]>, events: 0x7fc7d4372b10, maxevents: 8, timeout: 1000) = 0
11003.292 (1001.022 ms): gunicorn/120884 epoll_wait(epfd: 6<anon_inode:[eventpoll]>, events: 0x7fc7d4372b10, maxevents: 8, timeout: 1000) = 0
7999.198 (4955.722 ms): gunicorn/121887 futex(uaddr: 0x558536292a50, op: WAIT_BITSET|PRIV|CLKRT, val3: MATCH_ANY) = 0
7999.364 (4955.724 ms): gunicorn/121812 futex(uaddr: 0x558536292a50, op: WAIT_BITSET|PRIV|CLKRT, val3: MATCH_ANY) = 0
7999.699 (4963.423 ms): gunicorn/122002 futex(uaddr: 0x558536292a50, op: WAIT_BITSET|PRIV|CLKRT, val3: MATCH_ANY) = 0
8001.920 (4961.277 ms): gunicorn/121886 futex(uaddr: 0x558536292a50, op: WAIT_BITSET|PRIV|CLKRT, val3: MATCH_ANY) = 0
括号内的时间为系统调用运行的时间,结果中的这些系统调用的运行时间基本都超出了 100 ms
使用 perf trace 跟踪 ping 包
#perf trace --no-syscalls --event 'net:*' ping 127.0.0.1 -c1 > /dev/null
0.000 net:net_dev_queue:dev=lo skbaddr=0xffff8940561f0900 len=98
0.010 net:net_dev_start_xmit:dev=lo queue_mapping=0 skbaddr=0xffff8940561f0900 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
0.014 net:netif_rx_entry:dev=lo napi_id=0x49 queue_mapping=0 skbaddr=0xffff8940561f0900 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 hash=0x00000000 l4_hash=0 len=84 data_len=0 truesize=768 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0
0.018 net:netif_rx:dev=lo skbaddr=0xffff8940561f0900 len=84
0.022 net:net_dev_xmit:dev=lo skbaddr=0xffff8940561f0900 len=98 rc=0
0.027 net:netif_receive_skb:dev=lo skbaddr=0xffff8940561f0900 len=84
0.038 net:net_dev_queue:dev=lo skbaddr=0xffff8940561f1600 len=98
0.042 net:net_dev_start_xmit:dev=lo queue_mapping=0 skbaddr=0xffff8940561f1600 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 len=98 data_len=0 network_offset=14 transport_offset_valid=1 transport_offset=34 tx_flags=0 gso_size=0 gso_segs=0 gso_type=0
0.045 net:netif_rx_entry:dev=lo napi_id=0x49 queue_mapping=0 skbaddr=0xffff8940561f1600 vlan_tagged=0 vlan_proto=0x0000 vlan_tci=0x0000 protocol=0x0800 ip_summed=0 hash=0x00000000 l4_hash=0 len=84 data_len=0 truesize=768 mac_header_valid=1 mac_header=-14 nr_frags=0 gso_size=0 gso_type=0
0.049 net:netif_rx:dev=lo skbaddr=0xffff8940561f1600 len=84
0.052 net:net_dev_xmit:dev=lo skbaddr=0xffff8940561f1600 len=98 rc=0
0.056 net:netif_receive_skb:dev=lo skbaddr=0xffff8940561f1600 len=84
选择 perf trace 的跟踪点
我们可以使用 perf list 查看 perf trace 支持的跟踪点
#perf list "net:*"
List of pre-defined events (to be used in -e):
net:napi_gro_frags_entry [Tracepoint event]
net:napi_gro_receive_entry [Tracepoint event]
net:net_dev_queue [Tracepoint event]
net:net_dev_start_xmit [Tracepoint event]
net:net_dev_xmit [Tracepoint event]
net:netif_receive_skb [Tracepoint event]
net:netif_receive_skb_entry [Tracepoint event]
net:netif_receive_skb_list_entry [Tracepoint event]
net:netif_rx [Tracepoint event]
net:netif_rx_entry [Tracepoint event]
net:netif_rx_ni_entry [Tracepoint event]
除了 tracepoint event 类型事件外,还支持其它类型的事件,如 Hardware event 和 Software event
参考:
https://blog.csdn.net/hao_wujing/article/details/145101706?spm=1001.2101.3001.6650.2&utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7EYuanLiJiHua%7EPosition-2-145101706-blog-111839854.235%5Ev43%5Epc_blog_bottom_relevance_base4&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7EYuanLiJiHua%7EPosition-2-145101706-blog-111839854.235%5Ev43%5Epc_blog_bottom_relevance_base4&utm_relevant_index=5

浙公网安备 33010602011771号