什么叫精通linux?

一、 内核与底层架构(Kernel & Core Subsystems)

内核核心与调度机制:Linux Scheduler, CFS, EEVDF, Kernel Preemption, Interrupt Handling, Context Switching, CPU Affinity, NUMA, OOM Killer, Kernel Parameters, sysctl
内存与存储管理:Virtual Memory, Paging, Demand Paging, Memory Mapping, Page Cache, Buffer Cache, Slab Allocator, Buddy Allocator, Huge Pages, Transparent Huge Pages, Swap, zram, zswap
同步与并发机制:Kernel Synchronization, RCU, Spinlocks, Mutexes, Futexes, File Locking, flock, fcntl
硬件接口与驱动:Device Drivers, Block Devices, Character Devices, Network Devices, DMA, IOMMU, ACPI, PCIe, USB, HID, Firmware, Microcode, ACPI Power Management, Thermal Management, Power Management

二、 引导、初始化与系统管理(Boot, Init & System Management)

引导与加载程序:Boot Process, Secure Boot, TPM, BIOS, UEFI, GRUB, GRUB Configuration, Kernel Command Line, Bootloader
初始化与服务管理:systemd, SysVinit, OpenRC, systemd Services, Unit Files, Targets, Sockets, Mount Units, Automount Units, Initramfs, mkinitcpio, dracut, Initramfs Hooks
硬件热插拔与设备管理:udev, udevadm, D-Bus, D-Bus Activation, systemd-logind, logind, TTY, PTY, Pseudo-Terminal, Linux Console, VT

三、 系统接口、标准与库(ABIs, Standards & System Libraries)

标准与规范:POSIX, GNU, Filesystem Hierarchy Standard, FHS, XDG Base Directory Specification, XDG Desktop Entry Specification, MIME Types, Desktop Entry, xdg-utils, xdg-desktop-portal
系统调用与二进制接口:Syscalls, ELF, Dynamic Linker, Static Linking, Dynamic Linking, ABI
核心运行库:GNU C Library (glibc), musl, GNU Readline, gettext, ICU, Unicode, UTF-8

四、 文件系统与存储卷管理(Filesystems & Storage)

标准与特殊文件系统:procfs, sysfs, devtmpfs, tmpfs, ext4, XFS, Btrfs, F2FS, VFAT, FAT32, NTFS, ISO 9660, SquashFS, OverlayFS, NFS, SMB, CIFS, FUSE
存储与卷管理:LVM, Device Mapper, RAID, mdadm, LUKS, dm-crypt, Mount, Bind Mount, fstab, crypttab, mtab

五、 安全、权限与隔离(Security, Permissions & Isolation)

身份与访问控制:PAM, PAM Authentication, User Management, Group Management, Login Sessions, Authentication, Authorization, passwd, shadow, group, sudo, sudoers, visudo, NSS (Name Service Switch)
权限与属性:File Permissions, ACL, umask, Extended Attributes, Capabilities, Linux Capabilities, chroot
强制访问控制与沙盒:SELinux, SELinux Policy, AppArmor, AppArmor Profiles, seccomp, Seccomp Filtering
容器隔离技术:namespaces (mount, PID, user, network, IPC, UTS, Cgroup), cgroups, cgroups v2, Process Isolation, Resource Control, Containerization, Virtualization, KVM, QEMU

六、 网络与通信(Networking & IPC)

网络协议栈:TCP/IP, UDP, IPv4, IPv6, ICMP, ARP, DNS, DHCP, NAT, VLAN, VPN, Routing, Bridging, Bonding
网络配置与管理:NetworkManager, systemd-networkd, wpa_supplicant, iwd, iproute2, ethtool, ip, ss, ping, traceroute, tracepath, route
防火墙与包过滤:nftables, iptables, Netfilter, Firewall
安全通信与远程连接:WireGuard, OpenVPN, SSH, OpenSSH, SFTP, SCP, rsync, curl, wget, HTTP, HTTPS, TLS, SSL, OpenSSL
服务发现与时间同步:DNS Resolution, systemd-resolved, Avahi, mDNS, NTP, Chrony, systemd-timesyncd, resolvectl
进程间通信 (IPC):Signals, Pipes, FIFOs, Unix Domain Sockets, Shared Memory, Semaphores, Message Queues, POSIX Threads, pthreads, \
IPC, mmap, epoll, io_uring, inotify, fanotify, POSIX Shared Memory, POSIX Semaphores, POSIX Message Queues, System V IPC

七、 图形、音频与外设(Graphics, Audio & Peripherals)

显示与窗口系统:X11, Xorg, Wayland, XWayland, Mesa, OpenGL, Vulkan, wlroots, DRM, KMS, Framebuffer, XKB, libinput
UI 框架与渲染:GTK, Qt, SDL, Fontconfig, FreeType, Pango, Cairo
音频与蓝牙:ALSA, PipeWire, PulseAudio, BlueZ, Bluetooth

八、 开发与编译工具链(Toolchain & Compilation)

编译与构建工具:GCC, GNU Binutils, Make, GNU Make, pkg-config, Kernel Modules, DKMS, Kernel Headers

九、 包管理与系统维护(Package Management & Maintenance)

包管理器与仓库:APT, DPKG, RPM, DNF, Pacman, Portage, Zypper, Flatpak, Snap, AppImage, AUR, Package Repositories, Package Signing, Dependency Resolution, GPG Package Verification, GnuPG, GPG
系统生命周期更新:Kernel Updates, Firmware Updates, System Updates, Package Hooks, Package Cache, Package Database

十、 用户空间工具与排错诊断(Core Utilities & Diagnostics)

文档与手册:Man Pages, man, info, GNU Info, apropos, whatis, tldr, Texinfo, Groff, Documentation, Help System

核心文本与文件处理:GNU Coreutils Utilities, findutils, grep, sed, awk, diffutils, gawk, tar, gzip, bzip2, xz, \
zstd, zip, unzip, file, which, whereis, xargs, tee, cut, sort, uniq, tr, head, tail, wc, od, hexdump, strings, \
stat, touch, ln, cp, mv, rm, mkdir, rmdir, chmod, chown, chgrp, chattr, lsattr, dd, shred, sync


存储与硬件诊断:mount, umount, lsblk, blkid, fdisk, cfdisk, parted, mkfs, fsck, tune2fs, e2fsprogs, smartmontools, nvme-cli, hdparm, lsusb, lspci, lscpu, lsmem, lshw, dmidecode, hwinfo
性能、进程与内核调试:iostat, vmstat, free, uptime, top, ps, pgrep, pkill, kill, nice, renice, taskset, chrt, lsof, fuser, strace, ltrace, dmesg, lsmod, modprobe, insmod, rmmod, depmod, sysctl
日志与定时任务:journald, systemd-journald, journalctl, rsyslog, syslog, logrotate, auditd, Linux Audit Framework, coredump, core dump, Cron, crontab, Anacron, systemd Timers, at

学习大纲

该大纲仅为RHCA认证且有需要/想继续深挖下去的人而做

一、 核心破局点:从 "配置工程师" 转向 "系统观察者"

不要一开始就去读内核源码。第一步是把封装好的工具(如Ansible、各种封装好的集群软件)剥开,去直面Linux内核暴露出的一系列接口

切入点:把 Linux 当成一个 "输入-输出" 的巨大状态机
不要学 C 语言,先学 Shell 脚本与系统调用追踪:利用 strace、ltrace、perf 去看你平时跑的命令到底在向内核请求什么系统调用(Syscalls)
善用虚拟文件系统(procfs 和 sysfs):Linux 把一切皆文件做到了极致。内核的运行状态、内存分配、调度器参数全部映射在 /proc 和 /sys 里。理解了它们,你就不需要懂C语言也能知道内核在干嘛

二、 进阶学习四阶段大纲

阶段一:打破黑盒 --- 从系统调用与运行库理解"进程"

目标:跳出 "命令怎么用" 的层级,理解一个程序在 Linux 里是如何从磁盘变成内存中活生生的进程的
进程与生命周期:
ELF 文件格式与动态链接器(Dynamic Linker)的工作原理
fork()、execve() 与进程创建
进程状态转换(运行、睡眠、僵尸进程的本质)

目标:打破对命令的迷信,用工具链解剖一个二进制文件是如何变成内存中活生生的进程的
核心穿插点:
静态解剖(Binutils 与 ELF):
使用 readelf 和 objdump 查看一个编译好的二进制文件(如自定义脚本或工具)的段(Section)、符号表和动态链接依赖
使用 ldd 理解动态链接器(Dynamic Linker)如何在运行时加载共享库

动态追踪(Syscalls 与 strace):
使用 strace -f -T -tt 追踪一个程序从 execve 系统调用开始的每一个动作。观察它如何打开文件、分配内存、建立网络套接字
理解用户空间(User Space)与内核空间(Kernel Space)的切换开销

进程状态机与生命周期:
深入理解 fork() 的写时复制(Copy-on-Write)机制
剖析什么是孤儿进程、僵尸进程(Zombie),以及为什么父进程不回收 SIGCHLD 会导致资源耗尽。用 ps 和 top 定位它们




内存管理抽象:
虚拟内存与物理内存的映射(Page Table)
Page Cache 与 Buffer Cache 的区别(为什么 Linux 会吃满内存?)
OOM Killer 触发的底层逻辑

目标:彻底告别“Linux 内存占用高就是内存不足”的民科认知,建立虚拟内存与物理内存的映射全景
核心穿插点:
虚拟内存抽象与页表(Page Table):
理解为什么每个进程都拥有看似连续且独立寻址的 4GB/64TB 虚拟地址空间
通过 /proc/[pid]/maps 查看一个运行中的进程内存布局(Heap、Stack、Anonymous Mappings、Shared Libraries)

缓存与回收机制(Page Cache 与 Buffer Cache):
用 free 和 vmstat 结合剖析内存水位。区分可用内存(available)与空闲内存(free)的本质区别
理解脏页(Dirty Pages)回写机制(pdflush / flush 线程)对 I/O 性能的影响

终极审判:OOM Killer 的底层逻辑:
当物理内存和 Swap 耗尽时,内核的 OOM Killer 是如何通过 oom_score 计算并挑选“牺牲者”的
如何通过 sysctl 调整 vm.overcommit_memory 和 vm.oom_score_adj 来保护核心服务



核心工具链强化:
使用 strace 追踪某个服务启动时调用的每一个系统调用
使用 ldd 和 readelf 剖析程序的依赖关系


阶段二:资源边界与隔离 --- 玩转容器底层的三驾马车

目标:理解 Docker、Incus 等容器技术背后的底层机制(这是Linux运维向架构演进的分水岭)
Namespaces(名字空间):
6大隔离机制 (Mount, PID, Network, IPC, UTS, User)
手动用 unshare 或 ip netns 搭建隔离环境

Cgroups(控制组):
Cgroups v1 与 v2 的区别

如何通过 /sys/fs/cgroup 手动限制 CPU、内存和 I/O

Seccomp 与 Capabilities:
精细化收缩进程权限,理解 Linux Capabilities(如 CAP_NET_ADMIN)的本质,而非盲目使用 root。

阶段三:网络与存储的纵深 --- 突破"配置"看"流向"

目标:当网络不通或存储卡死时,能够具备独立定位到驱动或协议栈的能力

网络数据包旅程:
网卡中断 -> NAPI 机制 -> Netfilter/iptables/nftables 钩子 -> 协议栈 -> Socket 缓存
理解 epoll 与高并发网络模型的基本原理

存储与文件系统底层:
Inode、Dentry 与文件描述符(File Descriptors)的关系
Device Mapper 架构(LVM、LUKS 加密、Snapshots 的底层实现)
页缓存写回机制与 I/O 调度器(BFQ、Kyber、None)

阶段四:故障排查与性能调优(Performance Engineering)

目标:建立系统级的排查直觉,面对高负载、CPU 飙升、内存泄漏或内核崩溃能够降维打击
CPU 与调度器:
CFS(完全公平调度器)与现代内核引入的 EEVDF 调度器
上下文切换(Context Switch)的开销与CPU亲和性(CPU Affinity)

内核崩溃与诊断:
分析 dmesg 与 Kernel Panic 日志
使用 kdump 抓取和分析 coredump
使用 sysctl 调优内核参数 (如vm.swappiness, net.core.somaxconn)

小结

亲手构建一次 LFS (Linux From Scratch):
如果你还没有做过,我建议你花时间看着官方文档完整编译一次LFS(即便如此也会遇到各种报错需要你排错)。它会强迫你直面 GCC、Binutils、Make、GLIBC 和内核头文件的依赖关系,这是理解整个 Linux 用户空间与内核关系的绝佳途径

实操

一、核心破局点 -- 实操

因为 Redhat系 总感觉有种xx的感觉,所以以下所有实操都在 Ubuntu 24.04.01 LTS (Noble Numbat)上进行,需要在Redhat系上操作时会有说明

实验一:静态解剖二进制文件 (ELF与动态链接)
目的:彻底搞清楚一个可执行文件在硬盘上到底长什么样,它依赖哪些系统库
查看可执行文件的"身份证"(ELF头信息)
rambo@ub2404:~$ mkdir test4 && cd test4
rambo@ub2404:~$ readelf -h /usr/bin/ls               # 盲猜-h的head的意思
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Position-Independent Executable file)     # 通常为DYN(即位置无关可执行文件PIE)、Machine(架构如X86-64)以及 Entry point address(程序真正开始执行的内存虚拟地址入口)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x6d30
  Start of program headers:          64 (bytes into file)
  Start of section headers:          140328 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         13
  Size of section headers:           64 (bytes)
  Number of section headers:         31
  Section header string table index: 30


# 解剖程序依赖的动态库与链接器
rambo@ub2404:~/test4$ ldd /usr/bin/ls
	linux-vdso.so.1 (0x0000701a85125000)        # 内核直接映射到每个进程内存空间里的虚拟共享库,用来加速部分系统调用(如获取时间gettimeofday),无需陷入内核态
	libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x0000701a850bc000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000701a84e00000)           # 是GNU C Library(glibc),所有标准系统调用的封装都在这里
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x0000701a85022000)
	/lib64/ld-linux-x86-64.so.2 (0x0000701a85127000)          
rambo@ub2404:~/test4$ readelf -l /usr/bin/ls | grep interpreter
   [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]     # interpreter指向/lib64/ld-linux-x86-64.so.2,它才是操作系统加载ls时第一个运行的程序,负责把其他.so库搬进内存



实验二:动态追踪系统调用(穿透用户态与内核态边界)
目的:不看源码,直接用 strace 监视程序在运行期间向内核发出的每一个请求
追踪一次完整命令的系统调用耗时与详情
rambo@ub2404:~/test4$ strace -tt -T -f -o /tmp/ls_trace.log ls -l /etc/passwd       
-rw-r--r-- 1 root root 2998 Nov  3 07:57 /etc/passwd
参数释义:-tt 打印微秒级时间戳;-T 打印每个系统调用消耗的时间;-f 追踪派生出的子进程


rambo@ub2404:~/test4$ cat /tmp/ls_trace.log
释义: Shell进程通过fork()复制自己后,调用execve用ls的代码覆盖当前进程空间
480952 22:55:13.799860 execve("/usr/bin/ls", ["ls", "-l", "/etc/passwd"], 0x7ffcb3ae9ef8 /* 24 vars */) = 0 <0.000727>
释义: 程序向内核申请内存空间,并把libc.so.6映射到内存
480952 22:55:13.801042 brk(NULL)        = 0x64167fc71000 <0.000114>
480952 22:55:13.801526 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76ce31de7000 <0.000151>
480952 22:55:13.801838 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000096>
释义:动态链接器寻找共享库缓存
480952 22:55:13.802272 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 <0.000151>
480952 22:55:13.802647 fstat(3, {st_mode=S_IFREG|0644, st_size=60983, ...}) = 0 <0.000138>
480952 22:55:13.802949 mmap(NULL, 60983, PROT_READ, MAP_PRIVATE, 3, 0) = 0x76ce31dd8000 <0.000119>
480952 22:55:13.803190 close(3)         = 0 <0.000081>
480952 22:55:13.803473 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3 <0.000109>
480952 22:55:13.803723 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 <0.000075>
480952 22:55:13.803966 fstat(3, {st_mode=S_IFREG|0644, st_size=174472, ...}) = 0 <0.000079>
480952 22:55:13.804194 mmap(NULL, 181960, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76ce31dab000 <0.000187>
480952 22:55:13.804543 mmap(0x76ce31db1000, 118784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x76ce31db1000 <0.000119>
480952 22:55:13.804792 mmap(0x76ce31dce000, 24576, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x76ce31dce000 <0.000103>
480952 22:55:13.805021 mmap(0x76ce31dd4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x29000) = 0x76ce31dd4000 <0.000111>
480952 22:55:13.805345 mmap(0x76ce31dd6000, 5832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76ce31dd6000 <0.000117>
480952 22:55:13.805644 close(3)         = 0 <0.000090>
480952 22:55:13.805893 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 <0.000103>
480952 22:55:13.806149 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\243\2\0\0\0\0\0"..., 832) = 832 <0.000087>
480952 22:55:13.806423 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 <0.000109>
480952 22:55:13.806699 fstat(3, {st_mode=S_IFREG|0755, st_size=2129424, ...}) = 0 <0.000093>
480952 22:55:13.806939 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 <0.000085>
480952 22:55:13.807163 mmap(NULL, 2174352, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76ce31a00000 <0.000101>
480952 22:55:13.807517 mmap(0x76ce31a28000, 1609728, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x76ce31a28000 <0.000151>
480952 22:55:13.807834 mmap(0x76ce31bb1000, 323584, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b1000) = 0x76ce31bb1000 <0.000116>
480952 22:55:13.808136 mmap(0x76ce31c00000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1ff000) = 0x76ce31c00000 <0.000104>
480952 22:55:13.808456 mmap(0x76ce31c06000, 52624, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76ce31c06000 <0.000109>
480952 22:55:13.808703 close(3)         = 0 <0.000071>
480952 22:55:13.808892 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 3 <0.000082>
480952 22:55:13.809091 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 <0.000071>
480952 22:55:13.809271 fstat(3, {st_mode=S_IFREG|0644, st_size=625344, ...}) = 0 <0.000083>
480952 22:55:13.809483 mmap(NULL, 627472, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76ce31d11000 <0.000080>
480952 22:55:13.809666 mmap(0x76ce31d13000, 450560, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x76ce31d13000 <0.000089>
480952 22:55:13.809864 mmap(0x76ce31d81000, 163840, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x70000) = 0x76ce31d81000 <0.000116>
480952 22:55:13.810104 mmap(0x76ce31da9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x97000) = 0x76ce31da9000 <0.000097>
480952 22:55:13.810431 close(3)         = 0 <0.000079>
480952 22:55:13.810665 mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76ce31d0e000 <0.000076>
480952 22:55:13.810858 arch_prctl(ARCH_SET_FS, 0x76ce31d0e800) = 0 <0.000069>
480952 22:55:13.811029 set_tid_address(0x76ce31d0ead0) = 480952 <0.000068>
480952 22:55:13.811197 set_robust_list(0x76ce31d0eae0, 24) = 0 <0.000067>
480952 22:55:13.811402 rseq(0x76ce31d0f120, 0x20, 0, 0x53053053) = 0 <0.000080>
480952 22:55:13.811700 mprotect(0x76ce31c00000, 16384, PROT_READ) = 0 <0.000084>
480952 22:55:13.811892 mprotect(0x76ce31da9000, 4096, PROT_READ) = 0 <0.000076>
480952 22:55:13.812110 mprotect(0x76ce31dd4000, 4096, PROT_READ) = 0 <0.000080>
480952 22:55:13.812370 mprotect(0x641645f42000, 8192, PROT_READ) = 0 <0.000081>
480952 22:55:13.812563 mprotect(0x76ce31e27000, 8192, PROT_READ) = 0 <0.000087>
480952 22:55:13.812789 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 <0.000054>
480952 22:55:13.812984 munmap(0x76ce31dd8000, 60983) = 0 <0.000110>
480952 22:55:13.813224 statfs("/sys/fs/selinux", 0x7ffc51f5e530) = -1 ENOENT (No such file or directory) <0.000154>
480952 22:55:13.813522 statfs("/selinux", 0x7ffc51f5e530) = -1 ENOENT (No such file or directory) <0.000077>
480952 22:55:13.813727 getrandom("\x67\xed\xaa\x12\xb5\x3c\x15\x0b", 8, GRND_NONBLOCK) = 8 <0.000074>
480952 22:55:13.813926 brk(NULL)        = 0x64167fc71000 <0.000082>
480952 22:55:13.814127 brk(0x64167fc92000) = 0x64167fc92000 <0.000086>
480952 22:55:13.814430 openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3 <0.000101>
480952 22:55:13.814667 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 <0.000079>
480952 22:55:13.814873 read(3, "nodev\tsysfs\nnodev\ttmpfs\nnodev\tpr"..., 1024) = 405 <0.000115>
480952 22:55:13.815112 read(3, "", 1024) = 0 <0.000072>
480952 22:55:13.815312 close(3)         = 0 <0.000118>
480952 22:55:13.815546 access("/etc/selinux/config", F_OK) = -1 ENOENT (No such file or directory) <0.000074>
480952 22:55:13.815774 openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3 <0.000084>
480952 22:55:13.815993 fstat(3, {st_mode=S_IFREG|0644, st_size=5719296, ...}) = 0 <0.000073>
480952 22:55:13.816182 mmap(NULL, 5719296, PROT_READ, MAP_PRIVATE, 3, 0) = 0x76ce31400000 <0.000084>
480952 22:55:13.816448 close(3)         = 0 <0.000071>
480952 22:55:13.816717 ioctl(1, TCGETS, {c_iflag=ICRNL|IXON|IUTF8, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0 <0.000086>
480952 22:55:13.817051 openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3 <0.000081>
480952 22:55:13.817254 fstat(3, {st_mode=S_IFREG|0644, st_size=2996, ...}) = 0 <0.000085>
480952 22:55:13.817476 read(3, "# Locale name alias data base.\n#"..., 4096) = 2996 <0.000074>
480952 22:55:13.817676 read(3, "", 4096) = 0 <0.000070>
480952 22:55:13.817849 close(3)         = 0 <0.000071>
480952 22:55:13.818031 openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000075>
480952 22:55:13.818217 openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000101>
480952 22:55:13.818453 openat(AT_FDCWD, "/usr/share/locale/en_US/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000075>
480952 22:55:13.818640 openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000072>
480952 22:55:13.818822 openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000072>
480952 22:55:13.819004 openat(AT_FDCWD, "/usr/share/locale/en/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000073>
480952 22:55:13.819192 openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY|O_CLOEXEC) = 3 <0.000076>
480952 22:55:13.819427 fstat(3, {st_mode=S_IFREG|0644, st_size=27028, ...}) = 0 <0.000072>
480952 22:55:13.819626 mmap(NULL, 27028, PROT_READ, MAP_SHARED, 3, 0) = 0x76ce31de0000 <0.000080>
480952 22:55:13.819809 close(3)         = 0 <0.000069>
480952 22:55:13.819982 futex(0x76ce31c0572c, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000070>
释义:获取/etc/passwd的元数据(Inode、权限、大小等)
480952 22:55:13.820231 statx(AT_FDCWD, "/etc/passwd", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=2998, ...}) = 0 <0.000117>
释义:获取selinux的相关
480952 22:55:13.820508 lgetxattr("/etc/passwd", "security.selinux", 0x64167fc73290, 255) = -1 ENODATA (No data available) <0.000096>
480952 22:55:13.820735 listxattr("/etc/passwd", "", 152) = 0 <0.000082>
480952 22:55:13.820962 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 <0.000093>
480952 22:55:13.821169 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) <0.000121>
480952 22:55:13.821428 close(3)         = 0 <0.000091>
480952 22:55:13.821634 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 <0.000089>
480952 22:55:13.821836 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) <0.000119>
480952 22:55:13.822086 close(3)         = 0 <0.000094>
480952 22:55:13.822376 newfstatat(AT_FDCWD, "/etc/nsswitch.conf", {st_mode=S_IFREG|0644, st_size=594, ...}, 0) = 0 <0.000109>
480952 22:55:13.822613 newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0 <0.000075>
480952 22:55:13.822814 openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3 <0.000079>
480952 22:55:13.823003 fstat(3, {st_mode=S_IFREG|0644, st_size=594, ...}) = 0 <0.000070>
480952 22:55:13.823192 read(3, "# /etc/nsswitch.conf\n#\n# Example"..., 4096) = 594 <0.000072>
480952 22:55:13.823425 read(3, "", 4096) = 0 <0.000071>
480952 22:55:13.823605 fstat(3, {st_mode=S_IFREG|0644, st_size=594, ...}) = 0 <0.000066>
480952 22:55:13.823782 close(3)         = 0 <0.000082>
480952 22:55:13.823990 openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3 <0.000092>
480952 22:55:13.824199 fstat(3, {st_mode=S_IFREG|0644, st_size=2998, ...}) = 0 <0.000080>
480952 22:55:13.824459 lseek(3, 0, SEEK_SET) = 0 <0.000087>
480952 22:55:13.824658 read(3, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 2998 <0.000073>
480952 22:55:13.824845 close(3)         = 0 <0.000072>
480952 22:55:13.825023 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 <0.000082>
480952 22:55:13.825208 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) <0.000121>
480952 22:55:13.825449 close(3)         = 0 <0.000076>
480952 22:55:13.825627 socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 <0.000075>
480952 22:55:13.825804 connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) <0.000080>
480952 22:55:13.825995 close(3)         = 0 <0.000132>
480952 22:55:13.826232 newfstatat(AT_FDCWD, "/etc/nsswitch.conf", {st_mode=S_IFREG|0644, st_size=594, ...}, 0) = 0 <0.000087>
480952 22:55:13.826449 openat(AT_FDCWD, "/etc/group", O_RDONLY|O_CLOEXEC) = 3 <0.000076>
480952 22:55:13.826635 fstat(3, {st_mode=S_IFREG|0644, st_size=1146, ...}) = 0 <0.000069>
480952 22:55:13.826824 lseek(3, 0, SEEK_SET) = 0 <0.000069>
480952 22:55:13.826994 read(3, "root:x:0:\ndaemon:x:1:\nbin:x:2:\ns"..., 4096) = 1146 <0.000070>
480952 22:55:13.827174 close(3)         = 0 <0.000070>
480952 22:55:13.827392 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x3), ...}) = 0 <0.000072>
480952 22:55:13.827590 openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 3 <0.000076>
480952 22:55:13.827773 fstat(3, {st_mode=S_IFREG|0644, st_size=561, ...}) = 0 <0.000069>
480952 22:55:13.827948 fstat(3, {st_mode=S_IFREG|0644, st_size=561, ...}) = 0 <0.000085>
480952 22:55:13.828152 read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 561 <0.000070>
480952 22:55:13.828365 lseek(3, -342, SEEK_CUR) = 219 <0.000076>
480952 22:55:13.828569 read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 342 <0.000071>
480952 22:55:13.828769 close(3)         = 0 <0.000074>
释义:向文件描述符1(标准输出stdout)写入结果
480952 22:55:13.829060 write(1, "-rw-r--r-- 1 root root 2998 Nov "..., 53) = 53 <0.000078>
480952 22:55:13.829275 close(1)         = 0 <0.000093>
480952 22:55:13.829583 close(2)         = 0 <0.000067>
释义:通知内核销毁进程,返回退出码0
480952 22:55:13.829798 exit_group(0)    = ?
480952 22:55:13.830221 +++ exited with 0 +++




# 实战排错技巧(统计系统调用耗时瓶颈)
rambo@ub2404:~/test4$ strace -c ls -lR /usr/share/doc > /dev/null
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 33.88    0.452097          39     11376           statx
 20.55    0.274241          71      3831           getdents64
 17.35    0.231565          24      9461      9461 lgetxattr
 15.02    0.200375          22      8748           listxattr
  4.08    0.054438          27      1945        17 openat
  3.76    0.050134          25      1934           close
  3.09    0.041206          21      1931           fstat
  1.61    0.021475          30       713           readlink
  0.25    0.003283          21       155           write
  0.14    0.001904          90        21           mmap
  0.09    0.001147        1147         1           execve
  0.04    0.000517          39        13           read
  0.03    0.000343          68         5           mprotect
  0.02    0.000285          40         7           brk
  0.01    0.000186          46         4           socket
  0.01    0.000181          45         4         4 connect
  0.01    0.000105          52         2         2 statfs
  0.01    0.000094          47         2         2 access
  0.01    0.000090          90         1           munmap
  0.01    0.000090          45         2           pread64
  0.01    0.000088          29         3           newfstatat
  0.01    0.000068          68         1           set_robust_list
  0.00    0.000063          21         3           lseek
  0.00    0.000056          56         1           set_tid_address
  0.00    0.000055          27         2         2 ioctl
  0.00    0.000055          55         1           arch_prctl
  0.00    0.000043          43         1           rseq
  0.00    0.000042          42         1           getrandom
  0.00    0.000040          40         1           mremap
  0.00    0.000026          26         1           futex
  0.00    0.000018          18         1           prlimit64
------ ----------- ----------- --------- --------- ----------------
100.00    1.334310          33     40172      9488 total

各列详细释义:
% time:该系统调用耗时占整个程序运行期间所有系统调用总耗时的百分比。这是定位性能瓶颈的核心指标(例如发现90%的时间都卡在futex或read上)
seconds:该系统调用消耗的总 CPU 时间(秒),不包含用户空间的代码执行时间
usecs/call:平均每次调用消耗的微秒数(seconds x 1,000,000/calls。反映单个调用的开销
calls:该系统调用在整个生命周期中被执行的次数。像 ls -lR 递归遍历目录时,getdents64(读取目录内容)和 lstat(获取文件元数据)的调用次数通常最高
errors:该系统调用返回错误(如文件不存在、权限拒绝)的次数。如果某些调用报错率极高,这里会显现
syscall:系统调用名称。例如:
    getdents64:读取目录数据
    lstat / stat:查询文件属性(大小、权限、Inodes)
    openat / close:打开和关闭文件描述符
    write:向标准输出写入数据



实验三:深入 /proc 窥探进程内存真实布局
目的:打破"看 top 里的 VIRT 和 RES 猜内存" 的浅层认知,直接看内核维护的进程内存页表映射
rambo@ub2404:~/test4$ sleep 3600 &
[1] 480996
rambo@ub2404:~/test4$ PID=$!
rambo@ub2404:~/test4$ echo "Target PID: $PID"
Target PID: 480996

查看进程的虚拟地址空间分布
rambo@ub2404:~/test4$ cat /proc/$PID/maps
5f5e9af7c000-5f5e9af7e000 r--p 00000000 08:04 2395534                    /usr/bin/sleep
5f5e9af7e000-5f5e9af82000 r-xp 00002000 08:04 2395534                    /usr/bin/sleep
5f5e9af82000-5f5e9af83000 r--p 00006000 08:04 2395534                    /usr/bin/sleep
5f5e9af83000-5f5e9af84000 r--p 00007000 08:04 2395534                    /usr/bin/sleep
5f5e9af84000-5f5e9af85000 rw-p 00008000 08:04 2395534                    /usr/bin/sleep
5f5e9d0f8000-5f5e9d119000 rw-p 00000000 00:00 0                          [heap]
748ba6000000-748ba6575000 r--p 00000000 08:04 2364429                    /usr/lib/locale/locale-archive
748ba6600000-748ba6628000 r--p 00000000 08:04 2392277                    /usr/lib/x86_64-linux-gnu/libc.so.6
748ba6628000-748ba67b1000 r-xp 00028000 08:04 2392277                    /usr/lib/x86_64-linux-gnu/libc.so.6
748ba67b1000-748ba6800000 r--p 001b1000 08:04 2392277                    /usr/lib/x86_64-linux-gnu/libc.so.6
748ba6800000-748ba6804000 r--p 001ff000 08:04 2392277                    /usr/lib/x86_64-linux-gnu/libc.so.6
748ba6804000-748ba6806000 rw-p 00203000 08:04 2392277                    /usr/lib/x86_64-linux-gnu/libc.so.6
748ba6806000-748ba6813000 rw-p 00000000 00:00 0 
748ba698b000-748ba698e000 rw-p 00000000 00:00 0 
748ba699c000-748ba699d000 r--p 00000000 08:04 2630809                    /usr/share/locale-langpack/en/LC_MESSAGES/coreutils.mo
748ba699d000-748ba699f000 rw-p 00000000 00:00 0 
748ba699f000-748ba69a3000 r--p 00000000 00:00 0                          [vvar]
748ba69a3000-748ba69a5000 r--p 00000000 00:00 0                          [vvar_vclock]
748ba69a5000-748ba69a7000 r-xp 00000000 00:00 0                          [vdso]
748ba69a7000-748ba69a8000 r--p 00000000 08:04 2392274                    /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
748ba69a8000-748ba69d3000 r-xp 00001000 08:04 2392274                    /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
748ba69d3000-748ba69dd000 r--p 0002c000 08:04 2392274                    /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
748ba69dd000-748ba69df000 r--p 00036000 08:04 2392274                    /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
748ba69df000-748ba69e1000 rw-p 00038000 08:04 2392274                    /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7fffb7a65000-7fffb7a86000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

笔记记录重点:每一行代表一段连续的虚拟内存区域(VMA):
权限位(第二列):
    r-xp(可读、可执行、私有)通常是程序的代码段(Text Segment);
    rw-p(可读、可写、私有)通常是数据段或堆[heap];[stack] 是函数调用栈
没有对应文件路径的 rw-p 区域叫做匿名映射(Anonymous Mapping),程序通过malloc申请的大块内存就在这里


计算真实物理内存占用(剔除共享库重复计算)
rambo@ub2404:~/test4$ cat /proc/$PID/smaps_rollup
5f5e9af7c000-7fffb7a86000 ---p 00000000 00:00 0         [rollup]
Rss:                2100 kB       # 全称Resident Set Size:该进程占用的物理内存总和(包含了与其他进程共享的glibc等动态库,直接把所有进程的RSS相加会远超物理内存)
Pss:                 158 kB       # 全称Proportional Set Size:按比例分摊共享库后的真实物理内存占用(如10个进程共享10MB的库,每个进程算1MB)。排查内存泄漏看Pss才最准确
Pss_Dirty:           108 kB
Pss_Anon:            108 kB
Pss_File:             50 kB
Pss_Shmem:             0 kB
Shared_Clean:       1960 kB
Shared_Dirty:          0 kB
Private_Clean:        32 kB
Private_Dirty:       108 kB
Referenced:         2100 kB
Anonymous:           108 kB
KSM:                   0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB




实验四:内核缓存机制与 OOM Killer 触发演练
目的:理解Linux内存回收机制(Page Cache)以及极端内存耗尽时内核如何自保
观察 Page Cache(页缓存)的吞吐与手动释放
# 1. 查看当前内存水位(重点看 buff/cache 和 available)
rambo@ub2404:~/test4$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15941        1963        8448          38        5897       13977
Swap:           5722           0        5722

# 2. 生成一个500MB的测试文件,观察cache飙升
rambo@ub2404:~/test4$ dd if=/dev/zero of=/tmp/test_cache.img bs=1M count=500 oflag=direct
rambo@ub2404:~/test4$ cat /tmp/test_cache.img > /dev/null
rambo@ub2404:~/test4$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15941        1963        7948          38        6397       13977
Swap:           5722           0        5722

# 3. 模拟内核在内存紧张时主动回收干净的 Page Cache(生产环境慎用)
rambo@ub2404:~/test4$ sync
rambo@ub2404:~/test4$ sudo sh -c 'echo 1 > /proc/sys/vm/drop_caches'       或 echo 1 | sudo tee /proc/sys/vm/drop_caches
rambo@ub2404:~/test4$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15941        1839       13481          38         989       14102
Swap:           5722           0        5722

笔记记录重点:
free列低了完全正常,因为内核把空闲物理内存全拿去做磁盘缓存(buff/cache)了。只有当available列接近0时,系统才真正面临内存枯竭

rambo@ub2404:~/test4$ rm -f /tmp/test_cache.img


查看并调整进程的OOM死亡优先级
# 查看 sshd 守护进程的 OOM 分数(分数越高越容易被先杀,范围 -1000 到 1000)
rambo@ub2404:~/test4$ cat /proc/$(pgrep -o sshd)/oom_score
0
rambo@ub2404:~/test4$ cat /proc/$(pgrep -o sshd)/oom_score_adj
-1000

笔记记录重点:
当物理内存和 Swap 彻底耗尽,内核触发 OOM Killer(Out-Of-Memory Killer)
它遍历所有进程,根据内存占用比例加上 oom_score_adj 计算最终得分,杀掉得分最高的进程并记录到dmesg(搜索 Out of memory: Killed process)
核心数据库或SSH服务通常通过设置 oom_score_adj 为 -1000 来获得"免死金牌"


posted @ 2026-11-12 10:07  Linux大魔王  阅读(5)  评论(0)    收藏  举报