15 shell编程及linux三剑客：grep sed awk
全局替换:g/3306/s//3308/g 或者:%s/3306/3308/g
n1到n2行替换:n1,n2s/3306/3308/gc
shell编程
使用优点：
linux系统批量管理；提高工作效率；减少重复性工作；懒
准备知识：
vim；ssh终端；linux命令；linux正则表达式及三剑客命令
shell：
命令解释器 （输入的命令来给你运行、解释）
[root@oldboyedu-01 ~]# echo $SHELL //centos默认shell是bash命令解释器
/bin/bash
[root@oldboyedu-01 ~]# cat /etc/passwd |grep root
root:x:0:0:root:/root:/bin/bash
[root@oldboyedu-01 ~]# cat /etc/shells
/bin/sh
/bin/bash
/sbin/nologin
/bin/dash
/bin/tcsh
/bin/csh
shell脚本：
命令大礼包-一个程序文件，包含若干行linux命令；
循环、条件语句。
[root@oldboyedu-01 ~]# file /etc/init.d/iptables //查看是否是脚本文件
/etc/init.d/iptables: POSIX shell script text executable
#!/bin/bash //第一行指出由哪个解释器来执行脚本中的内容。
eg:
[root@oldboyedu-01 ~]# cat /service/script/ip.sh
#!/bin/bash
/sbin/ifconfig eth0|awk -F "[ :]+" 'NR==2{print $4}'
date +%F_%w
[root@oldboyedu-01 ~]# sh /service/script/ip.sh
10.0.0.200
2018-01-11_4
变量：用一个固定的字符串来替代更多复杂的内容。
[root@oldboyedu-01 ~]# x=1
[root@oldboyedu-01 ~]# echo $x
1
[root@oldboyedu-01 ~]# week=10
[root@oldboyedu-01 ~]# echo 10days
10days
[root@oldboyedu-01 ~]# echo $weekdays //犯了金庸新 著的迷惑
[root@oldboyedu-01 ~]# echo ${week}days//引用变量${}
10days
[root@oldboyedu-01 ~]# echo ${week} $week
10 10
变量分类：
（1）局部变量（普通变量）：只能在创建它的函数或者脚本中使用。
变量名=value；
变量名由字母、下划线和数字组成，必须由字母或者下划线开头；
见名知意；
驼峰命名法：oldboyAgeSex=1
将一个命令作为变量。
（2）全局变量（环境变量）：
[root@oldboyedu-01 ~]# echo $PATH $LANG $PS1 //bash内置的环境变量
/application/mysql/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin en_US.UTF-8 [\u@\h \W]\$
[root@oldboyedu-01 ~]# echo $UID
0
[root@oldboyedu-01 ~]# tail -1 /service/script/ip.sh
echo $OLDBOY
[root@oldboyedu-01 ~]# OLDBOY=10
[root@oldboyedu-01 ~]# echo $OLDBOY
10
[root@oldboyedu-01 ~]# sh /service/script/ip.sh
[root@oldboyedu-01 ~]# export OLDBOY=10 //export将局部变量OLDBOY变成全局变量
[root@oldboyedu-01 ~]# sh /service/script/ip.sh
10
[root@oldboyedu-01 ~]# env |grep OLDBOY //舍弃变量
OLDBOY=10
[root@oldboyedu-01 ~]# unset OLDBOY
[root@oldboyedu-01 ~]# env |grep OLDBOY
[root@oldboyedu-01 ~]# vim /etc/profile.d/ipshow.sh
[root@oldboyedu-01 ~]# cat /etc/profile.d/ipshow.sh
#!/bin/bash
ip a s eth0
[root@oldboyedu-01 ~]# chmod +x /etc/profile.d/ipshow.sh
[d:\~]$ //ssh重新登录测试！
Connecting to 10.0.0.200:22...
Connection established.
To escape to local shell, press 'Ctrl+Alt+]'.
Last login: Thu Jan 11 20:37:39 2018 from 10.0.0.1
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:db:5d:02 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.200/24 brd 10.0.0.255 scope global eth0
inet6 fe80::20c:29ff:fedb:5d02/64 scope link
valid_lft forever preferred_lft forever
[root@oldboyedu-01 ~]#
与用户环境有关的文件和目录：
a.全局环境变量配置文件：
[root@oldboyedu-01 ~]# ls /etc/profile /etc/bashrc /etc/profile.d/
/etc/bashrc /etc/profile
/etc/profile.d/: //用户登陆到系统会运行这个目录下的脚本（该脚本需加执行权限！）
colorls.csh cvs.csh glib2.csh ipshow.sh lang.sh less.sh vim.sh
colorls.sh cvs.sh glib2.sh lang.csh less.csh vim.csh which2.sh
b.用户环境变量配置文件：
[root@oldboyedu-01 ~]# ls ~/.bash_profile ~/.bashrc
/root/.bash_profile /root/.bashrc
（3）特殊变量：
a.位置变量：
[root@oldboyedu-01 ~]# cat /service/script/show_args.sh
#!/bin/sh
echo '$0:'$0 $1 $2 $3 $#
[root@oldboyedu-01 ~]# sh /service/script/show_args.sh
$0:/service/script/show_args.sh 0
[root@oldboyedu-01 ~]# sh /service/script/show_args.sh a b c
$0:/service/script/show_args.sh a b c 3
[root@oldboyedu-01 ~]# sh /service/script/show_args.sh a b c d e
$0:/service/script/show_args.sh a b c 5
说明：
$0表示脚本文件名；
$1,$2,$3等表示脚本第1,2,3个参数；
$#表示脚本参数的个数
b.状态变量：
[root@oldboyedu-01 ~]# sh /service/script/show_args.sh
$0:/service/script/show_args.sh 0
[root@oldboyedu-01 ~]# echo $?
0
[root@oldboyedu-01 ~]# ls /oldboy/meng.log
ls: cannot access /oldboy/meng.log: No such file or directory
[root@oldboyedu-01 ~]# echo $?
2
说明：
$? 显示上一个命令执行结果：
成功显示0；失败显示非0.
变量赋值方法：
[root@oldboyedu-01 ~]# cat /service/script/show_args.sh //法一：直接赋值
#!/bin/sh
x=$1
y=$2
echo $x $y
[root@oldboyedu-01 ~]# sh /service/script/show_args.sh 10 20
10 20
[root@oldboyedu-01 ~]# read -p "input x y:" x y //法二：交互式read读入
input x y:10 20
[root@oldboyedu-01 ~]# echo $x $y
10 20
[root@oldboyedu-01 ~]# cat /service/script/show_args.sh
#!/bin/sh
read -p "input x y:" x y
echo $x $y
[root@oldboyedu-01 ~]# sh /service/script/show_args.sh
input x y:30 40
30 40
循环条件
1.条件表达式：eg明天是否下雨？格式：[ <测试表达式> ]
（1）判断文件
[root@oldboyedu-01 ~]# [ -f /oldboy/new.log ] //判断文件，等价于test -f /oldboy/new.log
[root@oldboyedu-01 ~]# echo $?
0
[root@oldboyedu-01 ~]# man test
[root@oldboyedu-01 ~]# [ -f /oldboy/meng.log ]
[root@oldboyedu-01 ~]# echo $?
1
[root@oldboyedu-01 ~]# [ -d /root ] //判断目录
[root@oldboyedu-01 ~]# echo $?
0
（2）判断整数
-eq等于，-ne不等于，-gt大于等于，-ge大于等于，-lt小于，-le小于等于。
[root@oldboyedu-01 ~]# cat /service/script/arg.sh //eg：判断参数个数是否等于2
#!/bin/bash
[ $# -eq 2 ] && echo "args:"$#
[root@oldboyedu-01 ~]# sh /service/script/arg.sh
[root@oldboyedu-01 ~]# sh /service/script/arg.sh a b
args:2
[root@oldboyedu-01 ~]# [ -d /oldboy ] || mkdir -p /oldboy //eg:目录不存在则创建
[root@oldboyedu-01 ~]# echo $?
0
[root@oldboyedu-01 ~]# [ -f /oldboy/oldboy.txt ] && echo file exist! //eg:文件存在则提示
file exist!
[root@oldboyedu-01 ~]# echo $?
0
2.if条件语句
语法：if [ 条件 ];then 命令 fi
eg:输入两个数比较大小
[root@oldboyedu-01 ~]# cat /service/script/compare.sh
#!/bin/bash
sum1=$1
sum2=$2
if [ $sum1 -gt $sum2 ];then
echo $sum1 greater than $sum2
fi
[root@oldboyedu-01 ~]# sh /service/script/compare.sh
greater than
[root@oldboyedu-01 ~]# sh /service/script/compare.sh 10 20
[root@oldboyedu-01 ~]# sh /service/script/compare.sh 20 10
20 greater than 10
[root@oldboyedu-01 ~]# cat /service/script/compare.sh
#!/bin/bash
sum1=$1
sum2=$2
if [ $sum1 -gt $sum2 ];then
echo $sum1 greater than $sum2
else
echo $sum1 less than $sum2
fi
[root@oldboyedu-01 ~]# sh /service/script/compare.sh 10 20
10 less than 20
[root@oldboyedu-01 ~]# sh /service/script/compare.sh 20 10
20 greater than 10
[root@oldboyedu-01 ~]# cat /service/script/compare.sh
#!/bin/bash
sum1=$1
sum2=$2
if [ $# -ne 2 ];then
echo "Usage:please input 2 args:sum1 sum2"
exit
fi
if [ $sum1 -gt $sum2 ];then
echo $sum1 greater than $sum2
else
echo $sum1 less than $sum2
fi
[root@oldboyedu-01 ~]# sh /service/script/compare.sh
Usage:please input 2 args:sum1 sum2
[root@oldboyedu-01 ~]# sh /service/script/compare.sh 10
Usage:please input 2 args:sum1 sum2
[root@oldboyedu-01 ~]# sh /service/script/compare.sh 10 20
10 less than 20
[root@oldboyedu-01 ~]# cat /service/script/compare.sh
#!/bin/bash
sum1=$1
sum2=$2
if [ $# -ne 2 ];then
echo "Usage:please input 2 args:sum1 sum2"
exit
fi
if [ $sum1 -gt $sum2 ];then
echo $sum1 greater than $sum2
elif [ $sum1 -lt $sum2 ];then
echo $sum1 less than $sum2
else echo $sum1 equal $sum2
fi
[root@oldboyedu-01 ~]# sh /service/script/compare.sh
Usage:please input 2 args:sum1 sum2
[root@oldboyedu-01 ~]# sh /service/script/compare.sh 10
Usage:please input 2 args:sum1 sum2
[root@oldboyedu-01 ~]# sh /service/script/compare.sh 10 20
10 less than 20
[root@oldboyedu-01 ~]# sh /service/script/compare.sh 20 10
20 greater than 10
[root@oldboyedu-01 ~]# sh /service/script/compare.sh 10 10
10 equal 10
3.for循环语句
[root@oldboyedu-01 ~]# for num in 1 2 3 4 5 6
> do
> echo "the $num number is:$num"
> done
the 1 number is:1
the 2 number is:2
the 3 number is:3
the 4 number is:4
the 5 number is:5
the 6 number is:6
[root@oldboyedu-01 ~]# for num in {01..10}; do echo "the $num number is:$num"; done
the 01 number is:01
the 02 number is:02
the 03 number is:03
the 04 number is:04
the 05 number is:05
the 06 number is:06
the 07 number is:07
the 08 number is:08
the 09 number is:09
the 10 number is:10
eg:优化linux开机自启动项目，只保留crond,sshd,network,rsyslog,sysstat.其他都关闭。
[root@oldboyedu-01 ~]# chkconfig |egrep "crond|sshd|network|rsyslog|sysstat"
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sysstat 0:off 1:on 2:on 3:on 4:on 5:on 6:off
[root@oldboyedu-01 ~]# chkconfig |egrep -v "crond|sshd|network|rsyslog|sysstat"
abrt-ccpp 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrtd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
......
[root@oldboyedu-01 ~]# chkconfig |egrep -v "crond|sshd|network|rsyslog|sysstat"|awk '{print $1}'
abrt-ccpp
abrtd
acpid
......
[root@oldboyedu-01 ~]# for name in $(chkconfig |egrep -v "crond|sshd|network|rsyslog|sysstat" |awk '{print $1}');do echo chkconfig $name off;done //调试
chkconfig abrt-ccpp off
chkconfig abrtd off
chkconfig acpid off
......
[root@oldboyedu-01 ~]# for name in $(chkconfig |egrep -v "crond|sshd|network|rsyslog|sysstat" |awk '{print $1}');do chkconfig $name off;done //执行优化
[root@oldboyedu-01 ~]# chkconfig |egrep 3:on //检查
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sysstat 0:off 1:on 2:on 3:on 4:on 5:on 6:off
//利用脚本执行
[root@oldboyedu-01 ~]# cat /service/script/opt.sh
#!/bin/sh
#
for name in $(chkconfig |egrep -v "crond|sshd|network|rsyslog|sysstat" |awk '{print $1}');do chkconfig $name off;done
[root@oldboyedu-01 ~]# sh /service/script/opt.sh
//+x 运行脚本调试技巧查看执行过程
[root@oldboyedu-01 ~]# sh -x /service/script/opt.sh
linux三剑客之sed
定义：
字符流编辑器（stream editor），擅长替换。
功能：
处理纯文本文件、日志、配置文件等，进行增删改查。
[root@oldboyedu-01 ~]# sed --v
GNU sed version 4.2.1
[root@oldboyedu-01 ~]# man sed
语法格式：
sed [选项] [sed指令] [输入文件]
[root@oldboyedu-01 ~]# sed -i 's#oldboy#oldgirl#g' /oldboy/oldboybak.txt
命令执行过程：
--------------图sed1
模式空间：sed从文件读取一行文本后存入的缓冲区（该缓冲区是在内存中）
常用功能：-i直接修改
（1）查询--p(print)
[root@oldboyedu-01 oldboy]# cat person.txt
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboyedu-01 oldboy]# sed '1p' person.txt
101,oldboy,CEO
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboyedu-01 oldboy]# sed -n '1p' person.txt //-n取消默认输出，查询某一行
101,oldboy,CEO
[root@oldboyedu-01 oldboy]# sed -n '1,4p' person.txt //查询某个范围行
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
[root@oldboyedu-01 oldboy]# sed -n '/oldboy/p' person.txt //模糊查询，等价于grep 'oldboy' person.txt
101,oldboy,CEO
[root@oldboyedu-01 oldboy]# sed -n '/oldboy/,/^104/p' person.txt //模糊查询某个范围内容行
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
[root@oldboyedu-01 oldboy]# sed -rn '/oldboy|yy/p' person.txt //-r支持扩展正则表达式| ()
101,oldboy,CEO
104,yy,CFO
[root@oldboyedu-01 oldboy]# egrep 'oldboy|yy' person.txt
101,oldboy,CEO
104,yy,CFO
[root@oldboyedu-01 oldboy]# sed -n '1p;3p;4p' person.txt //查询指定行
101,oldboy,CEO
103,Alex,COO
104,yy,CFO
[root@oldboyedu-01 oldboy]# sed -n '/oldboy/p;3p;4p' person.txt
101,oldboy,CEO
103,Alex,COO
104,yy,CFO
（2）增加--a(append),i(insert)
[root@oldboyedu-01 oldboy]# sed -n '3a 103.5 Lee UFO' person.txt
103.5 Lee UFO
[root@oldboyedu-01 oldboy]# sed '3a 103.5 Lee UFO' person.txt //a-append在第3行后增加；-i直接修改!
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
103.5 Lee UFO
104,yy,CFO
105,feixue,CIO
[root@oldboyedu-01 oldboy]# sed '3i 103.5 Lee UFO' person.txt //i-insert在第3行前增加
101,oldboy,CEO
102,zhangyao,CTO
103.5 Lee UFO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboyedu-01 oldboy]# sed '$p' person.txt //最后一行为$
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
105,feixue,CIO
[root@oldboyedu-01 oldboy]# sed '$a new new new' person.txt
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
new new new
[root@oldboyedu-01 oldboy]# sed '$a new new new\nold old old' person.txt //换行符\n
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
new new new
old old old
[root@oldboyedu-01 oldboy]# cat >>person.txt <<EOF //等价
> new new new
> old old old
> EOF
（3）删除、去除空行--d(delete)
[root@oldboyedu-01 oldboy]# sed '1d' person.txt
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboyedu-01 oldboy]# sed '2,3d' person.txt
101,oldboy,CEO
104,yy,CFO
105,feixue,CIO
[root@oldboyedu-01 oldboy]# sed '$d' person.txt
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
[root@oldboyedu-01 oldboy]# sed '/oldboy/d' person.txt
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
eg:不显示文件中的空行。
[root@oldboyedu-01 oldboy]# cat -An person.txt
1 101,oldboy,CEO$
2 $
3 102,zhangyao,CTO$
4 103,Alex,COO$
5 $
6 104,yy,CFO$
7 105,feixue,CIO$
//法一：
[root@oldboyedu-01 oldboy]# sed '/^$/d' person.txt
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
//法二：
[root@oldboyedu-01 oldboy]# grep -nv '^$' person.txt
1:101,oldboy,CEO
3:102,zhangyao,CTO
4:103,Alex,COO
6:104,yy,CFO
7:105,feixue,CIO
//法三：
[root@oldboyedu-01 oldboy]# sed -n '/^$/p' person.txt
[root@oldboyedu-01 oldboy]# sed -n '/^$/!p' person.txt
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboyedu-01 oldboy]# sed '$d' person.txt
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
[root@oldboyedu-01 oldboy]# sed '$!d' person.txt
105,feixue,CIO
（4）修改/替换
a.文本替换
说明：
s--单独使用->将每一行中第一处匹配的字符串进行替换
g--每一行进行全部替换->sed指令s的替换标志之一（全局替换）
[root@oldboyedu-01 oldboy]# sed 's#o#AAAA#g' person.txt //
101,AAAAldbAAAAy,CEO
102,zhangyaAAAA,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboyedu-01 oldboy]# sed 's#o#AAAA#' person.txt
101,AAAAldboy,CEO
102,zhangyaAAAA,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboyedu-01 oldboy]# sed -i.ori 's#o#AAAA#' person.txt //先备份再修改内容
[root@oldboyedu-01 oldboy]# ls person.txt*
person.txt person.txt.ori
[root@oldboyedu-01 oldboy]# cat person.txt person.txt.ori
101,AAAAldboy,CEO
102,zhangyaAAAA,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
b.变量替换
[root@oldboyedu-01 oldboy]# x=oldboy
[root@oldboyedu-01 oldboy]# y=oldgirl
[root@oldboyedu-01 oldboy]# sed 's#$x#$y#g' person.txt
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboyedu-01 oldboy]# sed "s#$x#$y#g" person.txt //需加""，注意单双引号区别！
101,oldgirl,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
c.反向引用
复习。
拓展：
一条sed语句执行多条命令：每个-e选项后可接一个sed指令，;使用。
linux三剑客之awk
1.基本的awk执行过程：
[root@oldboyedu-01 ~]# awk --v
GNU Awk 3.1.7
[root@oldboyedu-01 ~]# awk -F ":" 'NR==2{print $1,$2}' /etc/passwd
bin x
格式：
awk 参数 ‘模式{动作}’ 文件
awk 参数 ‘条件（找谁）{干啥}’ 文件
2.通过正则表达式作为模式
模式匹配：模式与动作
[root@oldboyedu-01 ~]# mkdir -p /service/file
[root@oldboyedu-01 ~]# cd /service/file/
[root@oldboyedu-01 file]# vim reg.txt
[root@oldboyedu-01 file]# cat reg.txt
Zhang Dandan 41117397 :250:100:175
Zhang Xiaoyu 390320151 :155:90:201
Meng Feixue 80042789 :250:60:50
Wu Waiwai 70271111 :250:80:75
Liu Bingbing 41117483 :250:100:175
Wang Xiaoai 3515064655 :50:95:135
Zi Gege 1986787350 :250:168:200
Li Youjiu 918391635 :175:75:300
Lao Nanhai 918391635 :250:100:175
[root@oldboyedu-01 file]# awk '/Z/' reg.txt
Zhang Dandan 41117397 :250:100:175
Zhang Xiaoyu 390320151 :155:90:201
Zi Gege 1986787350 :250:168:200
eg：显示Xiaoyu的姓氏和ID号码。
[root@oldboyedu-01 file]# awk '/Xiaoyu/{print $1,$3}' reg.txt
[root@oldboyedu-01 file]# awk '$0~/Xiaoyu/{print $1,$3}' reg.txt //等价，$0表示这一整行
Zhang 390320151
//$2表示第2列，awk擅长取列！~表示包含、匹配的意思。
[root@oldboyedu-01 file]# awk '$2~/Xiaoyu/{print $1,$3}' reg.txt
Zhang 390320151
eg:显示所有以41开头的ID号码的人的全名和ID号码。
[root@oldboyedu-01 file]# awk '$3~/^41/' reg.txt
Zhang Dandan 41117397 :250:100:175
Liu Bingbing 41117483 :250:100:175
[root@oldboyedu-01 file]# awk '$3~/^41/{print $1$2,$3}' reg.txt
ZhangDandan 41117397
LiuBingbing 41117483
[root@oldboyedu-01 file]# awk '$3~/^41/{print $1,$2,$3}' reg.txt
Zhang Dandan 41117397
Liu Bingbing 41117483
eg:显示所有ID号码最后一位数字是1或5的人的全名。
[root@oldboyedu-01 file]# awk '$3~/[15]$/{print $1,$2}' reg.txt
Zhang Xiaoyu
Wu Waiwai
Wang Xiaoai
Li Youjiu
Lao Nanhai
[root@oldboyedu-01 file]# awk '$3~/(1|5)$/{print $1,$2}' reg.txt //等价
Zhang Xiaoyu
Wu Waiwai
Wang Xiaoai
Li Youjiu
Lao Nanhai
eg:显示Xiaoyu的捐款，每个值时都有以$开头。如$520$200$135
gsub函数说明：
gsub(/目标/,“替换为什么”,第几列)
gsub(/目标/,“替换为什么”) ==》gsub(/目标/,“替换为什么”,$0)
[root@oldboyedu-01 file]# awk '{gsub(/:/,"$",$4)}' reg.txt
[root@oldboyedu-01 file]# awk '{gsub(/:/,"$",$4);print}' reg.txt
Zhang Dandan 41117397 $250$100$175
Zhang Xiaoyu 390320151 $155$90$201
Meng Feixue 80042789 $250$60$50
Wu Waiwai 70271111 $250$80$75
Liu Bingbing 41117483 $250$100$175
Wang Xiaoai 3515064655 $50$95$135
Zi Gege 1986787350 $250$168$200
Li Youjiu 918391635 $175$75$300
Lao Nanhai 918391635 $250$100$175
[root@oldboyedu-01 file]# awk '$2~/Xiaoyu/' reg.txt //
Zhang Xiaoyu 390320151 :155:90:201
[root@oldboyedu-01 file]# awk '$2~/Xiaoyu/{gsub(/:/,"$",$4)}' reg.txt
[root@oldboyedu-01 file]# awk '$2~/Xiaoyu/{gsub(/:/,"$",$4);print}' reg.txt //
Zhang Xiaoyu 390320151 $155$90$201
[root@oldboyedu-01 file]# awk '$2~/Xiaoyu/{gsub(/:/,"$",$0);print}' reg.txt //等价
Zhang Xiaoyu 390320151 $155$90$201
[root@oldboyedu-01 file]# awk '$2~/Xiaoyu/{gsub(/:/,"$");print}' reg.txt //等价
Zhang Xiaoyu 390320151 $155$90$201
3.特殊模式BEGIN与END
BEGIN{}--BEGIN里面的内容会在awk读取文件内容之前运行。
一般用作测试，计算。
END{}--END里面的内容会在awk读取文件内容最后一行后运行。
一般用作显示最终结果。
先计算，END显示最终结果。
[root@oldboyedu-01 file]# awk 'BEGIN{print "this is kt..."}'
this is kt...
[root@oldboyedu-01 file]# awk 'BEGIN{print "this is kt..."} {print NR,$0}' reg.txt
this is kt...
1 Zhang Dandan 41117397 :250:100:175
2 Zhang Xiaoyu 390320151 :155:90:201
3 Meng Feixue 80042789 :250:60:50
4 Wu Waiwai 70271111 :250:80:75
5 Liu Bingbing 41117483 :250:100:175
6 Wang Xiaoai 3515064655 :50:95:135
7 Zi Gege 1986787350 :250:168:200
8 Li Youjiu 918391635 :175:75:300
9 Lao Nanhai 918391635 :250:100:175
eg:统计/etc/services文件中空行数量。
[root@oldboyedu-01 file]# awk '/^$/' /etc/services
[root@oldboyedu-01 file]# awk '/^$/{print NR}' /etc/services
22
266
299
320
326
393
461
474
479
486
494
506
512
518
583
584
[root@oldboyedu-01 file]# awk '/^$/{i=i+1;print i}' /etc/services
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@oldboyedu-01 file]# awk '/^$/{i=i+1}END{print i}' /etc/services
16
说明：
i=i+1 或者i++
用于统计出现多少次！计数！
4.awk数组-统计与计算
处理以下文件内容,将域名取出并根据域名进行计数排序处理:(百度和sohu面试题)
http://www.etiantian.org/index.html
http://www.etiantian.org/1.html
http://post.etiantian.org/index.html
http://mp3.etiantian.org/index.html
http://www.etiantian.org/3.html
http://post.etiantian.org/2.html
[root@oldboyedu-01 file]# awk 'BEGIN{h[110]="ge";h[112]="meng";print h[110],h[112]}'
ge meng
[root@oldboyedu-01 file]# awk -F "[/.]+" '{print $2}' url.txt
www
www
post
mp3
www
post
[root@oldboyedu-01 file]# awk -F "[/.]+" '{h[$2]=h[$2]+1;print h["www"]}' url.txt
1
2
2
2
3
3
[root@oldboyedu-01 file]# awk -F "[/.]+" '{h[$2]=h[$2]+1}END{print h["www"]}' url.txt
3
//法一：
[root@oldboyedu-01 file]# awk -F "[/.]+" '{h[$2]=h[$2]+1}END{print h["www"],h["post"],h["mp3"]}' url.txt
3 2 1
//法二：
[root@oldboyedu-01 file]# awk -F "[/.]+" '{h[$2]=h[$2]+1}END{for (pol in h)print pol}' url.txt
www
mp3
post
[root@oldboyedu-01 file]# awk -F "[/.]+" '{h[$2]=h[$2]+1}END{for (pol in h)print pol,h[pol]}' url.txt
www 3
mp3 1
post 2
总结：
1.awk数组进行统计
i=i+1 == i++ 统计次数 计数
i=i+$n == i+=$n 累计相加 累加
2.awk执行过程
3.awk通过正则作为模式
4.BEGIN和END特殊模式
secure系统日志分析练习
谁在破解你的密码（Failed password 每个ip地址出现的次数）
分析系统的每个用户被破解的次数
浙公网安备 33010602011771号