Shell：使用管道实现数据整理 | 正则表达式

示例1：截取http响应报文中的信息

karinto@server:/home/karinto$ curl --head --silent www.baidu.com > baidu #head请求(不返回请求对象), 将得到的http响应报文信息输出到baidu文件中
karinto@server:/home/karinto$ cat baidu
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform
Connection: keep-alive
Content-Length: 277
Content-Type: text/html
Date: Fri, 10 Mar 2023 09:23:47 GMT
Etag: "575e1f59-115"
Last-Modified: Mon, 13 Jun 2016 02:50:01 GMT
Pragma: no-cache
Server: bfe/1.0.8.18
karinto@server:/home/karinto$ curl --head --silent www.baidu.com | grep --ignore-case content-length #--ignore-case 忽略大小写
Content-Length: 277
karinto@server:/home/karinto$ curl --head --silent www.baidu.com | grep --ignore-case content-length | cut --delimiter=' ' -f1 #以' '字符为分割点, 截取第一段
Content-Length:
karinto@server:/home/karinto$ curl --head --silent www.baidu.com | grep --ignore-case content-length | cut --delimiter=' ' -f2 #截取第二段
277

 

示例2：使用根用户权限执行tee命令追加信息至文件

karinto@server:~$ sudo echo happy >> baidu #追加重定向, 使用root权限运行echo程序, 在baidu文件只能由根用户操作时报错
karinto@server:~$ echo happy | sudo tee -a baidu #双向重定向, -a 追加, 使用root权限运行tee程序, 在baidu文件只能由跟用户操作时不报错, tee支持多文件写入
happy
karinto@server:~$ cat baidu
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform
Connection: keep-alive
Content-Length: 277
Content-Type: text/html
Date: Fri, 10 Mar 2023 09:49:50 GMT
Etag: "575e1f59-115"
Last-Modified: Mon, 13 Jun 2016 02:50:01 GMT
Pragma: no-cache
Server: bfe/1.0.8.18

happy
happy

　　

示例3：seq命令配合正则表达式替换数据的某一部分

karinto@server:~$ cat log
Jan 10 23:09:16 thesquareplanet.com sshd[20093]: Disconnected from user jon 24.61.9.143 port 51086 [preauth]
Jan 13 15:00:14 thesquareplanet.com sshd[25065]: Disconnected from invalid user ftp_test 190.187.67.67 port 50091 [preauth]
Jan 17 03:13:00 thesquareplanet.com sshd[2631]: Disconnected from invalid user matt from 46.97.239.16 port 55920 [preauth]
karinto@server:~$ cat log | sed -E 's/^.*Disconnected from (invalid |authenticating )?user (.*) [0-9.]+ port [0-9]+( \[preauth\])?$/\2/'
jon
ftp_test
matt from

#sed语法格式:
sed -E 's/aa/bb/' filename #将每行的第一个匹配的aa替换为bb
#正则表达式思路: 
#任何圆括号()括起来的表达式称为捕获组, 共有两组, 该正则表达式将第二组\2提取出来, 即提取username, 替换掉原有行输出

　　

 附：Regular expressions notes

abc…    Letters
123…    Digits
\d      Any Digit
\D      Any Non-digit character
.       Any Character
\.      Period
[abc]   Only a, b, or c
[^abc]  Not a, b, nor c
[a-z]   Characters a to z
[0-9]   Numbers 0 to 9
\w      Any Alphanumeric character
\W      Any Non-alphanumeric character
{m}     m Repetitions
{m,n}   m to n Repetitions
*       Zero or more repetitions
+       One or more repetitions
?       Optional character
\s      Any Whitespace
\S      Any Non-whitespace character
^…$     Starts and ends
(…)     Capture Group
(a(bc)  Capture Sub-group
(.*)    Capture all
(abc|def)    Matches abc or def