mama大数据常用命令

1、检查spark工作状态

[hadoop@logmaster spark-2.3.0-bin-hadoop2.7]$ tail -f logs/log_parser_py.log

还有一个就是查看日志

[hadoop@logmaster spark-2.3.0-bin-hadoop2.7]$ tail -f nohup.out 

查看日志文件，如果日志里面没有事实打印日志的话，那就要重启spark了

2、查看自定义命令

[hadoop@logmaster spark-2.3.0-bin-hadoop2.7]$ grephdports 
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:50070               0.0.0.0:*                   LISTEN      7389/java           
tcp        0      0 0.0.0.0:50010               0.0.0.0:*                   LISTEN      7535/java           
tcp        0      0 0.0.0.0:9083                0.0.0.0:*                   LISTEN      2966/java           
tcp        0      0 192.168.10.131:9000         0.0.0.0:*                   LISTEN      7389/java           
tcp        0      0 192.168.10.131:9001         0.0.0.0:*                   LISTEN      7717/java           
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      -                   
tcp        0      0 :::10000                    :::*                        LISTEN      4731/java           
tcp        0      0 ::ffff:192.168.10.131:8088  :::*                        LISTEN      7886/java           
tcp        0      0 ::ffff:192.168.10.131:8030  :::*                        LISTEN      7886/java           
tcp        0      0 ::ffff:192.168.10.131:8032  :::*                        LISTEN      7886/java           
tcp        0      0 ::ffff:192.168.10.131:8033  :::*                        LISTEN      7886/java           
tcp        0      0 ::ffff:192.168.10.131:8035  :::*                        LISTEN      7886/java           
tcp        0      0 :::9092                     :::*                        LISTEN      27861/java          
tcp        0      0 :::2181                     :::*                        LISTEN      27191/java          
tcp        0      0 :::8040                     :::*                        LISTEN      8002/java 

编辑自定义命令

[hadoop@logmaster spark-2.3.0-bin-hadoop2.7]$ cat ~/.bashrc 
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# User specific aliases and functions
alias stopssc="ss -tanlp |  grep 55197|awk '{print $6}'|awk  -F, '{print $2}'|xargs kill -15"
alias yarnlist="yarn application -list"
alias yarnkill="yarn application -kill"
alias grephdports="netstat -nltp | grep -E '3306|8020|8032|8030|8035|8033|8088|9001|9000|8040|50010|9083|2181|9092|2181|10000|50070'"
alias gkill="kill -15"
alias listunknows="hdfs dfs -ls /tmp/spark/unknown_logs | sort -u"
alias reloadqueueset="yarn rmadmin -refreshQueues"
alias grep="grep --color=auto"

3、启动spark任务

[hadoop@logmaster spark-2.3.0-bin-hadoop2.7]$ vi ./run_log_parser.sh 

4、查看spark应用程序情况

[hadoop@logmaster spark-2.3.0-bin-hadoop2.7]$ yarn application -list
18/09/10 17:29:10 INFO client.RMProxy: Connecting to ResourceManager at logmaster/192.168.10.131:8032
Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):3
                Application-Id      Application-Name        Application-Type          User           Queue                   State             Final-State             Progress                        Tracking-URL
application_1531272653589_0086          PySparkShell                   SPARK        hadoop       root.dev2                 RUNNING               UNDEFINED                  10%               http://logmaster:4042
application_1531272653589_0074             LogParser                   SPARK        hadoop       root.dev1                 RUNNING               UNDEFINED                  10%               http://logmaster:4041
application_1531272653589_0041  Thrift JDBC/ODBC Server                SPARK        hadoop      root.default               RUNNING               UNDEFINED                  10%               http://logmaster:4040

5、查看Hadoop的hdfs目录

[hadoop@logmaster spark-2.3.0-bin-hadoop2.7]$ hdfs dfs -ls /
Found 4 items
drwxr-xr-x   - hadoop supergroup          0 2018-06-27 12:51 /data
drwxr-xr-x   - hadoop supergroup          0 2018-09-10 18:01 /eventLogs
drwx-wx-wx   - hadoop supergroup          0 2018-06-27 15:57 /tmp
drwxr-xr-x   - hadoop supergroup          0 2018-06-15 01:56 /user

或者

[hadoop@logmaster data]$ hadoop fs -ls /
Found 4 items
drwxr-xr-x   - hadoop supergroup          0 2018-06-27 12:51 /data
drwxr-xr-x   - hadoop supergroup          0 2018-09-10 18:01 /eventLogs
drwx-wx-wx   - hadoop supergroup          0 2018-06-27 15:57 /tmp
drwxr-xr-x   - hadoop supergroup          0 2018-06-15 01:56 /user

6、hive的命令

7、查看yarn的任务

[hadoop@logmaster spark-2.3.0-bin-hadoop2.7]$ yarn application -list