软件安装路径
| 软件名称 | 版本 | 安装路径 |
| jdk | 1.6.0_12 |
/usr/java/jdk1.6.0_12 |
| maven | 3.1.0 |
/usr/local//apache-maven-3.1.0 |
| Oozie | 3.3.1 |
/usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1 |
| hadoop | 1.0.3 |
/usr/local/hadoop-1.0.3 |
| mysql | 5.5.7 |
/usr/local/mysql |
环境变量.bash_profile
export HADOOP_HOME="/home/tnuser/hadoop" export HADOOP_PREFIX=$HADOOP_HOME export HBASE_HOME="/home/tnuser/hbase" export HIVE_HOME="/home/tnuser/hive" export OOZIE_HOME="/home/tnuser/oozie" export JAVA_HOME="/usr/java/jdk1.6.0_12" export JRE_HOME=$JAVA_HOME/jre export MAVEN_HOME="/usr/local/apache-maven-3.1.0" export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JRE_HOME/lib export PATH=$JAVA_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$SQOOP_HOME/bin:$PIG_HOME/bin:$HIVE_HOME/bin:$MAVEN_HOME/bin:$OOZIE_HOME/bin:/usr/bin:/sbin:/usr/sbin/:$PATH export PIG_CLASSPATH=/usr/java/jdk1.6.0_12/lib/tools.jar:$PIG_CLASSPATH export HIVE_LIB=$HIVE_HOME/lib/hive-hbase-handler-0.10.0.jar,$HIVE_HOME/lib/hbase-0.92.1.jar,$HIVE_HOME/lib/zookeeper-3.4.3.jar,$HIVE_HOME/lib/guava-r09.jar export "OOZIE_URL=http://localhost:11000/oozie"
家目录设置
$ ls -lh /home/tnuser/ total 0 lrwxrwxrwx 1 tnuser appuser 39 Jun 11 03:29 hadoop -> /usr/local/hadoop-1.0.3 lrwxrwxrwx 1 tnuser appuser 39 Jun 11 00:42 hbase -> /usr/local/hbase-0.92.1 lrwxrwxrwx 1 tnuser appuser 31 Jun 11 00:42 hdfs -> /usr/local/data drwxr-xr-x 4 tnuser appuser 45 Jun 11 00:42 logs lrwxrwxrwx 1 tnuser appuser 83 Jun 17 18:45 oozie -> /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1
1. maven安装
$ wget https://archive.apache.org/dist/maven/maven-3/3.1.0/binaries/apache-maven-3.1.0-bin.tar.gz $ tar -zxf apache-maven-3.1.0-bin.tar.gz
2.hadoop安装
略
3.mysql安装
略
4. 生成Oozie安装包
$ wget https://archive.apache.org/dist/oozie/3.3.1/oozie-3.3.1.tar.gz $ tar -zxf oozie-3.3.1.tar.gz $ cd /usr/local/contentplatform/oozie-3.3.1/bin $ ./mkdistro.sh -DskipTests
报错及解决
报错
ERROR] Failed to execute goal on project oozie-sharelib-hive: Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hive:jar:3.3.1: Failed to collect dependencies at org.apache.hive:hive-cli:jar:0.9.0 -> org.apache.hive:hive-metastore:jar:0.9.0 -> javax.jdo:jdo2-api:jar:2.3-ec: Failed to read artifact descriptor for javax.jdo:jdo2-api:jar:2.3-ec: Could not transfer artifact javax.jdo:jdo2-api:pom:2.3-ec from/to Codehaus repository (http://repository.codehaus.org/): repository.codehaus.org: Unknown host repository.codehaus.org -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :oozie-sharelib-hive ERROR, Oozie distro creation failed
解决
将安装根目录的pom.xml如下代码
<repository> <id>Codehaus repository</id> <url>http://repository.codehaus.org/</url> <snapshots> <enabled>false</enabled> </snapshots> </repository>
替换为
<repository> <id>Codehaus repository</id> <name>codehaus-mule-repo</name> <url>https://repository-master.mulesoft.org/nexus/content/groups/public/</url> <layout>default</layout> </repository>
重新运行
./mkdistro.sh -DskipTests
出现以下结果说明生成安装包成功
Oozie distro created, DATE[2019.06.18-01:40:02GMT] VC-REV[unavailable], available at [/usr/local/oozie-3.3.1/distro/target]
5.安装Oozie
5.1 启用Oozie Web Console
wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip
解压到$OOZIE_HOME根目录
5.2 复制$HADOOP_HOME以及$HADOOP_HOME/lib所有jar包到$OOZIE_HOME/libext
$ mkdir $OOZIE_HOME/libext $ scp $HADOOP_HOME/*.jar $OOZIE_HOME/libext $ scp $HADOOP_HOME/lib/*.jar $OOZIE_HOME/libext
5.3 添加mysql数据库驱动到Oozie
$ wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz $ mv mysql-connector-java-5.1.47.tar.gz $SQOOP_HOME/libext
5.4 安装Oozie
$ oozie-setup.sh -hadoop 0.20.2 $HADOOP_HOME -extjs ext-2.2
输出log
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" INFO: Adding extension: /home/tnuser/oozie/libext/asm-3.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/aspectjrt-1.6.5.jar INFO: Adding extension: /home/tnuser/oozie/libext/aspectjtools-1.6.5.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-1.7.0.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-core-1.8.0.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-cli-1.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-codec-1.4.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-collections-3.2.1.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-configuration-1.6.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-daemon-1.0.1.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-digester-1.8.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-el-1.0.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-httpclient-3.0.1.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-lang-2.4.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-1.1.1.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-api-1.0.4.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-math-2.1.jar INFO: Adding extension: /home/tnuser/oozie/libext/commons-net-1.4.1.jar INFO: Adding extension: /home/tnuser/oozie/libext/core-3.1.1.jar INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-ant-1.0.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-capacity-scheduler-1.0.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-client-1.0.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-core-1.0.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-examples-1.0.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-fairscheduler-1.0.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-minicluster-1.0.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-test-1.0.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-thriftfs-1.0.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-tools-1.0.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/hsqldb-1.8.0.10.jar INFO: Adding extension: /home/tnuser/oozie/libext/jackson-core-asl-1.8.8.jar INFO: Adding extension: /home/tnuser/oozie/libext/jackson-mapper-asl-1.8.8.jar INFO: Adding extension: /home/tnuser/oozie/libext/jasper-compiler-5.5.12.jar INFO: Adding extension: /home/tnuser/oozie/libext/jasper-runtime-5.5.12.jar INFO: Adding extension: /home/tnuser/oozie/libext/jdeb-0.8.jar INFO: Adding extension: /home/tnuser/oozie/libext/jersey-core-1.8.jar INFO: Adding extension: /home/tnuser/oozie/libext/jersey-json-1.8.jar INFO: Adding extension: /home/tnuser/oozie/libext/jersey-server-1.8.jar INFO: Adding extension: /home/tnuser/oozie/libext/jets3t-0.6.1.jar INFO: Adding extension: /home/tnuser/oozie/libext/jetty-6.1.26.jar INFO: Adding extension: /home/tnuser/oozie/libext/jetty-util-6.1.26.jar INFO: Adding extension: /home/tnuser/oozie/libext/jsch-0.1.42.jar INFO: Adding extension: /home/tnuser/oozie/libext/junit-4.5.jar INFO: Adding extension: /home/tnuser/oozie/libext/kfs-0.2.2.jar INFO: Adding extension: /home/tnuser/oozie/libext/log4j-1.2.16.jar INFO: Adding extension: /home/tnuser/oozie/libext/mockito-all-1.8.5.jar INFO: Adding extension: /home/tnuser/oozie/libext/mysql-connector-java-5.1.24-bin.jar INFO: Adding extension: /home/tnuser/oozie/libext/oro-2.0.8.jar INFO: Adding extension: /home/tnuser/oozie/libext/servlet-api-2.5-20081211.jar INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-api-1.4.3.jar INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-log4j12-1.4.3.jar INFO: Adding extension: /home/tnuser/oozie/libext/xmlenc-0.52.jar Injecting following Hadoop JARs /home/tnuser/hadoop/hadoop-core-1.0.3.jar New Oozie WAR file with added 'Hadoop JARs, ExtJS library, JARs' at /home/tnuser/oozie/oozie-server/webapps/oozie.war INFO: Oozie is ready to be started
5.5 生成Oozie数据库初始化文件
$ ooziedb.sh create -sqlfile oozie.sql
5.6 将初始化文件导入mysql数据库
CREATE DATABASE oozie_data DEFAULT CHARACTER SET uft8 ; CREATE USER 'oozie'@'*' IDENTIFIED BY 'oozie'; GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'*' use oozie; source oozie.sql;
5.7 配置hadoop core-site.xml，添加以下参数，如之前未添加，需要重启hadoop
<property> <name>hadoop.proxyuser.tnuser.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.tnuser.groups</name> <value>*</value> </property>
5.8 启动Oozie
$ oozie-start.sh $ netstat -antl | grep 11000 tcp 0 0 0.0.0.0:11000 0.0.0.0:* LISTEN $ oozie admin -oozie http://localhost:11000/oozie -status System mode: NORMAL
5.9 将Oozie共享库导入hadoop
$ cd $OOZIE_HOME $ tar xvfz oozie-sharelib-3.3.1.tar.gz $ HADOOP_HOME/bin/hadoop fs -put share share $ hadoop fs -ls /user/tnuser/share/lib Found 7 items drwxr-xr-x - tnuser supergroup 0 2019-06-19 06:07 /user/tnuser/share/lib/distcp drwxr-xr-x - tnuser supergroup 0 2019-06-19 06:07 /user/tnuser/share/lib/hive drwxr-xr-x - tnuser supergroup 0 2019-06-19 06:07 /user/tnuser/share/lib/mapreduce-streaming drwxr-xr-x - tnuser supergroup 0 2019-06-19 06:07 /user/tnuser/share/lib/oozie drwxr-xr-x - tnuser supergroup 0 2019-06-19 06:07 /user/tnuser/share/lib/pig -rw-r--r-- 3 tnuser supergroup 1354 2019-06-19 06:07 /user/tnuser/share/lib/sharelib.properties drwxr-xr-x - tnuser supergroup 0 2019-06-19 06:07 /user/tnuser/share/lib/sqoop
6. 测试Oozie
6.1 修改example包文件对应属性值
$ cd $OOZIE_HOME $ tar -zxf oozie-examples.tar.gz $ ls -lh examples/apps/map-reduce/ total 8.0K -rw-r--r-- 1 tnuser appuser 1021 Jun 20 06:00 job.properties drwxr-xr-x 2 tnuser appuser 38 Jun 19 07:41 lib -rw-r--r-- 1 tnuser appuser 2.5K Jan 17 2013 workflow.xml
将job.properties文件namenode和jobtracker地址修改为实际的地址
$ cat examples/apps/map-reduce/job.properties nameNode=hdfs://ec2d-newcntprocnn-01:9000 jobTracker=ec2d-newcntprocnn-01:9001 queueName=default examplesRoot=examples oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce outputDir=map-reduce
6.2 上传到hdfs
$ hadoop fs -put examples examples $ hadoop fs -ls /user/tnuser Found 2 items drwxr-xr-x - tnuser supergroup 0 2019-06-20 07:15 /user/tnuser/examples drwxr-xr-x - tnuser supergroup 0 2019-06-19 06:07 /user/tnuser/share
6.3 测试Oozie
报错
$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: User: tnuser is not allowed to impersonate tnuser] $ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: Unauthorized connection for super-user: tnuser from IP 10.189.102.176]
经测试以下hdfs参数的value值为“*“会出现以上报错，需要修改hdfs参数为对应的真实值
<property> <name>hadoop.proxyuser.tnuser.hosts</name> <value>ec2d-newcntprocdn-04</value> </property> <property> <name>hadoop.proxyuser.tnuser.groups</name> <value>appuser</value> </property>
正常运行结果
$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run job: 0000000-190620070628363-oozie-tnus-W $ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W Job ID : 0000000-190620070628363-oozie-tnus-W ------------------------------------------------------------------------------------------------------------------------------------ Workflow Name : map-reduce-wf App Path : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce Status : RUNNING Run : 0 User : tnuser Group : - Created : 2019-06-20 14:14 GMT Started : 2019-06-20 14:14 GMT Last Modified : 2019-06-20 14:14 GMT Ended : - CoordAction ID: - Actions ------------------------------------------------------------------------------------------------------------------------------------ ID Status Ext ID Ext Status Err Code ------------------------------------------------------------------------------------------------------------------------------------ 0000000-190620070628363-oozie-tnus-W@:start: OK - OK - ------------------------------------------------------------------------------------------------------------------------------------ 0000000-190620070628363-oozie-tnus-W@mr-node START_RETRY- - JA009 ------------------------------------------------------------------------------------------------------------------------------------ $ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W Job ID : 0000000-190620070628363-oozie-tnus-W ------------------------------------------------------------------------------------------------------------------------------------ Workflow Name : map-reduce-wf App Path : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce Status : SUCCEEDED Run : 0 User : tnuser Group : - Created : 2019-06-20 14:14 GMT Started : 2019-06-20 14:14 GMT Last Modified : 2019-06-20 14:16 GMT Ended : 2019-06-20 14:16 GMT CoordAction ID: - Actions ------------------------------------------------------------------------------------------------------------------------------------ ID Status Ext ID Ext Status Err Code ------------------------------------------------------------------------------------------------------------------------------------ 0000000-190620070628363-oozie-tnus-W@:start: OK - OK - ------------------------------------------------------------------------------------------------------------------------------------ 0000000-190620070628363-oozie-tnus-W@mr-node OK job_201906200714_0002 SUCCEEDED - ------------------------------------------------------------------------------------------------------------------------------------ 0000000-190620070628363-oozie-tnus-W@end OK - OK - ------------------------------------------------------------------------------------------------------------------------------------
===================来自一泽涟漪的博客,转载请标明出处 www.cnblogs.com/ilifeilong===================
浙公网安备 33010602011771号