Azkaban.Sqoop

1-Azkaban介绍

============================================================================================================================================================

2-Azkaban编译和安装模式

==========================================================================================================================================================

3-Azkaban-solo模式-安装

cd /export/softwares

tar ‐zxvf azkaban‐solo‐server‐0.1.0‐SNAPSHOT.tar.gz ‐C ../servers/

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

cd /export/servers/azkaban‐solo‐server‐0.1.0‐SNAPSHOT/conf

vim azkaban.properties

```shell

```properties

default.timezone.id=Asia/Shanghai

memCheck.enabled=false

cd /export/servers/azkaban‐solo‐server‐0.1.0‐SNAPSHOT

bin/start‐solo.sh

===========================================================================================================================================================

4-Azkaban-solo模式-使用

=============================================================================================================================================================

5-Azkaban-two_server模式-安装1

需要登录mysql: MYSQL的作用是用来存储azkaban中间产生的数据

CREATE DATABASE azkaban;

CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';

GRANT all privileges ON azkaban.* to 'azkaban'@'%' identified by

'azkaban' WITH GRANT OPTION;

flush privileges;

use azkaban;

source /export/software/create‐all‐sql‐0.1.0‐SNAPSHOT.sql;

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

第二步：centos6.9安装mysql

第一步：在线安装mysql相关的软件包

yum install mysql mysql-server mysql-devel

第二步：启动mysql的服务

/etc/init.d/mysqld start

第三步：通过mysql安装自带脚本进行设置

/usr/bin/mysql_secure_installation

第四步：进入mysql的客户端然后进行授权

grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;

flush privileges;

重新打开一个终端:

回到前一个终端:

cd /export/softwares

tar ‐zxvf azkaban‐web‐server‐0.1.0‐SNAPSHOT.tar.gz ‐C ../servers/

cd /export/servers

mv azkaban‐web‐server‐0.1.0‐SNAPSHOT/ azkaban‐web‐server‐3.51.0

* 解压azkaban‐exec‐server

```shell

cd /export/softwares

tar ‐zxvf azkaban‐exec‐server‐0.1.0‐SNAPSHOT.tar.gz ‐C ../servers/

cd /export/servers

mv azkaban‐exec‐server‐0.1.0‐SNAPSHOT/ azkaban‐exec‐server‐3.51.0

安装ssl安全认证，允许我们使用https的方式访问我们的azkaban的web服务(SSL为网络通信中的一种协议,保证网络通信的安全性;装了之后,以后访问azkaban的时候可以采用https这种方式访问)

密码一定要一个个的字母输入，或者粘贴也行

cd /export/servers/azkaban-web-server-3.51.0

keytool -keystore keystore -alias jetty -genkey -keyalg RSA

keytool为安全认证的工具

===============================================================================================================================================================

6-Azkaban-two_server模式-安装2

vim azkaban.properties

# Azkaban Personalization Settings

azkaban.name=Azkaban

azkaban.label=My Azkaban

azkaban.color=#FF3601

azkaban.default.servlet.path=/index

web.resource.dir=web/

default.timezone.id=Asia/Shanghai

# Azkaban UserManager class

user.manager.class=azkaban.user.XmlUserManager

user.manager.xml.file=conf/azkaban‐users.xml

# Loader for projects

executor.global.properties=conf/global.properties

azkaban.project.dir=projects

# Velocity dev mode

velocity.dev.mode=false

# Azkaban Jetty server properties.

jetty.use.ssl=true

jetty.maxThreads=25

jetty.port=8081

jetty.ssl.port=8443

jetty.keystore=/export/servers/azkaban‐web‐server‐3.51.0/keystore

jetty.password=azkaban

jetty.keypassword=azkaban

jetty.truststore=/export/servers/azkaban‐web‐server‐3.51.0/keystore

jetty.trustpassword=azkaban

# Azkaban Executor settings

# mail settings

mail.sender=

mail.host=

# User facing web server configurations used to construct the user facing

server URLs. They are useful when there is a reverse proxy between

Azkaban web servers and users.

# enduser ‐> myazkabanhost:443 ‐> proxy ‐> localhost:8081

# when this parameters set then these parameters are used to generate

email links.

# if these parameters are not set then jetty.hostname, and jetty.port(ifssl configured jetty.ssl.port) are used.

# azkaban.webserver.external_hostname=myazkabanhost.com

# azkaban.webserver.external_ssl_port=443

# azkaban.webserver.external_port=8081

job.failure.email=

job.success.email=

lockdown.create.projects=false

cache.directory=cache

# JMX stats

jetty.connector.stats=true

executor.connector.stats=true

# Azkaban mysql settings by default. Users should configure their own

username and password.

database.type=mysql

mysql.port=3306

mysql.host=node03

mysql.database=azkaban

mysql.user=azkaban

mysql.password=azkaban

mysql.numconnections=100

#Multiple Executor

azkaban.use.multiple.executors=true

#azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemo

ry,CpuStatus

azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1

azkaban.executorselector.comparator.Memory=1

azkaban.executorselector.comparator.LastDispatched=1

azkaban.executorselector.comparator.CpuUsage=1

azkaban.activeexecutor.refresh.milisecinterval=10000

azkaban.queueprocessing.enabled=true

azkaban.activeexecutor.refresh.flowinterval=10

azkaban.executorinfo.refresh.maxThreads=10

vim azkaban.properties

# Azkaban Personalization Settings

azkaban.name=Azkaban

azkaban.label=My Azkaban

azkaban.color=#FF3601

azkaban.default.servlet.path=/index

web.resource.dir=web/

default.timezone.id=Asia/Shanghai

# Azkaban UserManager class

user.manager.class=azkaban.user.XmlUserManager

user.manager.xml.file=conf/azkaban‐users.xml

# Loader for projects

executor.global.properties=conf/global.properties

azkaban.project.dir=projects

# Velocity dev mode

velocity.dev.mode=false

# Azkaban Jetty server properties.

jetty.use.ssl=true

jetty.maxThreads=25

jetty.port=8081

jetty.keystore=/export/servers/azkaban‐web‐server‐3.51.0/keystore

jetty.password=azkaban

jetty.keypassword=azkaban

jetty.truststore=/export/servers/azkaban‐web‐server‐3.51.0/keystore

jetty.trustpassword=azkaban

# Where the Azkaban web server is located

azkaban.webserver.url=https://node03:8443

# mail settings

mail.sender=

mail.host=

# User facing web server configurations used to construct the user facing

server URLs. They are useful when there is a reverse proxy between

Azkaban web servers and users.

# enduser ‐> myazkabanhost:443 ‐> proxy ‐> localhost:8081

# when this parameters set then these parameters are used to generate

email links.

# if these parameters are not set then jetty.hostname, and jetty.port(ifStep 6: azkaban executor server 安装第二步：添加插件

将我们编译后的C文件execute-as-user.c

上传到这个目录来/export/servers/azkaban-exec-server-3.51.0/plugins/jobtypes

或者直接将我们/export/softwares下面的文件拷贝过来也行

然后执行以下命令生成execute-as-user

ssl configured jetty.ssl.port) are used.

# azkaban.webserver.external_hostname=myazkabanhost.com

# azkaban.webserver.external_ssl_port=443

# azkaban.webserver.external_port=8081

job.failure.email=

job.success.email=

lockdown.create.projects=false

cache.directory=cache

# JMX stats

jetty.connector.stats=true

executor.connector.stats=true

# Azkaban plugin settings

azkaban.jobtype.plugin.dir=plugins/jobtypes

# Azkaban mysql settings by default. Users should configure their own

username and password.

database.type=mysql

mysql.port=3306

mysql.host=node03

mysql.database=azkaban

mysql.user=azkaban

mysql.password=azkaban

mysql.numconnections=100

# Azkaban Executor settings

executor.maxThreads=50

executor.flow.threads=30

============================================================================================================================================================

7-Azkaban-two_server模式-安装3和启动运行

/export/servers/azkaban-exec-server-3.51.0/plugins/jobtypes

yum ‐y install gcc‐c++

cd /export/servers/azkaban‐exec‐server‐3.51.0/plugins/jobtypes

gcc execute‐as‐user.c ‐o execute‐as‐user

chown root execute‐as‐user

chmod 6050 execute‐as‐user

cd /export/servers/azkaban‐exec‐server‐3.47.0/plugins/jobtypes

vim commonprivate.properties

execute.as.user=false

memCheck.enabled=false

azkaban.native.lib=/export/servers/azkaban‐exec‐server‐3.51.0/plugins/jobtypes

(1)第一步：启动azkaban exec server

cd /export/servers/azkaban‐exec‐server‐3.51.0

bin/start‐exec.sh

(2)第二步：激活我们的exec-server

node03机器任意目录下执行以下命令

curl ‐G "node03:$(<./executor.port)/executor?action=activate" && echo

(3)第三步：启动azkaban-web-server

cd /export/servers/azkaban‐web‐server‐3.51.0/

bin/start‐web.sh

============================================================================================================================================================

posted @ 2020-04-06 08:46 Curedfisher 阅读(153) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

Curedfisher

Azkaban.Sqoop

公告