airflow安装部署

准备工作：

安装python36版本：

[root@master ~]# yum install python36u-pip python36 python36-libs -y

升级pip版本：

[root@master ~]# pip3.6 install --upgrade pip
[root@master ~]# pip --version
pip 20.3.3 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)
[root@master ~]# pip list
Package    Version
---------- -------
pip        20.3.3
setuptools 39.2.0

安装数据库：

[root@master ~]# yum -y install mariadb mariadb-server
修改数据库密码：
[root@master ~]# systemctl start mariadb
[root@master ~]# mysqladmin -uroot -p password '1213456';

安装依赖的软件包：

[root@master ~]# yum install gcc gcc-c++ cyrus-sasl cyrus-sasl-devel cyrus-sasl-lib -y

登录数据库：

[root@master ~]# mysql -uroot -p1213456

创建库和用户密码：

mysql> create database airflow;
Query OK, 1 row affected (0.00 sec)

 
mysql> create user 'airflow'@'%' identified by 'airflow';
Query OK, 0 rows affected (0.00 sec)

 
mysql> create user 'airflow'@'localhost' identified by 'airflow';
Query OK, 0 rows affected (0.00 sec)

 
mysql> grant all on airflow.* to 'airflow'@'%';
Query OK, 0 rows affected (0.00 sec)

 
mysql> grant all privileges on *.* to 'airflow'@'%';
Query OK, 0 rows affected (0.00 sec)

查询airflow的文件：

[root@master ~]# pip show --files apache-airflow

airflow安装的路径：

[root@master ~]# find / -name airflow
/var/lib/mysql/airflow

创建airflow的配置文件：

[root@master ~]# mkdir -p /etc/airflow
[root@master ~]# vim /etc/airflow/airflow.cfg
sql_alchemy_conn = mysql://airflow:airflow@localhost:3306/airflow

设置airflow的文件路径：

[root@master ~]# vim /etc/profile
export AIRFLOW_HOME=/etc/airflow
export SLUGIFY_USES_TEXT_UNIDECODE=yes

安装airflow软件：

[root@master ~]# pip install apache-airflow
[root@master ~]# pip install apache-airflow[devel]
[root@master ~]# pip install apache-airflow[celery]
[root@master ~]# pip install apache-airflow[jdbc]
[root@master ~]# pip install apache-airflow[mysql]
[root@master ~]# pip install apache-airflow[password]
[root@master ~]# pip install apache-airflow[rabbitmq]
[root@master ~]# pip install apache-airflow[redis]

查询版本号：

[root@master ~]# pip list | grep -i airflow
apache-airflow                  2.0.0

初始化报错：

[root@master ~]# airflow db init
DB: sqlite:////root/airflow/airflow.db
[2020-12-04 12:45:16,925] {db.py:678} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
WARNI [unusual_prefix_619bc6aee900a08efa12755006b4aa599654a401_example_kubernetes_executor_config] Could not import DAGs in example_kubernetes_executor_config.py: No module named 'kubernetes'
WARNI [unusual_prefix_619bc6aee900a08efa12755006b4aa599654a401_example_kubernetes_executor_config] Install kubernetes dependencies with: pip install apache-airflow['cncf.kubernetes']

解决问题：

[root@master ~]# pip install apache-airflow['cncf.kubernetes']
再次初始化：
[root@master ~]# airflow db init
DB: sqlite:////root/airflow/airflow.db
[2020-12-04 12:46:43,211] {db.py:678} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Initialization done

启动sirflow服务：

[root@master ~]# airflow webserver -p 8080 &
[1] 23474
   ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-12-04 12:50:19,220] {dagbag.py:440} INFO - Filling up the DagBag from /dev/null
[2020-12-04 12:50:19,294] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
Access Logformat:
=================================================================
[2020-12-04 12:50:37 +0800] [23608] [INFO] Starting gunicorn 19.10.0
[2020-12-04 12:50:37 +0800] [23608] [INFO] Listening at: http://0.0.0.0:8080 (23608)
[2020-12-04 12:50:37 +0800] [23608] [INFO] Using worker: sync
[2020-12-04 12:50:37 +0800] [23618] [INFO] Booting worker with pid: 23618
[2020-12-04 12:50:37 +0800] [23619] [INFO] Booting worker with pid: 23619
[2020-12-04 12:50:37 +0800] [23620] [INFO] Booting worker with pid: 23620
[2020-12-04 12:50:37 +0800] [23621] [INFO] Booting worker with pid: 23621
[2020-12-04 12:50:44,587] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
[2020-12-04 12:50:44,604] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
[2020-12-04 12:50:44,659] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
[2020-12-04 12:50:44,713] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
[2020-12-04 12:51:09 +0800] [23608] [INFO] Handling signal: ttin
[2020-12-04 12:51:09 +0800] [23820] [INFO] Booting worker with pid: 23820
[2020-12-04 12:51:11,327] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.