pipelinewise 学习二 创建一个简单的pipeline
pipelinewise 提供了方便的创建简单pipeline的命令，可以简化pipeline 的创建，同时也可以帮我们学习
生成demo pipeline
pipelinewise init --name pipelinewise_samples
效果 
一个简单的pg 2 pg 的demo
- 生成tap 以及target yaml 配置
直接使用的demo 配置文件
cp tap_postgres.yml.sample tap_postgres.yml
cp target_postgres.yml.sample target_postgres.yml
- 修改配置
tap
---
# ------------------------------------------------------------------------------
# General Properties
# ------------------------------------------------------------------------------
id: "postgres_sample" # Unique identifier of the tap
name: "Sample Postgres Database" # Name of the tap
type: "tap-postgres" # !! THIS SHOULD NOT CHANGE !!
owner: "somebody@foo.com" # Data owner to contact
# ------------------------------------------------------------------------------
# Source (Tap) - PostgreSQL connection details
# ------------------------------------------------------------------------------
db_conn:
host: "localhost" # PostgreSQL host
port: 15432 # PostgreSQL port
user: "pipelinewise" # PostfreSQL user
password: "secret" # Plain string or vault encrypted
dbname: "postgres_source_db" # PostgreSQL database name
#filter_schemas: "schema1,schema2" # Optional: Scan only the required schemas
# to improve the performance of
# data extraction
# ------------------------------------------------------------------------------
# Destination (Target) - Target properties
# Connection details should be in the relevant target YAML file
# ------------------------------------------------------------------------------
target: "postgres_dwh" # ID of the target connector where the data will be loaded
batch_size_rows: 20000 # Batch size for the stream to optimise load performance
# ------------------------------------------------------------------------------
# Source to target Schema mapping
# ------------------------------------------------------------------------------
schemas:
- source_schema: "public" # Source schema in postgres with tables
target_schema: "repl_pg_public" # Target schema in the destination Data Warehouse
target_schema_select_permissions: # Optional: Grant SELECT on schema and tables that created
- grp_stats
# List of tables to replicate from Postgres to destination Data Warehouse
#
# Please check the Replication Strategies section in the documentation to understand the differences.
# For LOG_BASED replication method you might need to adjust the source mysql/ mariadb configuration.
tables:
- table_name: "city"
replication_method: "FULL_TABLE" # One of INCREMENTAL, LOG_BASED and FULL_TABLE
replication_key: "last_update" # Important: Incremental load always needs replication key
# OPTIONAL: Load time transformations
#transformations:
# - column: "last_name" # Column to transform
# type: "SET-NULL" # Transformation type
# You can add as many tables as you need...
- table_name: "country"
replication_method: "FULL_TABLE" # Important! Log based must be enabled in PostgreSQL
# You can add as many schemas as you need...
# Uncommend this if you want replicate tables from multiple schemas
#- source_schema: "another_schema_in_postgres"
# target_schema: "another
target
---
# ------------------------------------------------------------------------------
# General Properties
# ------------------------------------------------------------------------------
id: "postgres_dwh" # Unique identifier of the target
name: "Postgres Data Warehouse" # Name of the target
type: "target-postgres" # !! THIS SHOULD NOT CHANGE !!
# ------------------------------------------------------------------------------
# Target - Data Warehouse connection details
# ------------------------------------------------------------------------------
db_conn:
host: "localhost" # Postgres host
port: 15433 # Postgres port
user: "pipelinewise" # Postgres user
password: "secret" # Plain string or vault encrypted
dbname: "postgres_dwh" # Postgres database name
激活pipeline
- 激活部署的服务
pipelinewise import --dir pipelinewise_samples
效果
2019-09-17 05:07:55 INFO: Searching YAML config files in /app/wrk
2019-09-17 05:07:55 INFO: LOADING TARGET: target_postgres.yml
2019-09-17 05:07:55 INFO: LOADING TAP: tap_postgres.yml
2019-09-17 05:07:55 INFO: SAVING CONFIG
2019-09-17 05:07:55 INFO: SAVING MAIN CONFIG JSON to /root/.pipelinewise/config.json
2019-09-17 05:07:55 INFO: SAVING TARGET JSONS to /root/.pipelinewise/postgres_dwh/config.json
2019-09-17 05:07:55 INFO: SAVING TAP JSONS to /root/.pipelinewise/postgres_dwh/postgres_sample
2019-09-17 05:07:55 INFO: ACTIVATING TAP STREAM SELECTIONS
[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 4 concurrent workers.
2019-09-17 05:07:55 INFO: Discovering postgres_sample (tap-postgres) tap in postgres_dwh (target-postgres) target
2019-09-17 05:07:56 INFO: Loading pre defined selection from /root/.pipelinewise/postgres_dwh/postgres_sample/selection.json
2019-09-17 05:07:56 INFO: Mark postgres_source_db-public-edgydata tap_stream_id as not selected
2019-09-17 05:07:56 INFO: Mark postgres_source_db-public-city tap_stream_id as selected with properties {'replication_method': 'FULL_TABLE', 'tap_stream_id': 'postgres_source_db-public-city'}
2019-09-17 05:07:56 INFO: Mark postgres_source_db-public-country tap_stream_id as selected with properties {'replication_method': 'FULL_TABLE', 'tap_stream_id': 'postgres_source_db-public-country'}
2019-09-17 05:07:56 INFO: Mark postgres_source_db-public-countrylanguage tap_stream_id as not selected
2019-09-17 05:07:56 INFO: Loading pre defined selection from /root/.pipelinewise/postgres_dwh/postgres_sample/selection.json
2019-09-17 05:07:56 INFO: Mark postgres_source_db-public-edgydata tap_stream_id as not selected
2019-09-17 05:07:56 INFO: Mark postgres_source_db-public-city tap_stream_id as selected with properties {'replication_method': 'FULL_TABLE', 'tap_stream_id': 'postgres_source_db-public-city'}
2019-09-17 05:07:56 INFO: Mark postgres_source_db-public-country tap_stream_id as selected with properties {'replication_method': 'FULL_TABLE', 'tap_stream_id': 'postgres_source_db-public-country'}
2019-09-17 05:07:56 INFO: Mark postgres_source_db-public-countrylanguage tap_stream_id as not selected
2019-09-17 05:07:56 INFO: Writing new properties file with changes into /root/.pipelinewise/postgres_dwh/postgres_sample/properties.json
[Parallel(n_jobs=-1)]: Done 1 tasks | elapsed: 0.3s
[Parallel(n_jobs=-1)]: Done 1 out of 1 | elapsed: 0.3s finished
2019-09-17 05:07:56 INFO:
-------------------------------------------------------
IMPORTING YAML CONFIGS FINISHED
-------------------------------------------------------
Total targets to import : 1
Total taps to import : 1
Taps imported successfully : 1
Taps failed to import : []
Runtime : 0:00:00.409421
-------------------------------------------------------
- 查看状态
pipelinewise status
效果
Tap ID Tap Type Target ID Target Type Enabled Status Last Sync Last Sync Result
--------------- ------------ ------------ --------------- --------- -------- ----------- ------------------
postgres_sample tap-postgres postgres_dwh target-postgres True ready unknown
运行pipeline
- 执行命令
pipelinewise run_tap --tap postgres_sample --target postgres_dwh
效果：
2019-09-17 05:08:36 INFO: Running postgres_sample tap in postgres_dwh target
2019-09-17 05:08:36 INFO: No table available that needs to be sync by fastsync
2019-09-17 05:08:36 INFO: Table(s) selected to sync by singer: ['postgres_source_db-public-city', 'postgres_source_db-public-country']
2019-09-17 05:08:36 INFO: Writing output into /root/.pipelinewise/postgres_dwh/postgres_sample/log/postgres_dwh-postgres_sample-20190917_050836.singer.log
数据库效果 
- 查看状态
pipelinewise status
参考资料
https://transferwise.github.io/pipelinewise/installation_guide/creating_pipelines.html
https://transferwise.github.io/pipelinewise/installation_guide/running_pipelines.html
