Flink系列--Flink jdbc connector源码走读

Connector扩展说明

 

实现一个新的connector用户需要做的事情在上图中Planning部分，其中根据source还是sink来决定实现DynamicTableSourceFactory还是DynamicTableSinkFactory，也可以同时实现这两个接口。

以下针对JdbcDynamicTableFactory源码进行走读：

JdbcDynamicTableFactory

类主体结构

 

public static final String IDENTIFIER = "jdbc";

其他参数见类：JdbcConnectorOptions

public class JdbcConnectorOptions {

    public static final ConfigOption<String> URL =
            ConfigOptions.key("url")
                    .stringType()
                    .noDefaultValue()
                    .withDescription("The JDBC database URL.");

创建source动态表 ：createDynamicTableSource

JdbcDynamicTableSource

Lookup与scan接口对比请查看官网：

https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sourcessinks/#dynamic-table-source

lookup 简单理解为根据提交查找，不必读取整个表，并且可以在需要时从（可能不断变化的）外部表中延迟获取各个值。

scan 在运行时扫描来自外部存储系统的所有行

接下来就是构造如何读取数据库数据的select语句了

1、设置驱动、url，数据库用户名密码等信息

2、设置其他参数，fetchsize，limit，分区条件

3、根据方言构建select，设置数据库中数据类型与flink数据类型映射关系转换器

等

RuntimeProvider

 

 

build返回的为JdbcRowDataInputFormat

 public JdbcRowDataInputFormat build() {
            if (this.queryTemplate == null) {
                throw new NullPointerException("No query supplied");
            }
            if (this.rowConverter == null) {
                throw new NullPointerException("No row converter supplied");
            }
            if (this.parameterValues == null) {
                LOG.debug("No input splitting configured (data will be read with parallelism 1).");
            }
            return new JdbcRowDataInputFormat(
                    new SimpleJdbcConnectionProvider(connOptionsBuilder.build()),
                    this.fetchSize,
                    this.autoCommit,
                    this.parameterValues,
                    this.queryTemplate,
                    this.resultSetType,
                    this.resultSetConcurrency,
                    this.rowConverter,
                    this.rowDataTypeInfo);
        }

 

JdbcRowDataInputFormat

Connection获取

  Connection dbConn = connectionProvider.getOrEstablishConnection();

SimpleJdbcConnectionProvider

 @Override
    public Connection getOrEstablishConnection() throws SQLException, ClassNotFoundException {
        if (connection != null) {
            return connection;
        }
        if (jdbcOptions.getDriverName() == null) {
            connection =
                    DriverManager.getConnection(
                            jdbcOptions.getDbURL(),
                            jdbcOptions.getUsername().orElse(null),
                            jdbcOptions.getPassword().orElse(null));
        } else {
            Driver driver = getLoadedDriver();
            Properties info = new Properties();
            jdbcOptions.getUsername().ifPresent(user -> info.setProperty("user", user));
            jdbcOptions.getPassword().ifPresent(password -> info.setProperty("password", password));
            connection = driver.connect(jdbcOptions.getDbURL(), info);
            if (connection == null) {
                // Throw same exception as DriverManager.getConnection when no driver found to match
                // caller expectation.
                throw new SQLException(
                        "No suitable driver found for " + jdbcOptions.getDbURL(), "08001");
            }
        }
        return connection;
    }

读取一行数据

    /**
     * Stores the next resultSet row in a tuple.
     *
     * @param reuse row to be reused.
     * @return row containing next {@link RowData}
     * @throws IOException
     */
    @Override
    public RowData nextRecord(RowData reuse) throws IOException {
        try {
            if (!hasNext) {
                return null;
            }
            RowData row = rowConverter.toInternal(resultSet);
            // update hasNext after we've read the record
            hasNext = resultSet.next();
            return row;
        } catch (SQLException se) {
            throw new IOException("Couldn't read data - " + se.getMessage(), se);
        } catch (NullPointerException npe) {
            throw new IOException("Couldn't access resultSet", npe);
        }
    }

把resultSet转成rowData

AbstractJdbcRowConverter

  @Override
    public RowData toInternal(ResultSet resultSet) throws SQLException {
        GenericRowData genericRowData = new GenericRowData(rowType.getFieldCount());
        for (int pos = 0; pos < rowType.getFieldCount(); pos++) {
            Object field = resultSet.getObject(pos + 1);
            genericRowData.setField(pos, toInternalConverters[pos].deserialize(field));
        }
        return genericRowData;
    }

Dialect方言选择

根据URL 通过DialectFactory选择方言和driver，并通过继承AbstractDialect实现对应的方言

JdbcDialectFactory识别不同的URL，对应不同的数据库类型

 

JdbcDialect

方言的接口，提供类型验证和类型转换，和insert，update，delete等语句的构造

扩展FLink jdbc connector 支持其它类型的DB数据库

扩展jdbc connector支持其它数据库需要做如下工作

1、DialectFactory实现

2、如下截图中添加上述新的实现类

 

JdbcDynamicTableFactory如何生效呢？

FactoryUtil中类discoverFactories方法（java SPI机制）

static List<Factory> discoverFactories(ClassLoader classLoader) {
        final List<Factory> result = new LinkedList<>();
        ServiceLoaderUtil.load(Factory.class, classLoader)
                .forEach(
                        loadResult -> {
                            if (loadResult.hasFailed()) {
                                if (loadResult.getError() instanceof NoClassDefFoundError) {
                                    LOG.debug(
                                            "NoClassDefFoundError when loading a "
                                                    + Factory.class
                                                    + ". This is expected when trying to load a format dependency but no flink-connector-files is loaded.",
                                            loadResult.getError());
                                    // After logging, we just ignore this failure
                                    return;
                                }
                                throw new TableException(
                                        "Unexpected error when trying to load service provider for factories.",
                                        loadResult.getError());
                            }
                            result.add(loadResult.getService());
                        });
        return result;
    }