flink项目脚手架
# Flink 项目开发指南
> 基于 dataact-access 项目。Flink 1.20 + JDK 11。
---
## 一、项目结构
```
your-flink-project/
├── pom.xml # 根 POM（聚合父模块）
├── module-common/ # 公共模块（工具类、框架基类，无 Flink 依赖）
│ ├── pom.xml
│ └── src/main/java/com/xxx/common/
│ ├── Main.java
│ ├── app/
│ │ └── StreamCommonFrame.java # Flink 作业抽象基类
│ └── utils/
│ ├── EnvironmentArgUtils.java # 命令行参数解析
│ ├── PropertyLoaderUtils.java # Properties 文件加载
│ ├── CustomLogger.java
│ ├── LogUtil.java
│ ├── DateTimeUtil.java
│ └── AESUtil.java
├── module-app/ # 业务模块（Kafka → ODPS/MySQL）
│ ├── pom.xml
│ └── src/main/
│ ├── java/com/xxx/access/
│ │ ├── Main.java
│ │ ├── app/ # Flink 作业入口
│ │ ├── bean/ # POJO 实体（含 conf/、enums/）
│ │ ├── source/ # 数据源
│ │ ├── func/ # Flink 处理函数（flatmap/、map/）
│ │ ├── sink/ # 自定义 Sink
│ │ ├── dao/
│ │ └── utils/
│ └── resources/
│ ├── log4j.properties
│ └── module-app-{env}.properties
└── module-ota/ # 业务模块（Pulsar → ODPS）
└── ... # 结构同上
```
| 模块 | 职责 | 数据源 → 目标 |
|------|------|--------------|
| `common` | 框架基类、工具类 | — |
| `app` | Kafka 消息处理业务 | Kafka → ODPS / MySQL |
| `ota` | Pulsar 消息处理业务 | Pulsar → ODPS |
---
## 二、Maven 多模块配置
### 2.1 根 POM
```xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="..." ...>
<modelVersion>4.0.0</modelVersion>
<groupId>com.yourcompany</groupId>
<artifactId>your-project-root</artifactId>
<packaging>pom</packaging>
<version>1.0-SNAPSHOT</version>
<modules>
<module>your-project-common</module>
<module>your-project-app</module>
<module>your-project-ota</module>
</modules>
<properties>
<java.version>11</java.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.30</version>
<scope>provided</scope>
</dependency>
</dependencies>
</project>
```
### 2.2 业务模块 POM（app 示例）
```xml
<parent>
<artifactId>your-project-root</artifactId>
<groupId>com.yourcompany</groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<artifactId>your-project-app</artifactId>
<properties>
<flink.version>1.20.0</flink.version>
<scala.binary.version>2.12</scala.binary.version>
</properties>
<dependencies>
<!-- Flink 核心 (provided：集群已有) -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-java-bridge</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients</artifactId>
<version>${flink.version}</version>
<scope>provided</scope>
</dependency>
<!-- 连接器 (打入 fat jar) -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka</artifactId>
<version>${flink.version}</version>
</dependency>
<!-- ODPS / MySQL 根据实际选择 -->
<!-- 模块间依赖 -->
<dependency>
<groupId>com.yourcompany</groupId>
<artifactId>your-project-common</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.11.0</version>
<configuration>
<release>${java.version}</release>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.5.1</version>
<executions>
<execution>
<phase>package</phase>
<goals><goal>shade</goal></goals>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>fat</shadedClassifierName>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<!-- IDEA 本地运行：将 provided 改为 compile -->
<profiles>
<profile>
<id>add-dependencies-for-IDEA</id>
<activation>
<property><name>idea.version</name></property>
</activation>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-api-java-bridge</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-table-planner_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
</dependencies>
</profile>
</profiles>
```
### 2.3 公共模块 POM
不依赖 Flink，仅工具类 + 基类：hutool、commons-codec、commons-lang3、log4j、lombok。
---
## 三、核心开发模式
### 3.1 框架基类 StreamCommonFrame
使用 **模板方法模式** 统一作业生命周期：
```java
@Data
public abstract class StreamCommonFrame implements Serializable {
Logger logger;
String configFile; // 子类设置，如 "my-app-%s.properties"
protected Properties props;
public void run(String[] args) {
// 1. 解析 -Denv=test 参数
String env = EnvironmentArgUtils.getEnv(args);
System.setProperty("env", env);
// 2. 加载对应环境的 Properties 文件
this.props = PropertyLoaderUtils.getInstance(
String.format(configFile, env)).getProperties();
// 3. 子类实现业务逻辑
process();
}
protected abstract void process();
}
```
### 3.2 作业入口模板
```java
@Slf4j
public class YourApp extends StreamCommonFrame {
static String configProps = "your-app-%s.properties";
public static void main(String[] args) {
YourApp app = new YourApp();
app.setConfigFile(configProps);
app.run(args);
}
@Override
protected void process() {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
// Source → Transformation → Sink
KafkaSource<String> source = KafkaSourceInstance.create(props);
DataStreamSource<String> ds = env.fromSource(source,
WatermarkStrategy.noWatermarks(), "kafka-source");
SingleOutputStreamOperator<YourBean> beanDs = ds
.map(new YourMapFunc())
.filter(new YourFilterFunc(props));
// 输出到 ODPS (Flink SQL)
registerOdpsTable(tableEnv, props);
tableEnv.createTemporaryView("v", beanDs);
tableEnv.executeSql("insert into odps_table select ... from v");
env.execute("YourApp");
}
}
```
### 3.3 三种输出模式
| 模式 | 适用场景 | 示例 |
|------|---------|------|
| **Flink SQL → ODPS** | MaxCompute 写入 | `tableEnv.executeSql("insert into odps_table select ... from v")` |
| **DataStream Sink → MySQL** | 自定义 JDBC 写入 | `dataStream.addSink(new CustomMysqlSink(props))` |
| **StatementSet** | 同时写入多表 | `statementSet.addInsertSql(sql1); statementSet.addInsertSql(sql2); statementSet.execute()` |
### 3.4 Source 封装（简洁版）
```java
// Kafka
public static KafkaSource<String> create(Properties props) {
return KafkaSource.<String>builder()
.setBootstrapServers(props.getProperty("bootstrap.server"))
.setTopics(props.getProperty("topic"))
.setGroupId(props.getProperty("group"))
.setStartingOffsets(OffsetsInitializer.committedOffsets(
"latest".equals(props.get("auto.offset.reset"))
? OffsetResetStrategy.LATEST : OffsetResetStrategy.EARLIEST))
.setValueOnlyDeserializer(new SimpleStringSchema())
.build();
}
// Pulsar
public static PulsarSource<String> create(PulsarParamConf conf) {
return PulsarSource.builder()
.setServiceUrl(conf.getPulsarUrl())
.setAdminUrl(conf.getAdminUrl())
.setTopics(conf.getTopic())
.setStartCursor(StartCursor.earliest())
.setDeserializationSchema(new SimpleStringSchema())
.setSubscriptionName(conf.getGroup())
.setConfig(PULSAR_ENABLE_AUTO_ACKNOWLEDGE_MESSAGE, true)
.build();
}
```
---
## 四、配置文件规范
多环境：`module-{env}.properties`（dev / test / prod）。
```properties
bootstrap.server=kafka-host:9092
your.topic=your-topic
your.group=your-group
auto.offset.reset=earliest
endPoint=http://service.region.maxcompute.aliyun-inc.com/api
project=your_project
accessId=xxx
accessKey=xxx
your.tableName=your_odps_table
```
加载方式：`PropertyLoaderUtils.getInstance("your-app-dev.properties").getProperties()`（单例，线程安全）。
---
## 五、Maven 常用命令
```bash
mvn clean package -pl your-project-app -am # 构建模块及依赖
mvn clean package -DskipTests # 跳过测试
mvn clean install -DskipTests # 安装到本地仓库
mvn dependency:tree -pl your-project-app # 查看依赖树
```
### 常见问题
| 问题 | 解决 |
|------|------|
| 找不到 common 类 | 先 `mvn install -pl common` 安装到本地 |
| Flink 类找不到（本地） | IDEA 自动激活 `add-dependencies-for-IDEA` profile |
| 子模块版本与父不同 | 子模块独立声明 `<version>` 即可 |
---
## 六、版本兼容性
| Flink | JDK | Scala | Kafka Connector | Pulsar Connector |
|-------|-----|-------|----------------|------------------|
| 1.20.0 | 11 | 2.12 | 1.20.0 | 4.1.0-1.20 |
启动参数：`-Denv=test`（必需），`-Dpm=3`（并行度，可选）。
浙公网安备 33010602011771号