CoProcessFunction
对于连接流ConnectedStreams 的处理操作,需要分别定义对两条流的处理转换,因此接口中就会有两个相同的方法需要实现,用数字“1”“2”区分,在两条流中的数据到来时分别调用。我们把这种接口叫作“协同处理函数”(co-process function)。与 CoMapFunction 类似,如果是调用.flatMap()就需要传入一个 CoFlatMapFunction,需要实现 flatMap1()、flatMap2()两个方法;而调用.process()时,传入的则是一个 CoProcessFunction。
public abstract class CoProcessFunction<IN1, IN2, OUT> extends AbstractRichFunction { ... public abstract void processElement1(IN1 value, Context ctx, Collector<OUT> out) throws Exception; public abstract void processElement2(IN2 value, Context ctx, Collector<OUT> out) throws Exception; public void onTimer(long timestamp, OnTimerContext ctx, Collector<OUT> out) throws Exception {} public abstract class Context {...} ... }
需要实现的就是 processElement1()、processElement2()两个方法,在每个数据到来时, 会根据来源的流调用其中的一个方法进行处理。CoProcessFunction 同样可以通过上下文 ctx 来访问 timestamp、水位线,并通过 TimerService 注册定时器;另外也提供了.onTimer()方法,用于定义定时触发的处理操作。
实现一个实时对账的需求,也就是app 的支付操作和第三方的支付操作的一个双流 Join。App 的支付事件和第三方的支付事件将会互相等待 5 秒钟,如果等不来对应的支付事件,那么就输出报警信息。程序如下:
public class BillCheckTest { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); //平台支付 SingleOutputStreamOperator<Tuple3<String, String, Long>> appStream = env.fromElements(Tuple3.of("order-1", "app", 1000L), Tuple3.of("order-2", "app", 1000L)) .assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple3<String, String, Long>>forMonotonousTimestamps() .withTimestampAssigner(new SerializableTimestampAssigner<Tuple3<String, String, Long>>() { @Override public long extractTimestamp(Tuple3<String, String, Long> element, long recordTimestamp) { return element.f2; } })); //银行支付 SingleOutputStreamOperator<Tuple4<String, String, String, Long>> bankStream = env.fromElements( Tuple4.of("order-1", "third-party", "success", 3000L), Tuple4.of("order-3", "third-party", "success", 4000L)) .assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple4<String, String, String, Long>>forMonotonousTimestamps() .withTimestampAssigner(new SerializableTimestampAssigner<Tuple4<String, String, String, Long>>() { @Override public long extractTimestamp(Tuple4<String, String, String, Long> element, long recordTimestamp) { return element.f3; } })); //合并两个流 appStream.connect(bankStream) .keyBy(appTuple -> appTuple.f0, bankTuple -> bankTuple.f0) .process(new CoProcessFunction<Tuple3<String, String, Long>, Tuple4<String, String, String, Long>, String>() { //定义状态变量,保存已经到达的事件 private ValueState<Tuple3<String, String, Long>> appEventState; private ValueState<Tuple4<String, String, String, Long>> thirdPartyEventState; /** * 状态的初始化 * @param parameters * @throws Exception */ @Override public void open(Configuration parameters) throws Exception { appEventState = getRuntimeContext().getState( new ValueStateDescriptor<Tuple3<String, String, Long>>("app-event", Types.TUPLE(Types.STRING, Types.STRING, Types.LONG))); thirdPartyEventState = getRuntimeContext().getState( new ValueStateDescriptor<Tuple4<String, String, String, Long>>("bank-event", Types.TUPLE(Types.STRING, Types.STRING, Types.STRING, Types.LONG)) ); } @Override public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out) throws Exception { // 定时器触发,判断状态,如果某个状态不为空,说明另一条流中事件没来 if (appEventState.value() != null) { out.collect("对账失败:" + appEventState.value() + "银行支付平台信息未到"); } if (thirdPartyEventState.value() != null) { out.collect("对账失败:" + thirdPartyEventState.value() + "app信息未到"); } appEventState.clear(); thirdPartyEventState.clear(); } @Override public void processElement1(Tuple3<String, String, Long> value, Context ctx, Collector<String> out) throws Exception { if (thirdPartyEventState.value() != null) { out.collect("对账成功" + value + "--" + thirdPartyEventState.value()); thirdPartyEventState.clear(); } else { //更新状态 appEventState.update(value); //注册一个 5 秒后的定时器,开始等待另一条流的事件 ctx.timerService().registerEventTimeTimer(value.f2 + 5000L); } } @Override public void processElement2(Tuple4<String, String, String, Long> value, Context ctx, Collector<String> out) throws Exception { if (appEventState.value() != null) { out.collect("对账成功" + value + "++" + appEventState.value()); appEventState.clear(); } else { thirdPartyEventState.update(value); ctx.timerService().registerEventTimeTimer(value.f3 + 5000L); } } }).print(); env.execute(); } }
App 的支付信息到达以后,会检查对应的第三方支付信息是否已经先到达(先到达会保存在对应的状态变量中),如果已经到达了,那么对账成功,直接输出对账成功的信息,并将保存第三方支付消息的状态变量清空。如果 App 对应的第三方支付信息没有到来,那么我们会注册一个 5 秒钟之后的定时器,也就是说等待第三方支付事件 5 秒钟。当定时器触发时,检查保存app 支付信息的状态变量是否还在,如果还在,说明对应的第三方支付信息没有到来,所以输出报警信息