CoProcessFunction

       对于连接流ConnectedStreams 的处理操作,需要分别定义对两条流的处理转换,因此接口中就会有两个相同的方法需要实现,用数字“1”“2”区分,在两条流中的数据到来时分别调用。我们把这种接口叫作“协同处理函数”(co-process function)。与 CoMapFunction 类似,如果是调用.flatMap()就需要传入一个 CoFlatMapFunction,需要实现 flatMap1()、flatMap2()两个方法;而调用.process()时,传入的则是一个 CoProcessFunction。

public    abstract    class    CoProcessFunction<IN1,    IN2,    OUT>    extends AbstractRichFunction {
...

public abstract void processElement1(IN1 value, Context ctx, Collector<OUT> out) throws Exception;
public abstract void processElement2(IN2 value, Context ctx, Collector<OUT> out) throws Exception;
public void onTimer(long timestamp, OnTimerContext ctx, Collector<OUT> out) throws Exception {}
public abstract class Context {...}
...
}

  需要实现的就是 processElement1()、processElement2()两个方法,在每个数据到来时, 会根据来源的流调用其中的一个方法进行处理。CoProcessFunction 同样可以通过上下文 ctx 来访问 timestamp、水位线,并通过 TimerService 注册定时器;另外也提供了.onTimer()方法,用于定义定时触发的处理操作。

  实现一个实时对账的需求,也就是app 的支付操作和第三方的支付操作的一个双流 Join。App 的支付事件和第三方的支付事件将会互相等待 5 秒钟,如果等不来对应的支付事件,那么就输出报警信息。程序如下:

public class BillCheckTest {

    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        //平台支付
        SingleOutputStreamOperator<Tuple3<String, String, Long>> appStream = env.fromElements(Tuple3.of("order-1", "app", 1000L),
                Tuple3.of("order-2", "app", 1000L))
                .assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple3<String, String, Long>>forMonotonousTimestamps()
                        .withTimestampAssigner(new SerializableTimestampAssigner<Tuple3<String, String, Long>>() {
                            @Override
                            public long extractTimestamp(Tuple3<String, String, Long> element, long recordTimestamp) {
                                return element.f2;
                            }
                        }));

        //银行支付
        SingleOutputStreamOperator<Tuple4<String, String, String, Long>> bankStream = env.fromElements(
                Tuple4.of("order-1", "third-party", "success", 3000L),
                Tuple4.of("order-3", "third-party", "success", 4000L))
                .assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple4<String, String, String, Long>>forMonotonousTimestamps()
                        .withTimestampAssigner(new SerializableTimestampAssigner<Tuple4<String, String, String, Long>>() {
                            @Override
                            public long extractTimestamp(Tuple4<String, String, String, Long> element, long recordTimestamp) {
                                return element.f3;
                            }
                        }));
        //合并两个流
        appStream.connect(bankStream)
                .keyBy(appTuple -> appTuple.f0, bankTuple -> bankTuple.f0)
                .process(new CoProcessFunction<Tuple3<String, String, Long>, Tuple4<String, String, String, Long>, String>() {
                    //定义状态变量,保存已经到达的事件
                    private ValueState<Tuple3<String, String, Long>> appEventState;
                    private ValueState<Tuple4<String, String, String, Long>> thirdPartyEventState;

                    /**
                     * 状态的初始化
                     * @param parameters
                     * @throws Exception
                     */
                    @Override
                    public void open(Configuration parameters) throws Exception {
                        appEventState = getRuntimeContext().getState(
                                new ValueStateDescriptor<Tuple3<String, String, Long>>("app-event",
                                        Types.TUPLE(Types.STRING, Types.STRING, Types.LONG)));
                        thirdPartyEventState = getRuntimeContext().getState(
                                new ValueStateDescriptor<Tuple4<String, String, String, Long>>("bank-event",
                                        Types.TUPLE(Types.STRING, Types.STRING, Types.STRING, Types.LONG))
                        );
                    }

                    @Override
                    public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out) throws Exception {
                        // 定时器触发,判断状态,如果某个状态不为空,说明另一条流中事件没来
                        if (appEventState.value() != null) {
                            out.collect("对账失败:" + appEventState.value() + "银行支付平台信息未到");
                        }
                        if (thirdPartyEventState.value() != null) {
                            out.collect("对账失败:" + thirdPartyEventState.value() + "app信息未到");
                        }
                        appEventState.clear();
                        thirdPartyEventState.clear();
                    }

                    @Override
                    public void processElement1(Tuple3<String, String, Long> value, Context ctx, Collector<String> out) throws Exception {
                        if (thirdPartyEventState.value() != null) {
                            out.collect("对账成功" + value + "--" + thirdPartyEventState.value());
                            thirdPartyEventState.clear();
                        } else {
                            //更新状态
                            appEventState.update(value);
                            //注册一个 5 秒后的定时器,开始等待另一条流的事件
                            ctx.timerService().registerEventTimeTimer(value.f2 + 5000L);
                        }
                    }

                    @Override
                    public void processElement2(Tuple4<String, String, String, Long> value, Context ctx, Collector<String> out) throws Exception {
                        if (appEventState.value() != null) {
                            out.collect("对账成功" + value + "++" + appEventState.value());
                            appEventState.clear();
                        } else {
                            thirdPartyEventState.update(value);
                            ctx.timerService().registerEventTimeTimer(value.f3 + 5000L);
                        }
                    }
                }).print();

        env.execute();
    }
}

    App 的支付信息到达以后,会检查对应的第三方支付信息是否已经先到达(先到达会保存在对应的状态变量中),如果已经到达了,那么对账成功,直接输出对账成功的信息,并将保存第三方支付消息的状态变量清空。如果 App 对应的第三方支付信息没有到来,那么我们会注册一个 5 秒钟之后的定时器,也就是说等待第三方支付事件 5 秒钟。当定时器触发时,检查保存app 支付信息的状态变量是否还在,如果还在,说明对应的第三方支付信息没有到来,所以输出报警信息

posted on 2022-11-26 16:55  溪水静幽  阅读(375)  评论(0)    收藏  举报