窗口同组联结(Window CoGroup)
窗口同组联结(window coGroup):将两条流合并之后开窗处理匹配的元素,调用时只需要将.join()换为.coGroup()就可以。
stream1.coGroup(stream2) .where(<KeySelector>) .equalTo(<KeySelector>) .window(TumblingEventTimeWindows.of(Time.hours(1))) .apply(<CoGroupFunction>)
CoGroupFunction函数
public interface CoGroupFunction<IN1, IN2, O> extends Function, Serializable { void coGroup(Iterable<IN1> first, Iterable<IN2> second, Collector<O> out) throws Exception; }
有三个参数, 分别代表两条流中的数据以及用于输出的收集器(Collector)。传入了可遍历的数据集合。也就是说,现在不会再去计算窗口中两条流数据集的笛卡尔积,而是直接把收集到的所有数据一次性传入,至于要怎样配对完全是自定义的。
public class CoGroupTest { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream<Tuple2<String, Long>> stream1 = env.fromElements(Tuple2.of("a", 1000L), Tuple2.of("b", 1000L), Tuple2.of("a", 2000L), Tuple2.of("b", 2000L)) .assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple2<String, Long>>forMonotonousTimestamps() .withTimestampAssigner(new SerializableTimestampAssigner<Tuple2<String, Long>>() { @Override public long extractTimestamp(Tuple2<String, Long> element, long recordTimestamp) { return element.f1; } })); DataStream<Tuple2<String, Long>> stream2 = env.fromElements(Tuple2.of("a", 3000L), Tuple2.of("b", 3000L), Tuple2.of("a", 4000L), Tuple2.of("b", 4000L)) .assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple2<String, Long>>forMonotonousTimestamps() .withTimestampAssigner(new SerializableTimestampAssigner<Tuple2<String, Long>>() { @Override public long extractTimestamp(Tuple2<String, Long> element, long recordTimestamp) { return element.f1; } })); stream1.coGroup(stream2) .where(stream1Tuple -> stream1Tuple.f0) .equalTo(stream2Tuple -> stream2Tuple.f0) .window(TumblingEventTimeWindows.of(Time.seconds(5))) .apply(new CoGroupFunction<Tuple2<String, Long>, Tuple2<String, Long>, String>() { @Override public void coGroup(Iterable<Tuple2<String, Long>> first, Iterable<Tuple2<String, Long>> second, Collector<String> out) throws Exception { out.collect("stream1:" + first + "==> stream2" + second); } }).print(); env.execute(); } }
立志如山 静心求实
浙公网安备 33010602011771号