6 Kafka Streams Joins

Joining means taking KStream and / or KTable and creating a new KStream or KTable from it

There are 4 kind of joins (SQL-like)

https://docs.confluent.io/current/streams/developer-guide/dsl-api.html#joining

 

Joins Constraints - Co-Partition of Data

This can only be happened when the data is co-partitioned. Otherwise the join won't be doable and Kafka streams will fail with a Runtime Error.

co-partition means that the same number of partition is there on the stream and / or table

For example we have two topics, test1 and test2, both of them must have the same number of Partitions.

If not, Kafka Streams will not able to join them using join operation. The only way is, that you write one of this data into another one, which has the same number of partitions.

GlobalKTable

If your KTable data is reasonably small, and can fit on each of your Kafka Streams application, you can read it is as GlobalKTable.

With GlobalKTables, you can join any stream to your table even if the data doesn't have the same number of partition. -> it is like distributed cache in classical Hadoop MapReduce. B

Different Types of Joins

There are 3 Kafka objects, which could be joint:

• KStream - KStream
• KTable - KTable
• KStream - KTable

Inner Join

Join the data only if it has matches in both of data

 

Left Join

Join all the data from the left whether or not it has a match on the right

 

Outer Join

• Only available for KStream / KStream joins
• It's a left join combined with a right join
• From the API doc, it looks like this: