Flink - flink-connector-kafka-4.0.1

 

(airflow-venv) frank@ZZHPC:~/zproject$ flink run -py my_job.py --jar-file $FLINK_HOME/lib/flink-connector-kafka-4.0.1-2.0.jar
Traceback (most recent call last):
  File "/home/frank/zproject/my_job.py", line 12, in <module>
    kafka_source = KafkaSource.builder() \
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/frank/download/flink-2.0.1/opt/python/pyflink.zip/pyflink/datastream/connectors/kafka.py", line 387, in builder
  File "/home/frank/download/flink-2.0.1/opt/python/pyflink.zip/pyflink/datastream/connectors/kafka.py", line 430, in __init__
  File "/home/frank/download/flink-2.0.1/opt/python/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/home/frank/download/flink-2.0.1/opt/python/pyflink.zip/pyflink/util/exceptions.py", line 162, in deco
  File "/home/frank/download/flink-2.0.1/opt/python/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.flink.connector.kafka.source.KafkaSource.builder.
: java.lang.NoClassDefFoundError: org/apache/kafka/clients/consumer/OffsetResetStrategy
        at org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer.earliest(OffsetsInitializer.java:147)
        at org.apache.flink.connector.kafka.source.KafkaSourceBuilder.<init>(KafkaSourceBuilder.java:113)
        at org.apache.flink.connector.kafka.source.KafkaSource.builder(KafkaSource.java:141)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
        at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
        at org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
        at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.clients.consumer.OffsetResetStrategy
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
        ... 14 more

 

1. Download Flink connector for Kafka jar file from https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/4.0.1-2.0/flink-connector-kafka-4.0.1-2.0.jar .

2. Download Kafka client jar file from https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/4.0.1/kafka-clients-4.0.1.jar .

3. Put the two jar files to $FLINK_HOME/lib .

4. Restart Flink.