如何将Flume kafka source 插件从0.9升级到0.10.1

在使用flume的kafka插件为0.9版本，由于channel队列满后，导致心跳问题，进而导致flume kafka不接收消息，

所以打算升级到0.10版本，fluem首先需要做如下：

将flume中的lib包中kafka 0.9相关升级到0.10，（前提，服务端已经升级至0.10）即：

a）kafka_2.10-0.9.0.1.jar -> kafka_2.10-0.10.0.1.jar

b）kafka-clients-0.9.0.1.jar -> kafka-clients-0.10.0.1.jar

 

替换了jar之后，启动flume，报如下异常

09 Jan 2018 14:29:36,149 ERROR [lifecycleSupervisor-1-3] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:251)  - Unable to start PollableSourceRunner: { source:org.apache.flume.source.kafka.KafkaSource{name:r1,state:IDLE} counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/kafka/clients/consumer/ConsumerRebalanceListener;)V
        at org.apache.flume.source.kafka.KafkaSource$TopicListSubscriber.subscribe(KafkaSource.java:152)
        at org.apache.flume.source.kafka.KafkaSource.doStart(KafkaSource.java:516)
        at org.apache.flume.source.BasicSourceSemantics.start(BasicSourceSemantics.java:83)
        at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

 

对KafkaSource源码分析发现，kafka在0.10的时将Consumer接口的subscription方法进行了改变：

老的0.9版本（使用反编译工具看的）

public abstract interface Consumer<K, V> extends Closeable
{
  public abstract Set<TopicPartition> assignment();

  public abstract Set<String> subscription();

  public abstract void subscribe(List<String> paramList);

  public abstract void subscribe(List<String> paramList, ConsumerRebalanceListener paramConsumerRebalanceListener);

  public abstract void assign(List<TopicPartition> paramList);

  public abstract void subscribe(Pattern paramPattern, ConsumerRebalanceListener paramConsumerRebalanceListener);

  public abstract void unsubscribe();
...

 

而新0.10 Consumer接口如下：

public interface Consumer<K, V> extends Closeable {

    /**
     * @see KafkaConsumer#assignment()
     */
    public Set<TopicPartition> assignment();

    /**
     * @see KafkaConsumer#subscription()
     */
    public Set<String> subscription();

    /**
     * @see KafkaConsumer#subscribe(Collection)
     */
    public void subscribe(Collection<String> topics);

    /**
     * @see KafkaConsumer#subscribe(Collection, ConsumerRebalanceListener)
     */
    public void subscribe(Collection<String> topics, ConsumerRebalanceListener callback);
　　...

可见接口中使用了更通用的Collection作为参数。

但按理讲List集成了Collection，所以原有List应该不用修改可直接访问。（public interface List<E> extends Collection<E> ），但实际却是报错。具体原有见下：

 

现在的问题转换为如果依赖的第三方jar更改了，那么你的项目需不需要重新编译的问题？

通过求证得到如下结果：

1）如果依赖的接口发生变化，那么需要重新编译

2）如果依赖的常量发生变化，也需要重新编译

（参考：https://stackoverflow.com/questions/536971/do-i-have-to-recompile-my-application-when-i-upgrade-a-third-party-jar）

可见Consumer接口发生变化，所以需要重新编译。

将对应的flume源码得到，将flume-parent中的pom.xml kakfa依赖修改如下

    <!-- <kafka.version>0.9.0.1</kafka.version> -->
    <kafka.version>0.10.0.1</kafka.version>

重新编译对flume-kafka-source进行打包，更新原flume lib下flume-kafka-source-1.7.0.jar，重启，一切正常。

 

总结：

1.flume kafka从0.9升级到0.10时，首先将依赖的kafka jar包升级

2.需要依赖0.10对flume-kafka-source项目重新编译