SpringBoot整合WebMagic一次踩坑记录：java.lang.IncompatibleClassChangeError: null

今天第一次做Springboot项目整合webMagic项目时。刚刚创建的一个新的项目启动时突然发现报错了！！！

报错如下：

java.lang.IncompatibleClassChangeError: null
at us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler.push(DuplicateRemovedScheduler.java:35) ~[classes/:na]
at us.codecraft.webmagic.Spider.setScheduler(Spider.java:197) ~[classes/:na]
at com.itheima.utils.JobProcessor.startSpider(JobProcessor.java:106) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_131]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_131]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

我疯狂百度呀，找资料呀也找不出个所以然来。

异步任务的代码：

package com.wangjie.crawlerjob.task;

import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import us.codecraft.webmagic.Page;
import us.codecraft.webmagic.Site;
import us.codecraft.webmagic.Spider;
import us.codecraft.webmagic.processor.PageProcessor;
import us.codecraft.webmagic.scheduler.BloomFilterDuplicateRemover;
import us.codecraft.webmagic.scheduler.QueueScheduler;
import us.codecraft.webmagic.selector.Html;

/**
 * @author 落魄书生
 * @data 2020/9/23 - 10:50
 */
@Component
public class JobProcessor implements PageProcessor {

    private String url = "https://search.51job.com/list/000000,000000,0000,01%252c32,9,99,java,2,1.html?lang=c&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line=&welfare=";


    @Override
    public void process(Page page) {
        Html html = page.getHtml();
        System.out.println(html.toString());

    }


    private Site site = Site.me()
            .setCharset("gbk") // 设置编码
            .setTimeOut(10000) // 设置超时时间
            .setRetrySleepTime(3000) // 设置 重试的间隔时间
            .setRetryTimes(3); // 设置重试的次数
    
    @Override
    public Site getSite() {
        return site;
    }

    /**
     * initialDelay: 当项目启动之后 隔多久运行这个定时任务
     * fixedDelay：  每隔多久执行这个方法
     */
    @Scheduled(initialDelay = 1000, fixedDelay = 100 * 1000)
    public void process() {

        Spider.create(new JobProcessor())
                .addUrl(url)
                .setScheduler(new QueueScheduler().setDuplicateRemover(new BloomFilterDuplicateRemover(100000)))
                .thread(10)
                .run();


    }


}

 

后来我就试试换了一个SpringBoot的版本。和把一些乱七八糟的依赖删掉。把版本降低到2.0.2版本。然后抱着试试的心态竟然成功了！神奇

我又尝试把版本换回2.3.4版本。发现它又行了。然后就解决。问题是啥我一脸懵逼，解决后我还是一脸懵。自己技术还是太菜了，努力学习吧，骚年~