scrapy splash 实现下滑加载

实现滚轴下拉加载页面的splash script(Lua 脚本)

 

方法1

function main(splash, args)
  splash:set_viewport_size(1028, 10000)
  splash:go(args.url)
  local scroll_to = splash:jsfunc("window.scrollTo")
  scroll_to(0, 2000)
  splash:wait(5)
  return {png=splash:png()}
end

 

方法2

function main(splash, args)
  splash:set_viewport_size(1028, 10000)
  splash:go(args.url)
  splash.scroll_position={0,2000}
  splash:wait(5)
  return {png=splash:png()}
end

 

爬虫实现下滑加载

 

    def start_requests(self):
        script = """
                function main(splash)
                    splash:set_viewport_size(1028, 10000)
                    splash:go(splash.args.url)
                    local scroll_to = splash:jsfunc("window.scrollTo")
                    scroll_to(0, 2000)
                    splash:wait(15)
                    return {
                        html = splash:html()
                    }
                end
                """
 
        for url in self.start_urls:
            yield Request(url,callback=self.parse_info_index,meta = {
                'dont_redirect': True,
                'splash':{
                    'args':{'lua_source':script,'images':0},
                    'endpoint':'execute',
 
                }
            })