work in while 

url='http://www.baidu.com'
while True:
    resp=requests.get(url).content.decode()
    selector=etree.HTML(resp)
    self.parse_data(selector)

    next_url=''.join(selector.xpath("//div[@class='next']/text()"))
    if next_url:
        url=next_url
    else:
        break
posted @ 2020-03-03 23:31  hank-li  阅读(108)  评论(0)  编辑  收藏  举报