scrapy基础知识之随机切换fake-useragent 库的使用：

pip install fake-useragent

from fake_useragent import UserAgent
ua = UserAgent()

middlewares.py

from fake_useragent import UserAgent

class RandomUserAgent(object):
    # def __init__(self,crawl):
    #     super(RandomUserAgent,self).__init__()
    #     self.ua=UserAgent()
    def process_request(self, request, spider):
        #useragent = random.choice(USER_AGENTS)
        ua=UserAgent()
        request.headers.setdefault("User-Agent",ua.random)

settints.py

DOWNLOADER_MIDDLEWARES = {
      'LaGou.middlewares.RandomUserAgent': 1,
#    'LaGou.middlewares.MyCustomDownloaderMiddleware': 543,
}

 更多访问地址：https://github.com/hellysmile/fake-useragent