爬虫 1 -IP代理池

爬虫 1 -IP代理池

IP代理池

[time=Tue, Jan 25, 2022 9:34 PM]
原文链接
dokcer-compose文件下载

# docker-compose 一键部署
docker-compose up

测试

随机一个代理IP http://ippool.gwozai.com/random
IP数量统计 http://ippool.gwozai.com/count

用python代码获取代理IP

import requests

proxypool_url = 'http://ippool.gwozai.com/random'
target_url = 'http://www.baidu.com'

def get_random_proxy():
    """
    get random proxy from proxypool
    :return: proxy
    """
    return requests.get(proxypool_url).text.strip()

def crawl(url, proxy):
    """
    use proxy to crawl page
    :param url: page url
    :param proxy: proxy, such as 8.8.8.8:8888
    :return: html
    """
    proxies = {'http': 'http://' + proxy}
    return requests.get(url, proxies=proxies).text


def main():
    """
    main method, entry point
    :return: none
    """
    proxy = get_random_proxy()
    print('get random proxy', proxy)
    html = crawl(target_url, proxy)
    print(html)

if __name__ == '__main__':
    main()