Spider

流程：

迭代进行：种子url->Element遍历获取超链接lin->作为新种子id

⚠️注意频率和遍历深度

1.设定url及请求参数

 

headers = {"user-agent": "Baiduspider"}
base_url = "https://www.zhihu.com"
suffix = "/explore"
entry_url = base_url + suffix

 

2.读取请求返回结果，及解析

response = requests.get(entry_url, headers=headers).text
soup = bs4.BeautifulSoup(response, "lxml")
page_set = soup.find_all("a")

3.设定正则匹配规则，读取内容，取出结果

links = set()
for ele in page_set:
    raw_link = ele["href"]
    # print(f"raw_link = {raw_link}")
    proper_link = re.compile(r"^/question/.*").findall(raw_link)
    for final_link in proper_link:
        links.add(base_url + final_link)

 该流程仅遍历了一个页面的结果