中秋放假期间运气真的不要太好,收到了交警送的“中秋礼物”,内心真的无法开心呀。大家或许都一样因为各种违章收到过交警的罚单,被扣分扣钱是我们不乐意的,所以我们在开车期间还是需要随时都牢记交通规则。

刚好最近初学Python,就想练练手,今天就学习下用Python爬虫自动化抓取违章信息,目标网站https://ah.122.gov.cn

基本代码如下:

import org.apache.commons.httpclient.Credentials;
import org.apache.commons.httpclient.HostConfiguration;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpMethod;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.GetMethod;

import java.io.IOException;

public class Main {
    # 代理服务器(产品官网 www.16yun.cn)
    private static final String PROXY_HOST = "t.16yun.cn";
    private static final int PROXY_PORT = 31111;

    public static void main(String[] args) {
        HttpClient client = new HttpClient();
        HttpMethod method = new GetMethod("https://httpbin.org/ip");

        HostConfiguration config = client.getHostConfiguration();
        config.setProxy(PROXY_HOST, PROXY_PORT);

        client.getParams().setAuthenticationPreemptive(true);

        String username = "16ABCCKJ";
        String password = "712323";
        Credentials credentials = new UsernamePasswordCredentials(username, password);
        AuthScope authScope = new AuthScope(PROXY_HOST, PROXY_PORT);

        client.getState().setProxyCredentials(authScope, credentials);

        try {
            client.executeMethod(method);

            if (method.getStatusCode() == HttpStatus.SC_OK) {
                String response = method.getResponseBodyAsString();
                System.out.println("Response = " + response);
            }
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            method.releaseConnection();
        }
    }
}

在整个抓取过程中遇到很多的问题,然后经过和朋友探讨都得到了解决,尤其是网站网ip,封爬虫行为。还有就是违章行为可大可小的,所以我们平时开车过程中必须时刻牢记交通规则。

posted on 2021-09-22 16:19  小橙子11  阅读(234)  评论(0编辑  收藏  举报