中秋放假期间运气真的不要太好,收到了交警送的“中秋礼物”,内心真的无法开心呀。大家或许都一样因为各种违章收到过交警的罚单,被扣分扣钱是我们不乐意的,所以我们在开车期间还是需要随时都牢记交通规则。
刚好最近初学Python,就想练练手,今天就学习下用Python爬虫自动化抓取违章信息,目标网站https://ah.122.gov.cn。
基本代码如下:
import org.apache.commons.httpclient.Credentials;
import org.apache.commons.httpclient.HostConfiguration;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpMethod;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.commons.httpclient.methods.GetMethod;
import java.io.IOException;
public class Main {
# 代理服务器(产品官网 www.16yun.cn)
private static final String PROXY_HOST = "t.16yun.cn";
private static final int PROXY_PORT = 31111;
public static void main(String[] args) {
HttpClient client = new HttpClient();
HttpMethod method = new GetMethod("https://httpbin.org/ip");
HostConfiguration config = client.getHostConfiguration();
config.setProxy(PROXY_HOST, PROXY_PORT);
client.getParams().setAuthenticationPreemptive(true);
String username = "16ABCCKJ";
String password = "712323";
Credentials credentials = new UsernamePasswordCredentials(username, password);
AuthScope authScope = new AuthScope(PROXY_HOST, PROXY_PORT);
client.getState().setProxyCredentials(authScope, credentials);
try {
client.executeMethod(method);
if (method.getStatusCode() == HttpStatus.SC_OK) {
String response = method.getResponseBodyAsString();
System.out.println("Response = " + response);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
method.releaseConnection();
}
}
}
在整个抓取过程中遇到很多的问题,然后经过和朋友探讨都得到了解决,尤其是网站网ip,封爬虫行为。还有就是违章行为可大可小的,所以我们平时开车过程中必须时刻牢记交通规则。
浙公网安备 33010602011771号