爬虫笔记之teambition登录验证码
一、缘起
想做的事情太多，计划乱糟糟，想找个工具理一下，想起来了的很久之前用过teambition，打算看一下，然后在登录界面看到一个比较有意思的验证码：
这种倒是比较有意思哈，看着像是模仿12306的那种，12306的破不了（我真人都要刷几次才能对。。。），这个简单版的还破不了吗，于是激发了我强烈的破解兴趣。
二、分析
打开开发者工具，先选中看一下先：
首先比较雷的是“地球”竟然是文本显示在页面上的，这就比较尴尬了，不过其实这个无所谓，即使是图片也没关系，这里的重点是要每次返回的都有所区分（区分度越大越安全，否则使用使用一些基于统计的方式很容易就能够破掉），否则的话会被以比较低的成本作为一个标识，然后就是那几张图片的显示，里面有个uid，然后还有个index，那么这两个变量是从哪里来的呢，点击刷新按钮，然后观察网络请求会发现有几个：
这个uid和value的数组下标一拼装就是页面上显示的图标的url，至此看起来没啥毛病。
然后就是考虑如何破解的问题了，我看这几个图标画的如此清新脱俗，应该是手工画的，既然是手工画的，那么其数量应该是有限的，最多几百个吧，那么完全可以采用打标签的方式来，但是打标签的话几百个也是太多了，而且只是手动打标签识别这种平平无奇的做法，也不值一提了，这有一种无须手动打标签的方式，就是上面的接口中，“地球”所对应的图片一定在下面的values数组中，而我只需要对这个接口多请求几次，然后对它们按照imageName分组，比如“地球”这个分组会对应着很多个values，每个values中都有一张图片是真的“地球”，哪张是呢，所有的values的交集就是，这样进行一个group by imageName --> mapGroup求分组内values交集 --> 得到一个imageName对应的图片的特征，这个就作为模型，识别的时候只需要根据imageName取出模型中对应的图片特征，然后破解时从新请求返回的values找到哪张图片的特征是能够对应上的，就实现了从imageName到图片的识别。
三、编码实现
首先请求获取验证码的接口，得到一批图片：
package cc11001100.misc.crawler.captcha.teambition;
import cc11001100.misc.crawler.utils.HttpUtil;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import org.jsoup.Connection;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
/**
*
* 下载一些原始的验证码图片以用作分析
*
* https://account.teambition.com/login
*
* @author CC11001100
*/
public class TeambitionCaptchaCrawler {
public static void handleSingleCaptcha(String captchaResponseJsonStr) throws IOException {
JSONObject responseJson = JSON.parseObject(captchaResponseJsonStr);
String uid = responseJson.getString("uid");
JSONArray values = responseJson.getJSONArray("values");
for (int i = 0; i < values.size(); i++) {
String url = "https://auth_services.teambition.com/captcha/image?uid=" + uid + "&lang=zh&index=" + i;
byte[] captchaBytes = HttpUtil.request(url, null, Connection.Response::bodyAsBytes);
String outputLocation = "data/captcha/teambition/captcha-imgs/" + values.getString(i) + ".png";
IOUtils.write(captchaBytes, new FileOutputStream(outputLocation));
}
}
public static void downloadRawData() throws IOException {
for (int i = 0; i < 10000; i++) {
String url = "https://auth_services.teambition.com/captcha/setup?num=5&lang=zh&_=" + System.currentTimeMillis();
String responseBody = HttpUtil.request(url, connection -> {
connection.userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36");
}, Connection.Response::body);
FileUtils.writeStringToFile(new File("data/captcha/teambition/raw-captcha.jsonl"), responseBody + "\n", "UTF-8", true);
handleSingleCaptcha(responseBody);
}
}
public static void main(String[] args) throws IOException {
downloadRawData();
}
}
得到一批验证码的原始图片：
raw-captcha.json:
{"values":["f38d8ea6e916762a57be2108c7c0b29c027650f3","cc6fc6dd8ef8f17b7fa99f44dfaa65df221af4f4","801d66a2673d0ba30ba9d02412d2321e1ba1de94","66cecc1d439a74d10a6a2d628f2a358fa90df1df","5d8029335a0330a1ff9f0c5766f3502dbba1ad1f"],"imageName":"飞机","uid":"27419090-0b57-11ea-8fd8-e31d1dab49d9"}
{"values":["f54416fb674ae827c35c004e35057bdbc14fc0fe","ca1cc93bdc4236889e154b7542bf5675c7ffedc0","368a7a84132450e43f5020802b397b071dfe7840","0a449565c02a16adbe17b799c30947c8c904ad73","29d6a3b0ff1c42af3a9bb47fd759872a5e8f5931"],"imageName":"锁","uid":"27852940-0b57-11ea-a598-c15471c1be2e"}
{"values":["84ba343b153e45c4c9aae9b260cfefa297587eda","fffe2e988c105b50c902d6372340a306abda0ce5","04ab1f94a0a73fdbf37d6aa40beb9878ef737c8f","ec2b8e8612ced4cd43f656bce3050dc3ef58c656","e61b104c2e7fbc1c6ab1dfa1b2a23f2c6fde1680"],"imageName":"相机","uid":"27bab830-0b57-11ea-8fd8-e31d1dab49d9"}
{"values":["295da38531279e1938aa4a87f354f27f62feb159","b3ed0b06f53373ba6d8ffdccea65e807d26b53e6","f6baa4dcea4f4d845f4d13301a39dbbe9bbe9fe9","61ce83c8c500d9cf96b79e34e9147993a0c6b359","93244a67523c11554f3ce8b49950d8fcbdbbf8fd"],"imageName":"锁","uid":"27ecebc0-0b57-11ea-8fd8-e31d1dab49d9"}
{"values":["b2c5fa8ec472914cbdaff3d790fc0eb0c8a45adf","5ebd5d74430628a34fb189e9efc919c12afa069e","369adc4406bd49e7876e760b3e13dcc2637daa87","d664b3e2334880096f88fd50349e7d5b9e4e0fcd","cef15533490a94c27eeb8ae8dd97efb47ba717ce"],"imageName":"相机","uid":"282587f0-0b57-11ea-8fd8-e31d1dab49d9"}
...然后就是刚才从刚才下载到的验证码图片中生成imageName到图片特征的一个map：
package cc11001100.misc.crawler.captcha.teambition;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.StringUtils;
import javax.imageio.ImageIO;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
/**
* @author CC11001100
*/
@Slf4j
public class DerivationLabel {
// 将用到的图片集找出来并打个标注
private static void derivationLabel() throws IOException {
Map<String, Integer> imageNameToHashCodeMap = new HashMap<>();
FileUtils.readLines(new File("data/captcha/teambition/raw-captcha.jsonl"), "UTF-8").stream()
.filter(StringUtils::isNotBlank)
.collect(Collectors.groupingBy(line -> JSON.parseObject(line).getString("imageName")))
.forEach((imageName, lineList) -> {
Set<Integer> interceptingSet = new HashSet<>();
for (String line : lineList) {
JSONArray values = JSON.parseObject(line).getJSONArray("values");
Set<Integer> currentSet = new HashSet<>();
// 下载的时候有几次强制中断观察效果，所以一组values的图片可能会下得不全，不全的这种就直接忽略掉了
boolean hasError = false;
for (int i = 0; i < values.size(); i++) {
String f = "data/captcha/teambition/captcha-imgs/" + values.getString(i) + ".png";
try {
currentSet.add(ImageUtil.hash(ImageIO.read(new FileInputStream(f))));
} catch (Exception e) {
log.error("Exception, path=" + f, e);
hasError = true;
break;
}
}
if (hasError) {
return;
}
if (interceptingSet.isEmpty()) {
interceptingSet.addAll(currentSet);
} else {
interceptingSet.retainAll(currentSet);
if (interceptingSet.isEmpty()) {
log.info("数据不足，imageName={}", imageName);
break;
}
}
}
if (interceptingSet.size() != 1) {
log.info("imageName={}, derivation failed", imageName);
} else {
log.info("imageName={}, set={}", imageName, interceptingSet);
imageNameToHashCodeMap.put(imageName, interceptingSet.iterator().next());
}
});
imageNameToHashCodeMap.forEach((k, v) -> System.out.printf("map.put(\"%s\", %d);\n", k, v));
}
public static void main(String[] args) throws IOException {
derivationLabel();
}
}
这里对图片的特征就是取hash值，用到的工具类如下：
package cc11001100.misc.crawler.captcha.teambition;
import lombok.extern.slf4j.Slf4j;
import java.awt.image.BufferedImage;
/**
* @author CC11001100
*/
@Slf4j
public class ImageUtil {
public static int hash(BufferedImage image) {
StringBuilder msg = new StringBuilder();
for (int i = 0; i < image.getWidth(); i++) {
for (int j = 0; j < image.getHeight(); j++) {
msg.append(image.getRGB(i, j)).append("|");
}
}
return msg.toString().hashCode();
}
}
生成的map如下：
map.put("音符", 182834422);
map.put("锁", 825168351);
map.put("机器人", -1714422141);
map.put("汽车", -769011042);
map.put("钥匙", 975258806);
map.put("树叶", -179264444);
map.put("信封", -702966573);
map.put("相机", 663652535);
map.put("文件夹", -1425863546);
map.put("云朵", 2106631124);
map.put("飞机", 1640044711);
map.put("T恤", -258338857);
map.put("眼睛", 1647675580);
map.put("树", -2063289315);
map.put("放大镜", -1715725768);
map.put("闹钟", 1335715652);
map.put("回形针", 1654053339);
map.put("地球", -1592219546);
map.put("脚印", -1438760947);
map.put("标签", 761482882);
map.put("剪刀", 1998833602);
map.put("灯泡", 418507311);
map.put("伞", -2104015908);
map.put("图表", -824773152);
map.put("气球", 1423728112);
map.put("太阳眼镜", 1204904862);
map.put("椅子", 193112560);
map.put("打印机", -939522792);
map.put("旗帜", 834329993);
map.put("猫", 1911236121);
map.put("女人", 2047088238);
map.put("男人", 664214693);
map.put("卡车", -1453025175);
map.put("电脑", -1970735883);
map.put("裤子", -337658120);
map.put("铅笔", 1993614559);
map.put("房子", -1299209990);然后就是识别部分了，这里只是将答案打印出来，并不提交，提交的话短信就真的发出去了：
package cc11001100.misc.crawler.captcha.teambition;
import cc11001100.misc.crawler.utils.HttpUtil;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import lombok.Builder;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.jsoup.Connection;
import javax.imageio.ImageIO;
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
/**
* @author CC11001100
*/
@Slf4j
public class TeambitionCaptchaCracker {
private static final Map<String, Integer> map = new HashMap<>();
static {
map.put("音符", 182834422);
map.put("锁", 825168351);
map.put("机器人", -1714422141);
map.put("汽车", -769011042);
map.put("钥匙", 975258806);
map.put("树叶", -179264444);
map.put("信封", -702966573);
map.put("相机", 663652535);
map.put("文件夹", -1425863546);
map.put("云朵", 2106631124);
map.put("飞机", 1640044711);
map.put("T恤", -258338857);
map.put("眼睛", 1647675580);
map.put("树", -2063289315);
map.put("放大镜", -1715725768);
map.put("闹钟", 1335715652);
map.put("回形针", 1654053339);
map.put("地球", -1592219546);
map.put("脚印", -1438760947);
map.put("标签", 761482882);
map.put("剪刀", 1998833602);
map.put("灯泡", 418507311);
map.put("伞", -2104015908);
map.put("图表", -824773152);
map.put("气球", 1423728112);
map.put("太阳眼镜", 1204904862);
map.put("椅子", 193112560);
map.put("打印机", -939522792);
map.put("旗帜", 834329993);
map.put("猫", 1911236121);
map.put("女人", 2047088238);
map.put("男人", 664214693);
map.put("卡车", -1453025175);
map.put("电脑", -1970735883);
map.put("裤子", -337658120);
map.put("铅笔", 1993614559);
map.put("房子", -1299209990);
}
public static Answer getAnswer(JSONObject responseJsonObject) throws IOException {
String imageName = responseJsonObject.getString("imageName");
Integer targetHashcode = map.get(imageName);
if (targetHashcode == null) {
return null;
}
JSONArray values = responseJsonObject.getJSONArray("values");
String uid = responseJsonObject.getString("uid");
for (int i = 0; i < values.size(); i++) {
String url = "https://auth_services.teambition.com/captcha/image?uid=" + uid + "&lang=zh&index=" + i;
byte[] imageBytes = HttpUtil.request(url, null, Connection.Response::bodyAsBytes);
if (imageBytes == null) {
log.info("image download failed, imageName={}, uid={}, index={}", imageName, uid, i);
continue;
}
BufferedImage image = ImageIO.read(new ByteArrayInputStream(imageBytes));
int currentImageHashcode = ImageUtil.hash(image);
if (currentImageHashcode == targetHashcode) {
return Answer.builder().imageName(imageName).imageUrl(url).index(i).build();
}
}
return null;
}
public static void test() throws IOException {
String url = "https://auth_services.teambition.com/captcha/setup?num=5&lang=zh&_=" + System.currentTimeMillis();
String responseJsonStr = HttpUtil.request(url, connection -> {
connection.userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36");
}, Connection.Response::body);
JSONObject responseJsonObject = JSON.parseObject(responseJsonStr);
Answer answer = getAnswer(responseJsonObject);
if (answer == null) {
log.info("not find answer, responseJsonStr={}", responseJsonStr);
} else {
System.out.println(JSON.toJSONString(answer, true));
}
}
@Data
@Builder
public static class Answer {
private String imageName;
private String imageUrl;
private int index;
}
public static void main(String[] args) throws IOException {
test();
}
}
输出如下：
{
"imageName":"气球",
"imageUrl":"https://auth_services.teambition.com/captcha/image?uid=814b9d80-0b5a-11ea-8fd8-e31d1dab49d9&lang=zh&index=3",
"index":3
}点一下查看图片（点自己控制台上的，验证码图片都是会过期的，这里的链接过不多久就不能用了），发现是气球，多试个几次也都是对的，至此破解完毕。
相关资料：
1. https://account.teambition.com/login
.
浙公网安备 33010602011771号