Nginx之配置反爬虫

1.　　根据user_agent

if ($http_user_agent ~* "qihoobot|Baiduspider|Googlebot|Googlebot-Mobile|Googlebot-Image|Mediapartners-Google|Adsbot-Google|Feedfetcher-Google|Yahoo! Slurp|Yahoo! Slurp China|YoudaoBot|Sosospider|Sogou spider|Sogou web spider|MSNBot|ia_archiver|Tomato Bot")
{
return 403;
}
 
if ($http_user_agent ~* "mpcrawler" ) {
           return 302  https://www.baidu.com;
        }　

2.　　根据referer

location ~* ^/product-.*\.html$ {
    if ($http_referer = "") {
        return 404;
    }
}