整站爬虫命令
原文链接：https://user.qzone.qq.com/296917342/blog/1551108123
整站爬虫命令：
linux平台
wget --mirror --convert-links --adjust-extension --page-requisites --retry-connrefused --exclude-directories=comments -U "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0" --execute robots=off http://xxxx.com
wget --mirror --convert-links --adjust-extension --page-requisites --retry-connrefused --exclude-directories=comments -U "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0" --execute robots=off http://xxxx.com
