python利用beautifulSoup写爬虫

python BeautifulSoup模块的安装

安装包下载地址：http://www.crummy.com/software/BeautifulSoup/#Download

文档：http://www.crummy.com/software/BeautifulSoup/documentation.html 

下载后解压， 然后进入目录执行 ：

python setup.py build 
python setup.py install

引入包要用

 import bs4
 from bs4 import BeautifulSoup

利用BeautifulSoup抓取网页内容

# coding=utf-8
from bs4 import BeautifulSoup
import urllib
import re

url ='http://www.baidu.com/s'
values ={'wd':u'渗透'}
encoded_param = urllib.urlencode(values)
full_url = url +'?'+ encoded_param
response = urllib.urlopen(full_url)
soup =BeautifulSoup(response)
alinks = soup.find_all('a', href=re.compile('^http|^/'))