BS4-爬虫

简介
BeautifulSoup提供一些简单的、python式的函数用来处理导航、搜索、修改分析树等功能。它是一个工具箱，通过解析文档为用户提供需要抓取的数据，因为简单，所以不需要多少代码就可以写出一个完整的应用程序。BeautifulSoup自动将输入文档转换为Unicode编码，输出文档转换为utf-8编码。你不需要考虑编码方式，除非文档没有指定一个编码方式，这时你仅仅需要说明一下原始编码方式就可以了。BeautifulSoup已成为和lxml、html6lib一样出色的python解释器，为用户灵活地提供不同的解析策略或强劲的速度

from bs4 import BeautifulSoup

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html, 'lxml')
print(soup.prettify())
print(soup.title.string)

<br> The Dormouse's story<br>

The Dormouse's story

Once upon a time there were three little sisters; and their names were

,

Lacie

and

Tillie

;
and they lived at the bottom of a well.

...

The Dormouse's story

标签选择器

# 选择元素
soup.p # 标签加其内部的所有内容，且只获取第一个匹配的标签

type(soup.p) # bs4.element.Tag

bs4.element.Tag

soup.p

The Dormouse's story

data = []
for i in soup.find('body').children:
    data.append(i)

data

['\n',

The Dormouse's story

,
'\n',

Once upon a time there were three little sisters; and their names were
,
Lacie and
Tillie;
and they lived at the bottom of a well.

,
'\n',

...

,
'\n']

print(soup.body.prettify())

The Dormouse's story

Once upon a time there were three little sisters; and their names were

,

Lacie

and

Tillie

;
and they lived at the bottom of a well.

...

# 提取信息
soup.p.string
# 获取名称
soup.p.name 
soup.p.parent.name
# 获取属性
print(soup.p.attrs) # 返回字典
print(soup.p.attrs['name'])
print(soup.p['name'])
# 获取内容
print(soup.p.string)

soup.title.string

type(soup.title.string)

type(soup.title.contents)

soup.head

soup.p

soup.p.string

"The Dormouse's story"

嵌套选择

# 每一返回结果是bs4.element.Tag
print(type(soup.p))
print(type(soup.p.b))

print(soup.p.string)
print(soup.p.b.string)

<class 'bs4.element.Tag'>
<class 'bs4.element.Tag'>
The Dormouse's story
The Dormouse's story

关联选择

# 选取到了一个节点元素之后，如果想要获取它的直接子节点可以调用contents属性
print(soup.body.contents)  # contents返回的是列表类型

['\n',

The Dormouse's story

, '\n',

Once upon a time there were three little sisters; and their names were
,
Lacie and
Tillie;
and they lived at the bottom of a well.

, '\n',

...

, '\n']

print(soup.title.contents)

["The Dormouse's story"]

list_ = soup.body.contents
len(list_)

子节点和子孙节点

# print(soup.body.children)
for i,child in enumerate(soup.body.children):
    print(i,child)

The Dormouse's story

Once upon a time there were three little sisters; and their names were
,
Lacie and
Tillie;
and they lived at the bottom of a well.

...

print(soup.body.descendants) # 生成器,递归查询所有子孙节点
for i,childs in enumerate(soup.body.descendants):
    print(i,childs)

<generator object Tag.descendants at 0x000002435F075ED0>
0

The Dormouse's story

2 The Dormouse's story
3 The Dormouse's story
4

Once upon a time there were three little sisters; and their names were
,
Lacie and
Tillie;
and they lived at the bottom of a well.

6 Once upon a time there were three little sisters; and their names were

7
8 Elsie
9 ,

10 Lacie
11 Lacie
12 and

13 Tillie
14 Tillie
15 ;
and they lived at the bottom of a well.
16

...

18 ...
19

父节点和祖先节点

print(soup.a.parent) # 返回其直接父节点即其下内容
type(soup.a.parent) # bs4.element.Tag

Once upon a time there were three little sisters; and their names were
,
Lacie and
Tillie;
and they lived at the bottom of a well.

bs4.element.Tag

print(soup.a.parents) # 
for i,des in enumerate(soup.a.parents):
    print(i,des)

<generator object Tag.descendants at 0x000002435F075ED0>
0

Once upon a time there were three little sisters; and their names were
,
Lacie and
Tillie;
and they lived at the bottom of a well.

The Dormouse's story

Once upon a time there were three little sisters; and their names were
,
Lacie and
Tillie;
and they lived at the bottom of a well.

...

2 The Dormouse's story

The Dormouse's story

Once upon a time there were three little sisters; and their names were
,
Lacie and
Tillie;
and they lived at the bottom of a well.

...

3 The Dormouse's story

The Dormouse's story

Once upon a time there were three little sisters; and their names were
,
Lacie and
Tillie;
and they lived at the bottom of a well.

...

兄弟节点

html = """
<html>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            Hello
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print('Next Sibling', soup.a.next_sibling)
print('Prev Sibling', soup.a.previous_sibling)
print('Next Siblings', list(enumerate(soup.a.next_siblings)))
print('Prev Siblings', list(enumerate(soup.a.previous_siblings)))

Next Sibling
Hello

Prev Sibling
Once upon a time there were three little sisters; and their names were

Next Siblings [(0, '\n Hello\n '), (1, Lacie), (2, ' \n and\n '), (3, Tillie), (4, '\n and they lived at the bottom of a well.\n ')]
Prev Siblings [(0, '\n Once upon a time there were three little sisters; and their names were\n ')]

方法选择器

# find_all()、find()等方法，我们可以调用方法然后传入相应等参数就可以灵活地进行查询了
find_all(name , attrs , recursive , text , **kwargs)
find_all，顾名思义，就是查询所有符合条件的元素，可以给它传入一些属性或文本来得到符合条件的元素
soup.find_all(True) # 返回所有标签

html='''
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
'''
soup = BeautifulSoup(html, 'lxml')

name属性

print(soup.find_all(name='ul')) # 查询所有ul标签，返回list
print('#'*50)
print(type(soup.find_all(name='ul')))
print(type(soup.find_all(name='ul')[0]))

[

]
##################################################
<class 'bs4.element.ResultSet'>
<class 'bs4.element.Tag'>

for ul in soup.find_all(name='ul'):
    print(ul.find_all(name='li')) # 结果返回列表类型，列表种的每个元素依然是Tagl类型
    for li in ul.find_all(name='li'):
        print(li.string)

[]
[

Foo

Bar

Jay

]
Foo
Bar
Jay
[]
[

Foo

Bar

]
Foo
Bar

attrs属性

print(soup.find_all(attrs={'class':'element'})) # 传入的是attrs参数，参数的类型是字典类型

print(soup.find_all('ul',attrs={'id':'list-1'}))

[

Foo

Bar

Jay

Foo

Bar

]

常用属性如id,class可以不用attrs来传递，而直接用id和class_

print(soup.find_all(class_='element'))

print(soup.find_all(id='list-1'))

[

Foo

Bar

Jay

Foo

Bar

]

text属性

# text参数可以用来匹配节点的文本，传入的形式可以是字符串，可以是正则表达式
import re
html='''
<div class="panel">
    <div class="panel-body">
        <a>Hello, this is a link</a>
        <a>Hello, this is a link, too</a>
    </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(text=re.compile('link'))) # 返回所有匹配正则表达式的节点文本组成的列表

['Hello, this is a link', 'Hello, this is a link, too']

find(name,attrs,recursive,text,**kwargs)
find()方法返回的是单个元素，也就是第一个匹配的元素

html='''
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
'''

from bs4 import BeautifulSoup
soup = BeautifulSoup(html,'lxml')

print(soup.find(neme='div')) # Return only the first child of this Tag matching the given
print('#'*10)
print(soup.find(name='ul'))
print('#'*10)
print(soup.find(attrs={'panel-heading'}))
print('#'*10)
print(soup.find(class_='element'))

None
##########

##########

Hello

##########

Foo

soup.find_parent() # 返回直接父节点
soup.find_parents() # 返回所有祖先节点
soup.find_next_sibling()# 返回后面第一个兄弟节点
soup.find_next_siblings() # 返回后面所有兄弟节点
soup.find_next() # 返回第一个符合条件的节点
soup.find_all_next() # 返回节点后所有符合条件的节点

CSS选择器

使用CSS选择器，只需要调用select()方法，传入相应的CSS选择器即可

html='''
<div class="panel">
<div class="panel-heading">
        <h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
</ul>
</div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')

print(soup.select('.panel .panel-heading')) # 符合CSS选择器的节点组成的列表

print(soup.select('ul li')) # 符合CSS选择器的节点组成的列表

print(soup.select('#list-2 .element')) # 符合CSS选择器的节点组成的列表
print(type(soup.select('ul')[0])) # print(type(soup.select('ul')[0]))

[

Hello

]

嵌套选择

soup.select('ul')

[

]

for ul in soup.select('ul'):
    print(ul.select('li'))

[

Foo

Bar

Jay

]
[

Foo

Bar

]

# 获取属性
for ul in soup.select('ul'):
    print(ul['id'])
    print(ul.attrs['id'])
    print(ul.get_text()) # 获取文本

list-1
list-1

Foo
Bar
Jay

list-2
list-2

Foo
Bar

print(soup.find_all('div')[1])
[out]
<div class="panel-heading">
<h4>Hello</h4>
</div>
soup.find_all('div')[1].get_text()
[out]
'\nHello\n'
print(soup.find_all('div')[1].h4.string)
[out]
Hello
print(soup.find_all('div')[1].string)
[out]
None

posted @ 2020-11-23 18:19 三两研几阅读(41) 评论(0) 收藏举报

刷新页面返回顶部

Aspiration

风朗气晴，江山稳固

BS4-爬虫

标签选择器

嵌套选择

关联选择

子节点和子孙节点

父节点和祖先节点

兄弟节点

方法选择器

name属性

attrs属性

text属性

Hello

CSS选择器

Hello

嵌套选择

公告