《鲜活的数据：数据可视化指南》第2章 收集数据 Python3.3源码

 全部源码和数据文件下载：仅供参考

《鲜活的数据：数据可视化指南》书中的python代码为2.x，在看书过程中把部分源代码转换为Python3.3格式。

• 运行环境：Windows7, Python3.3

（1） get-weather-data.py 从网页抓取天气信息（类似于小爬虫）

• 说明：主要修改包括urllib2的变化，timestamp的修改，采集数据的地点，数据位置等

# coding = utf-8
__author__ = 'hillfree'

from urllib.request import urlopen
from bs4 import BeautifulSoup

# Create/open a file called wunder.txt (which will be a comma-delimited file)
f = open('wunder-data.txt', 'a')

# Iterate through months and day
for month in range(1, 13):
    for day in range(1, 32):

        # Check if already gone through month
        if (month == 2 and day > 28):
            break
        elif (month in [4, 6, 9, 11] and day > 30):
            break

        # Open wunderground.com url
        url = "http://www.wunderground.com/history/airport/ZBAA/2012/{0}/{1}/DailyHistory.html".format(month, day, )
        page = urlopen(url)

        # Get temperature from page
        soup = BeautifulSoup(page)
        # 取得最高温度
        max_temp = soup.findAll(attrs={"class":"nobr"})[3].span.string

        # Build day record with timestamp
        record = "2012{0:02d}{1:02d}, {2}\n".format(month, day, max_temp)
        print(record)
        # Write timestamp and temperature to file
        f.write(record)

# Done getting data! Close file.
f.close()

（2） get-weather-data-full.py 从网页抓取天气信息强化版

• 说明：在get-weather-data.py基础上增加的年份的循环，以及闰年的判断等

# coding = utf-8 
__author__ = 'hillfree'


from urllib.request import urlopen
from bs4 import BeautifulSoup

# Create/open a file called wunder.txt (which will be a comma-delimited file)
f = open('wunder-data.txt', 'a')

# Iterate through year, months and day
for year in range(2013, 2014):
    for month in range(1, 2):
        for day in range(1, 32):

        # Check if leap year
            if year % 400 == 0:
                leap = True
            elif year % 100 == 0:
                leap = False
            elif year % 4 == 0:
                leap = True
            else:
                leap = False

            # Check if already gone through month
            if (month == 2 and leap and day > 29):
                continue
            elif (month == 2 and day > 28):
                continue
            elif (month in [4, 6, 9, 10] and day > 30):
                continue

            # Check if already gone through month
            if (month == 2 and day > 28):
                break
            elif (month in [4, 6, 9, 11] and day > 30):
                break

            # Open wunderground.com url
            url = "http://www.wunderground.com/history/airport/ZBAA/{0}/{1}/{2}/DailyHistory.html".format(year, month, day, )
            page = urlopen(url)

            # Get temperature from page
            soup = BeautifulSoup(page)
            # 取得最高温度
            max_temp = soup.findAll(attrs={"class":"nobr"})[3].span.string

            # Build day record with timestamp
            record = "{0:04d}{1:02d}{2:02d}, {3}\n".format(year, month, day, max_temp)
            print(record)
            # Write timestamp and temperature to file
            f.write(record)

# Done getting data! Close file.
f.close()

（3）add-csv-flag.py 为CSV文件内容增加标志位

• 说明：在之前生成的CSV文件的基础上，进行判断，并添加is_freezing的标志位，用print输出

# coding = utf-8 
__author__ = 'hillfree'

import csv


reader = csv.reader(open('wunder-data.txt', 'r'), delimiter=",")

for row in reader:
    if int(row[1]) < 0:
        is_freezing = '1'
        print("{0}, {1}, {2}".format(row[0], row[1], is_freezing)) # 列出冰冻日
    else:
        is_freezing = '0'

    # print("{0}, {1}, {2}".format(row[0], row[1], is_freezing)) # 可写入文件

（4）csv-to-xml.py 把CSV文件转换为XML格式

• 说明：在print输出的基础上，增加了写入文件“wunder-data.xml"

# coding = utf-8

"""
source: <Visualize This> by Nathan Yau
name: csv-to-xml.py
python: v3.3
description: Convert CSV file to Xml format
"""
__author__ = 'hillfree'

import csv

reader = csv.reader(open('wunder-data.txt', 'r'), delimiter=",")
output = open("wunder-data.xml", 'w')

print('<weather_data>')
output.write('<weather_data>\n')

for row in reader:
    print('<observation>')
    print('<date>' + row[0] + '</date>')
    print('<max_temperature>' + row[1] + '</max_temperature>')
    print('</observation>')

    output.write('<observation>\n')
    output.write('<date>' + row[0] + '</date>\n')
    output.write('<max_temperature>' + row[1] + '</max_temperature>\n')
    output.write('</observation>\n')

print('</weather_data>')
output.write('</weather_data>\n')

（5）csv-to-json.py 把CSV文件转换为Json格式

• 说明：原文件是利用csv模块引入，然后print输出。但是原文件中利用365条记录作为文件的终结过于僵硬。这里没有引入csv模块，而是利用file的操作，并且做了简单的格式化。

# coding = utf-8

"""
source: <Visualize This> by Nathan Yau
name: csv-to-json.py
python: v3.3
description: Convert CSV file to json format, also write to file.
use file module operation instead of csv module
"""
__author__ = 'hillfree'

lines = open('wunder-data.txt', 'r').readlines()
output = open("wunder-data.json", 'w')

output.write('{"observations": [\n')

max = len(lines)
count = 0
for line in lines:
    count += 1
    row = line.split(',')
    record ='\t{\n\t\t"date": "%s", \n\t\t"temperature": %s' % (row[0], row[1].lstrip())

    if count < max:
        record += '\t},\n'
    else:
        record += '\t}]\n}'

    output.write(record)

（6）xml-to-csv.py 把xml格式文件转换为csv格式

• 说明：原文件是利用BeautifulSoup模块的BeautifulStoneSoup来处理xml。但在python3.3中相应的lxml无法使用（？）。因此只好采用了python3.3自带的xml.dom.minidom模块，第一次使用感觉很别扭，尤其是子节点的使用很不习惯。加入了很多判断才能将日期和最高气温捏合在一起。求其他更好的办法。

# coding = utf-8

"""
source: <Visualize This> by Nathan Yau
name: xml-to-csv.py
python: v3.3
description: Convert xml file to csv format, also write to file.
因为Python3.3下lxml无法应用，所以采用python自带的minidom，
用起来比较别扭，不知道如何改进？
"""
__author__ = 'hillfree'

from xml.dom.minidom import parse, Node, NodeList

xml_file =parse("wunder-data.xml")

date = ""
max_temperature = ""
for node1 in xml_file.getElementsByTagName("observation"):
    for node2 in node1.childNodes:

        for node3 in node2.childNodes:
            if node3.nodeType == Node.TEXT_NODE and node2.nodeName == "date":
                if node3.nodeValue != "":
                    date = node3.nodeValue
            if node3.nodeType == Node.TEXT_NODE and node2.nodeName == "max_temperature":
                if node3.nodeValue != "":
                    max_temperature = node3.nodeValue.lstrip()

        print(date + "," + max_temperature)