Python网络爬虫——爬取和分析福建二手房房价及各项数据

一、 选题的背景介绍

随着越来越多城市的房地产市场进入存量时代，二手房市场的地位愈发重要，其走势对于房地产整体市场的影响也逐渐加强。在很多二手房市场规模占比较高的一二线城市，二手房市场与新房市场通过“卖旧买新”的链条形成较强的联动：二手房卖家通过置换现有住房获得资金，转而在新房市场实现住房改善。

       在买房之前，人们会在相关网站上搜索自己想要了解的住房信息，但是面对庞大且来源广泛的网络数 据，如何缩短网页下载时间，如何分析大量数据并找出有用信息，就需要用到网络爬虫技术。本文通过 Python爬虫程序爬取链家网上福建省份的二手房数据，爬取的字段包括所在区域、户型结构、装修情况、总价、单价等，并对采集到的这些数据进行清洗。最后对清洗过的数据用echarts进行可视化分析，探索福建二手房建筑面积、位置、户型等对二手房房价的影响情况。

二、 数据可视化分析的步骤

 1.     爬虫名称：爬取链家网站二手房数据

 2.     爬取内容：爬取房子名称、价格、平方价格、面积、所在城市、所在地区、户型等

 3.     方案描述：观察链家网站二手房的访问地址，找到规律，进行地址拼接，再爬取每个页面的房子详情页的地址，循环解析提取房子详情页里的信息，存储到数据库中，方便接下来的数据分析和数据可视化的展示。

技术难点：

1.在爬取过程中除了目标数据以外还存在广告等其他数据，需要做异常处理

2.使用pandas将数据库里的数据进行处理和分析

三、 主题页面的结构特征分析

1.主题页面的结构特征分析

 我们要爬取的是链家网福建省内所有城市的二手房数据，链家支持的福建城市有福州、厦门、泉州、漳州，每个城市里面有100页数据，每一页里有30条房屋信息，我们需要将这三十条房屋信息的地址爬取出来逐一进行数据提取。所以爬虫的结构有三层，第一层是城市循环，第二层是页面循环，第三层是该页面的房屋信息循环。

2.节点（标签）查找方法与遍历方法

name = html.xpath("//h1/text()")  
   price = html.xpath("//span[@class='total']/text()")  
   area = html.xpath("//div[@class='area']/div[1]/text()")  
   priceAvg = html.xpath("//span[@class='unitPriceValue']/text()")  
   houseType= html.xpath("//div[@class='room']/div[@class='mainInfo']/text()")  
   orientation = html.xpath("//div[@class='type']/div[@class='mainInfo']/text()")  
   city = html.xpath("//div[4]/div/div/a[1]/text()")  
   district= html.xpath("//div[@class='areaName']/span[@class='info']/a[1]/text()") 
   community = html.xpath("//div[@class='communityName']//a[1]/text()")  
   decoration=html.xpath("//div[@class='base']/div[@class='content']/ul/li[9]/text()")  
   propertyRight= html.xpath("//div[@class='transaction']/div[@class='content']/ul/li[6]/span[2]/text()") 
   lift = html.xpath("//div[@class='base']/div[@class='content']/ul/li[11]/text()")  
   lifeRate= html.xpath("//div[@class='base']/div[@class='content']/ul/li[10]/text()") 
   builtType = html.xpath("//div[@class='base']/div[@class='content']/ul/li[6]/text()") 
 builtStructure= tml.xpath("//div[@class='base']/div[@class='content']/ul/li[8]/text()") 

四、 网络爬虫设计程序设计

  1.   数据的爬取与采集

 

def start():
    # 爬取所需城市的名称缩写，用于地址拼接，链家网站福建省内支持的城市仅有以下四个、支持的城市列表可以查看https://map.lianjia.com/map/350200/ESF/
    cityList = ['xm', 'fz', 'quanzhou', 'zhangzhou']
    # 设置请求头，以防被网站识别为爬虫
    headers = {
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.46",
        "Cookie": "lianjia_uuid=60c54ae6-6b42-4222-8692-efe4fb2c554e; crosSdkDT2019DeviceId=eu2b5l-mfq32x-njycw4jyhevekso-ud4cezlw8; _smt_uid=632e762f.20c8106d; _ga=GA1.2.203719967.1663989297; Hm_lvt_9152f8221cb6243a53c83b956842be8a=1669339641; lianjia_ssid=917d080d-af96-4114-92ce-e9ec6ded0cde; _gid=GA1.2.1343492366.1671084252; lianjia_token=2.0111ad29317b6152f82536df72a8e72e0777cf78c4; beikeBaseData=%7B%22parentSceneId%22:%226413099907731748097%22%7D; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%221836d7da82957c-0d0e66116b0617-78565473-1327104-1836d7da82a940%22%2C%22%24device_id%22%3A%221836d7da82957c-0d0e66116b0617-78565473-1327104-1836d7da82a940%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_referrer%22%3A%22%22%2C%22%24latest_referrer_host%22%3A%22%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%7D%7D; Hm_lpvt_9152f8221cb6243a53c83b956842be8a=1671089180; select_city=350200"
    }
    # 写入csv文件，只是备份一份数据，后面并无用到
    with open(r'二手房房价2.csv', 'a', encoding='utf_8_sig', newline='')as f:
        table_label = ['房屋名称', '房屋总价', '建筑面积', '每平方价格', '房屋户型', '房屋朝向',
                       '城市', '地区', '小区名称', '装修情况', '房屋产权', '电梯情况', '梯户比例', '建筑类型', '建筑结构']
        wt = csv.writer(f)
        wt.writerow(table_label)
    # 按城市列表里的值进行第一轮地址拼接
    for i in cityList:
        url = 'https://' + i + '.lianjia.com/ershoufang/'
        # 第二轮城市地址拼接
        all_url = Get_url(url)
        # 将所有url地址和请求头传入方法内进行采集
        Get_house_url(all_url, headers)
# 每个城市有100页的地址，将拼接后的数据装入列表后返回
def Get_url(url):
    all_url = []
    for i in range(1, 101):
        all_url.append(url + 'pg' + str(i) + '/')
    return all_url


# 获取每一个页面里面每一套房的url
def Get_house_url(all_url, headers):
    num = 0
    # 简单统计页数
    for i in all_url:
        # 获取当前页面的代码
        r = requests.get(i, headers=headers)
        html = etree.HTML(r.text)
        # 使用xpath匹配该页面每套房子的url，将会以列表形式存储
        url_ls = html.xpath("//ul[@class ='sellListContent']/li/a/@href")
        # 爬取房子详情页的内容
        Analysis_html(url_ls, headers)
        time.sleep(4)
        num += 1
        print("第%d页爬完了" % num)  # num记录爬取索成功的索引值


# 获取每一套房子的详情信息
def Analysis_html(url_ls, headers):
    for i in url_ls:
        r = requests.get(i, headers=headers)
        html = etree.HTML(r.text)
        name = html.xpath("//h1/text()")  # 在获取房名
        price = html.xpath("//span[@class='total']/text()")  # 获取总价
        area = html.xpath("//div[@class='area']/div[1]/text()")  # 获取面积
        priceAvg = html.xpath("//span[@class='unitPriceValue']/text()")  # 获取每平方的价格
        houseType = html.xpath("//div[@class='room']/div[@class='mainInfo']/text()")  # 获取房屋类型
        orientation = html.xpath("//div[@class='type']/div[@class='mainInfo']/text()")  # 获取房屋朝向
        city = html.xpath("//div[4]/div/div/a[1]/text()")  # 获取城市
        district = html.xpath("//div[@class='areaName']/span[@class='info']/a[1]/text()")  # 获取地区
        community = html.xpath("//div[@class='communityName']//a[1]/text()")  # 获取社区
        decoration = html.xpath("//div[@class='base']/div[@class='content']/ul/li[9]/text()")  # 获取装修情况
        propertyRight = html.xpath(
            "//div[@class='transaction']/div[@class='content']/ul/li[6]/span[2]/text()")  # 获取房屋产权
        lift = html.xpath("//div[@class='base']/div[@class='content']/ul/li[11]/text()")  # 获取电梯情况
        lifeRate = html.xpath("//div[@class='base']/div[@class='content']/ul/li[10]/text()")  # 获取梯户比例
        builtType = html.xpath("//div[@class='base']/div[@class='content']/ul/li[6]/text()")  # 获取建筑类型
        builtStructure = html.xpath("//div[@class='base']/div[@class='content']/ul/li[8]/text()")  # 获取建筑结构
        # 爬取过程中可能会出现广告，异常捕获跳过
        try:
            # 将爬取下来的数据存储到数据库
            Save_data(name, price, area, priceAvg, houseType, orientation, city, district, community, decoration,
                      propertyRight, lift, lifeRate, builtType, builtStructure)
            print(name, price, area, priceAvg, houseType, orientation, city, district, community, decoration,
                      propertyRight, lift, lifeRate, builtType, builtStructure)
        except:
            continue
        # 设置休眠实现
        time.sleep(random.randint(1, 3))   

2.对数据进行清洗和处理
将多余的字符去掉并将某些需要计算的列转为数值类型

# 数据清洗
def data_clean(df):
    print(df.houseCity)
    # 将城市列的后三个字房地产去掉
    df['houseCity'] = df['houseCity'].str.strip('房产网')
    # 将字符串转换为数值，用于接下来的统计
    df['housePrice'] = df['housePrice'].astype(float, errors='raise')
    df['housePriceAvg'] = df['housePriceAvg'].astype(float, errors='raise')
    print(df.houseCity)

处理每个城市房价和平米价格的数据

# 生成城市房价图和每平米价格图所需数据
def get_house_price(df):
    # 获取最贵的单价数据
    maxPrice = df['housePrice'].max(axis=0)
    # 将数据帧里的数据，按城市分组，求房价和平米价的平均值保留1位小数
    housePriceList = round(df.groupby(['houseCity'])['housePrice'].mean(), 1)
    houseUnitPriceList = round(df.groupby(['houseCity'])['housePriceAvg'].mean(), 1)
    # 转为数据帧(因为使用mean方法的返回值是序列)
    housePriceList = pd.DataFrame({'houseCity': housePriceList.index, 'housePriceAvg': housePriceList.values})
    houseUnitPriceList = pd.DataFrame(
        {'houseCity': houseUnitPriceList.index, 'houseUnitPriceAvg': houseUnitPriceList.values})
    # 排序，按照房价这个字段降序，ascending True降序
    housePriceList.sort_values(by=['housePriceAvg'], axis=0, ascending=[True], inplace=True)
    # 将俩个数据帧以城市名为关联字段，重新级联，保持数据位置的一致性
    cityAvg = pd.merge(housePriceList, houseUnitPriceList, on='houseCity', how='inner')
    # 以下将数据转为列表返回前端用于图标生成
    cityList = np.array(cityAvg.houseCity)
    cityList = cityList.tolist()
    priceList = np.array(cityAvg.housePriceAvg)
    priceList = priceList.tolist()
    unitPriceList = np.array(cityAvg.houseUnitPriceAvg)
    unitPriceList = unitPriceList.tolist()
    print(cityList, priceList, unitPriceList)
    return cityList, priceList, unitPriceList, len(cityList), len(df), maxPrice, housePriceList.houseCity[0]

处理所选城市户型、区县户数、装修情况、建筑类型的数据

# 四张饼图生成
def get_pie(df, cityName=None):
    # 如果有传入城市参数，将数据帧缩小到该城市数据
    if cityName != None:
        df = df[df['houseCity'].str.contains(cityName)]
    # 使用size统计每个列里面字段出现过的次数
    houseTypeList = df.groupby(['houseType']).size()
    houseDistrictList = df.groupby(['houseDistrict']).size()
    houseDecorationList = df.groupby(['houseDecoration']).size()
    builtTypeList = df.groupby(['builtType']).size()

    # 将各个字段的值以字典的格式添加到临时列表里面
    templist = []
    for i, j in zip(houseTypeList.index, houseTypeList.values):
        templist.append({'value': str(j), 'name': str(i)})
    templist1 = []
    for i, j in zip(houseDistrictList.index, houseDistrictList.values):
        templist1.append({'value': str(j), 'name': str(i)})

    templist2 = []
    for i, j in zip(houseDecorationList.index, houseDecorationList.values):
        templist2.append({'value': str(j), 'name': str(i)})
    templist3 = []
    for i, j in zip(builtTypeList.index, builtTypeList.values):
        templist3.append({'value': str(j), 'name': str(i)})
    all_list = []
    all_list.append(templist)
    all_list.append(templist1)
    all_list.append(templist2)
    all_list.append(templist3)
    print(all_list)
    return all_list

处理每个城市的区县单价数据

# 城市里各区县的单价统计
def analyse_district(df, cityName=None):
    if cityName != None:
        df = df[df['houseCity'].str.contains(cityName)]
    houseDistrictPrice = round(df.groupby(['houseDistrict'])['housePrice'].mean(), 1)

    districtList = np.array(houseDistrictPrice.index)
    districtList = districtList.tolist()
    priceList = np.array(houseDistrictPrice.values)
    priceList = priceList.tolist()
    print(districtList,'\n', priceList)
    return districtList, priceList

3. 文本分析

      分析每个城市的小区的热门程度

 

# 词云生成
def wordCloud(df):
    # 循环生成每个城市的词云
    for i in df.houseCity.unique():
        if os.path.exists(r'D:\Python\workspace\HousePriceAnalysis\static\images\\' + i + '.jpg'):
            pass
        else:
            # 分词
            df = df[df['houseCity'].str.contains(i)]
            strAll = ''
            for j in df['houseCommunity']:
                strAll += j
            cut = jieba.cut(strAll)
            strList = " ".join(cut)
            print(strList)
            # 生成遮罩
            img = Image.open(r'D:\Python\workspace\HousePriceAnalysis\static\images\遮罩.jpg')
            img_array = np.array(img)  # 图片转为数组
            wc = WordCloud(
                background_color='white',
                mask=img_array,
                font_path="simhei.ttf",
                height=100,
                width=300
            )
            wc.generate_from_text(strList)
            # fig = plt.figure(1)
            plt.imshow(wc)
            plt.axis('off')
            plt.savefig(r'D:\Python\workspace\HousePriceAnalysis\static\images\%s.jpg' % i)

     4.    数据分析与可视化

      城市房价图和每平米的价格图，可以清晰的看见几个城市的数据对比。北京无愧于帝都的称号，在房价方面还是遥遥领先。

var chartDom = document.getElementById('main1');
var myChart = echarts.init(chartDom);
var option;
option = {
  title: {
    text: '每平米价格图'
  },
  tooltip: {
    trigger: 'axis',
    axisPointer: {
      type: ''
    }
  },
  legend: {},
  grid: {
    left: '3%',
    right: '4%',
    bottom: '3%',
    containLabel: true
  },
  xAxis: {
    type: 'value',
    boundaryGap: [0, 0.01]
  },
  yAxis: {
    type: 'category',
    data: {{cityList|safe}}
  },
  series: [
    {
      name: '每平米价格/元',
      type: 'bar',
      data: {{unitPriceList|safe}}
    }
  ]
};
option && myChart.setOption(option);

var chartDom = document.getElementById('main');
var myChart = echarts.init(chartDom);
var option;
option = {
  title: {
    text: '城市房价图'
  },
  tooltip: {
    trigger: 'axis',
    axisPointer: {
      type: ''
    }
  },
  legend: {},
  grid: {
    left: '3%',
    right: '4%',
    bottom: '3%',
    containLabel: true
  },
  xAxis: {
    type: 'value',
    boundaryGap: [0, 0.01]
  },
  yAxis: {
    type: 'category',
    data: {{cityList|safe}}
  },
  series: [
    {
      name: '平均房价/万元',
      type: 'bar',
      data: {{priceList|safe}}
    }
  ]
};
option && myChart.setOption(option);

每个城市的房子户型占比图、各区房屋数量图、装修情况图、建筑类型图，可以看到的是福州的二手房最主要还是以3室2厅为主，闽侯县、仓山区、鼓楼区、晋安区的房屋占据福州二手房的主要部分，装修情况精装过半，建筑类型多为板楼

var chartLegend=['房子户型占比图','各区/县户数占比图','装修情况图','建筑类型图']
                all_list = json.all_list
                for(var i=2;i<6;i++){
                    tempList = all_list[i-2]
                    var chartDom = document.getElementById('main'+i);
                    var myChart = echarts.init(chartDom);
                    var option;

                    option = {
                      title: {
                        text: chartLegend[i-2],
                        left: 'center'
                      },
                      tooltip: {
                        trigger: 'item'
                      },
                      legend: {
                        orient: 'vertical',
                        left: 'left'
                      },
                      series: [
                        {
                          name: '户',
                          type: 'pie',
                          radius: '50%',
                          data: tempList,
                          emphasis: {
                            itemStyle: {
                              Blur: 10,
                              OffsetX: 0,
                              Color: 'rgba(0, 0, 0, 0.5)'
                            }
                          }
                        }
                      ]
                    };
                    option && myChart.setOption(option);

各个区的房屋均价,可以根据所选城市查看，现在可以看见的是鼓楼区的价格是福州最高的。

var chartDom = document.getElementById('main6');
                var myChart = echarts.init(chartDom);
                var option;
                let dataAxis = json.districtList;
                // prettier-ignore
                let data = json.priceList;
                let yMax = 500;
                let dataShadow = [];
                for (let i = 0; i < data.length; i++) {
                  dataShadow.push(yMax);
                }
                option = {
                  title: {
                    text: '各个区/县/地级市均价',
                    subtext: '单位/万元'
                  },
                  xAxis: {
                    data: dataAxis,
                    axisLabel: {
                      inside: true,
                      color: '#fff'
                    },
                    axisTick: {
                      show: false
                    },
                    axisLine: {
                      show: false
                    },
                    z: 10
                  },
                  yAxis: {
                    axisLine: {
                      show: false
                    },
                    axisTick: {
                      show: false
                    },
                    axisLabel: {
                      color: '#999'
                    }
                  },
                  dataZoom: [
                    {
                      type: 'inside'
                    }
                  ],
                  series: [
                    {
                      type: 'bar',
                      showBackground: true,
                      itemStyle: {
                        color: new echarts.graphic.LinearGradient(0, 0, 0, 1, [
                          { offset: 0, color: '#83bff6' },
                          { offset: 0.5, color: '#188df0' },
                          { offset: 1, color: '#188df0' }
                        ])
                      },
                      emphasis: {
                        itemStyle: {
                          color: new echarts.graphic.LinearGradient(0, 0, 0, 1, [
                            { offset: 0, color: '#2378f7' },
                            { offset: 0.7, color: '#2378f7' },
                            { offset: 1, color: '#83bff6' }
                          ])
                        }
                      },
                      data: data
                    }
                  ]
                };
                // Enable data zoom when user click bar.
                const zoomSize = 6;
                myChart.on('click', function (params) {
                  console.log(dataAxis[Math.max(params.dataIndex - zoomSize / 2, 0)]);
                  myChart.dispatchAction({
                    type: 'dataZoom',
                    startValue: dataAxis[Math.max(params.dataIndex - zoomSize / 2, 0)],
                    endValue:
                      dataAxis[Math.min(params.dataIndex + zoomSize / 2, data.length - 1)]
                  });
                });
                option && myChart.setOption(option);

5.数据持久化

将数据存储到关系型数据库内

# 存储数据库函数
def Save_data(name, price, area, priceAvg, houseType, orientation, city, district, community, decoration, propertyRight,
              lift, lifeRate, builtType, builtStructure):
    result = []
    result = name + price + area + priceAvg + houseType + orientation + city + district + community + decoration + propertyRight + lift + lifeRate + builtType + builtStructure

    # 存储在数据库
    houseDetail = models.house(houseName=result[0], housePrice=result[1], houseArea=result[2], housePriceAvg=result[3],
                               houseType=result[4], houseOrientation=result[5], houseCity=result[6],
                               houseDistrict=result[7],
                               houseCommunity=result[8], houseDecoration=result[9], propertyRight=result[10],
                               lift=result[11], lifeRate=result[12], builtType=result[13], builtStructure=result[14])
    houseDetail.save()

6.  主要代码汇总

   

"""
    @Name:rj
    @Class:
    @DateTime:2022/12/17 11:09
    @SoftWare:PyCharm
"""
import numpy as np
import pandas as pd

from rj import models

import os
import jieba
from wordcloud import WordCloud, ImageColorGenerator
from PIL import Image
from matplotlib import pyplot as plt


# 数据清洗
def data_clean(df):
    # 将城市列的后三个字房地产去掉
    df['houseCity'] = df['houseCity'].str.strip('房产网')
    df['housePrice'] = df['housePrice'].astype(float, errors='raise')
    df['housePriceAvg'] = df['housePriceAvg'].astype(float, errors='raise')
    return df


# 数据转换，将数据转为数据帧
def get_pandas_data():
    houseAll = models.house.objects.all()
    houseName, housePrice, houseArea, housePriceAvg, houseType, houseOrientation, houseCity, houseDistrict, houseCommunity, houseDecoration, propertyRight, lift, lifeRate, builtType, builtStructure = [], [], [], [], [], [], [], [], [], [], [], [], [], [], []
    for i in houseAll:
        houseName.append(i.houseName)
        housePrice.append(i.housePrice)
        houseArea.append(i.houseArea)
        housePriceAvg.append(i.housePriceAvg)
        houseType.append(i.houseType)
        houseOrientation.append(i.houseOrientation)
        houseCity.append(i.houseCity)
        houseDistrict.append(i.houseDistrict)
        houseCommunity.append(i.houseCommunity)
        houseDecoration.append(i.houseDecoration)
        propertyRight.append(i.propertyRight)
        lift.append(i.lift)
        lifeRate.append(i.lifeRate)
        builtType.append(i.builtType)
        builtStructure.append(i.builtStructure)

    df = pd.DataFrame(
        {'houseName': houseName, 'housePrice': housePrice, 'houseArea': houseArea, 'housePriceAvg': housePriceAvg,
         'houseType': houseType, 'houseOrientation': houseOrientation,
         'houseCity': houseCity, 'houseDistrict': houseDistrict, 'houseCommunity': houseCommunity,
         'houseDecoration': houseDecoration, 'propertyRight': propertyRight, 'lift': lift,
         'lifeRate': lifeRate, 'builtType': builtType,
         'builtStructure': builtStructure})
    df = data_clean(df)
    return df


# 生成城市房价图和每平米价格图所需数据
def get_house_price(df):
    # 获取最贵的单价数据
    maxPrice = df['housePrice'].max(axis=0)
    # 将数据帧里的数据，按城市分组，求房价和平米价的平均值保留1位小数
    housePriceList = round(df.groupby(['houseCity'])['housePrice'].mean(), 1)
    houseUnitPriceList = round(df.groupby(['houseCity'])['housePriceAvg'].mean(), 1)
    # 转为数据帧(因为使用mean方法的返回值是序列)
    housePriceList = pd.DataFrame({'houseCity': housePriceList.index, 'housePriceAvg': housePriceList.values})
    houseUnitPriceList = pd.DataFrame(
        {'houseCity': houseUnitPriceList.index, 'houseUnitPriceAvg': houseUnitPriceList.values})
    # 排序，按照房价这个字段降序，ascending True降序
    housePriceList.sort_values(by=['housePriceAvg'], axis=0, ascending=[True], inplace=True)
    # 将俩个数据帧以城市名为关联字段，重新级联，保持数据位置的一致性
    cityAvg = pd.merge(housePriceList, houseUnitPriceList, on='houseCity', how='inner')
    # 以下将数据转为列表返回前端用于图标生成
    cityList = np.array(cityAvg.houseCity)
    cityList = cityList.tolist()
    priceList = np.array(cityAvg.housePriceAvg)
    priceList = priceList.tolist()
    unitPriceList = np.array(cityAvg.houseUnitPriceAvg)
    unitPriceList = unitPriceList.tolist()

    return cityList, priceList, unitPriceList, len(cityList), len(df), maxPrice, housePriceList.houseCity[0]


# 四张饼图生成
def get_pie(df, cityName=None):
    # 如果有传入城市参数，将数据帧缩小到该城市数据
    if cityName != None:
        df = df[df['houseCity'].str.contains(cityName)]
    # 使用size统计每个列里面字段出现过的次数
    houseTypeList = df.groupby(['houseType']).size()
    houseDistrictList = df.groupby(['houseDistrict']).size()
    houseDecorationList = df.groupby(['houseDecoration']).size()
    builtTypeList = df.groupby(['builtType']).size()

    # 将各个字段的值以字典的格式添加到临时列表里面
    templist = []
    for i, j in zip(houseTypeList.index, houseTypeList.values):
        templist.append({'value': str(j), 'name': str(i)})
    templist1 = []
    for i, j in zip(houseDistrictList.index, houseDistrictList.values):
        templist1.append({'value': str(j), 'name': str(i)})

    templist2 = []
    for i, j in zip(houseDecorationList.index, houseDecorationList.values):
        templist2.append({'value': str(j), 'name': str(i)})
    templist3 = []
    for i, j in zip(builtTypeList.index, builtTypeList.values):
        templist3.append({'value': str(j), 'name': str(i)})
    all_list = []
    all_list.append(templist)
    all_list.append(templist1)
    all_list.append(templist2)
    all_list.append(templist3)
    return all_list

# 词云生成
def wordCloud(df):
    # 循环生成每个城市的词云
    for i in df.houseCity.unique():
        if os.path.exists(r'D:\python\链家热门城市爬取\链家热门城市爬取\HousePriceAnalysis\static\images\\' + i + '.jpg'):
            pass
        else:
            # 分词
            df = df[df['houseCity'].str.contains(i)]
            strAll = ''
            for j in df['houseCommunity']:
                strAll += j
            cut = jieba.cut(strAll)
            strList = " ".join(cut)
            print(strList)
            # 生成遮罩

            img = Image.open(r'D:\python\链家热门城市爬取\链家热门城市爬取\HousePriceAnalysis\static\images\遮罩.jpg')
            img_array = np.array(img)  # 图片转为数组
            wc = WordCloud(
                background_color='white',
                mask=img_array,
                font_path="simhei.ttf",
                height=100,
                width=300
            )
            wc.generate_from_text(strList)
            # fig = plt.figure(1)
            plt.imshow(wc)
            plt.axis('off')
            plt.savefig(r'D:\python\链家热门城市爬取\链家热门城市爬取\HousePriceAnalysis\static\images\%s.jpg' % i)


# 城市里各区县的单价统计
def analyse_district(df, cityName=None):
    if cityName != None:
        df = df[df['houseCity'].str.contains(cityName)]
    houseDistrictPrice = round(df.groupby(['houseDistrict'])['housePrice'].mean(), 1)

    districtList = np.array(houseDistrictPrice.index)
    districtList = districtList.tolist()
    priceList = np.array(houseDistrictPrice.values)
    priceList = priceList.tolist()
    return districtList, priceList

"""
    @Name:rj
    @Class:
    @DateTime:2022/9/24 l0:3l
    @SoftWare:PyCharm
"""
import requests, time, csv
from lxml import etree
import random
from rj import models


# 每个城市有100页的地址，将拼接后的数据装入列表后返回
def Get_url(url):
    all_url = []
    for i in range(1, 101):
        all_url.append(url + 'pg' + str(i) + '/')
    return all_url


# 获取每一个页面里面每一套房的url
def Get_house_url(all_url, headers):
    num = 0
    # 简单统计页数
    for i in all_url:
        # 获取当前页面的代码
        r = requests.get(i, headers=headers)
        html = etree.HTML(r.text)
        # 使用xpath匹配该页面每套房子的url，将会以列表形式存储
        url_ls = html.xpath("//ul[@class ='sellListContent']/li/a/@href")
        # 爬取房子详情页的内容
        Analysis_html(url_ls, headers)
        time.sleep(4)
        num += 1
        print("第%d页爬完了" % num)  # num记录爬取索成功的索引值


# 获取每一套房子的详情信息
def Analysis_html(url_ls, headers):
    for i in url_ls:
        r = requests.get(i, headers=headers)
        html = etree.HTML(r.text)
        name = html.xpath("//h1/text()")  # 在获取房名
        price = html.xpath("//span[@class='total']/text()")  # 获取总价
        area = html.xpath("//div[@class='area']/div[1]/text()")  # 获取面积
        priceAvg = html.xpath("//span[@class='unitPriceValue']/text()")  # 获取每平方的价格
        houseType = html.xpath("//div[@class='room']/div[@class='mainInfo']/text()")  # 获取房屋类型
        orientation = html.xpath("//div[@class='type']/div[@class='mainInfo']/text()")  # 获取房屋朝向
        city = html.xpath("//div[4]/div/div/a[1]/text()")  # 获取城市
        district = html.xpath("//div[@class='areaName']/span[@class='info']/a[1]/text()")  # 获取地区
        community = html.xpath("//div[@class='communityName']//a[1]/text()")  # 获取社区
        decoration = html.xpath("//div[@class='base']/div[@class='content']/ul/li[9]/text()")  # 获取装修情况
        propertyRight = html.xpath(
            "//div[@class='transaction']/div[@class='content']/ul/li[6]/span[2]/text()")  # 获取房屋产权
        lift = html.xpath("//div[@class='base']/div[@class='content']/ul/li[11]/text()")  # 获取电梯情况
        lifeRate = html.xpath("//div[@class='base']/div[@class='content']/ul/li[10]/text()")  # 获取梯户比例
        builtType = html.xpath("//div[@class='base']/div[@class='content']/ul/li[6]/text()")  # 获取建筑类型
        builtStructure = html.xpath("//div[@class='base']/div[@class='content']/ul/li[8]/text()")  # 获取建筑结构
        # 爬取过程中可能会出现广告，异常捕获跳过
        try:
            # 将爬取下来的数据存储到数据库
            Save_data(name, price, area, priceAvg, houseType, orientation, city, district, community, decoration,
                      propertyRight, lift, lifeRate, builtType, builtStructure)
            print(name, price, area, priceAvg, houseType, orientation, city, district, community, decoration,
                      propertyRight, lift, lifeRate, builtType, builtStructure)
        except:
            continue
        # 设置休眠实现
        time.sleep(random.randint(1, 3))


# 存储数据库函数
def Save_data(name, price, area, priceAvg, houseType, orientation, city, district, community, decoration, propertyRight,
              lift, lifeRate, builtType, builtStructure):
    result = []
    result = name + price + area + priceAvg + houseType + orientation + city + district + community + decoration + propertyRight + lift + lifeRate + builtType + builtStructure

    # 存储在数据库
    houseDetail = models.house(houseName=result[0], housePrice=result[1], houseArea=result[2], housePriceAvg=result[3],
                               houseType=result[4], houseOrientation=result[5], houseCity=result[6],
                               houseDistrict=result[7],
                               houseCommunity=result[8], houseDecoration=result[9], propertyRight=result[10],
                               lift=result[11], lifeRate=result[12], builtType=result[13], builtStructure=result[14])
    houseDetail.save()
    # 存储在csv文件
    with open(r'二手房房价2.csv', 'a', encoding='utf_8_sig', newline='')as f:
        wt = csv.writer(f)
        wt.writerow(result)
        print('已写入')
        f.close()


# 爬虫启动方法
def start():
    # 爬取所需城市的名称缩写，用于地址拼接，链家网站福建省内支持的城市仅有以下四个、支持的城市列表可以查看https://map.lianjia.com/map/350200/ESF/
    #cityList = ['xm', 'fz', 'quanzhou', 'zhangzhou']
    cityList = ['sh']
    # 设置请求头，以防被网站识别为爬虫
    headers = {
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.46",
        "Cookie": "lianjia_uuid=60c54ae6-6b42-4222-8692-efe4fb2c554e; crosSdkDT2019DeviceId=eu2b5l-mfq32x-njycw4jyhevekso-ud4cezlw8; _smt_uid=632e762f.20c8106d; _ga=GA1.2.203719967.1663989297; Hm_lvt_9152f8221cb6243a53c83b956842be8a=1669339641; lianjia_ssid=917d080d-af96-4114-92ce-e9ec6ded0cde; _gid=GA1.2.1343492366.1671084252; lianjia_token=2.0111ad29317b6152f82536df72a8e72e0777cf78c4; beikeBaseData=%7B%22parentSceneId%22:%226413099907731748097%22%7D; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%221836d7da82957c-0d0e66116b0617-78565473-1327104-1836d7da82a940%22%2C%22%24device_id%22%3A%221836d7da82957c-0d0e66116b0617-78565473-1327104-1836d7da82a940%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_referrer%22%3A%22%22%2C%22%24latest_referrer_host%22%3A%22%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%7D%7D; Hm_lpvt_9152f8221cb6243a53c83b956842be8a=1671089180; select_city=350200"
    }
    # 写入csv文件，只是备份一份数据，后面并无用到
    with open(r'二手房房价2.csv', 'a', encoding='utf_8_sig', newline='')as f:
        table_label = ['房屋名称', '房屋总价', '建筑面积', '每平方价格', '房屋户型', '房屋朝向',
                       '城市', '地区', '小区名称', '装修情况', '房屋产权', '电梯情况', '梯户比例', '建筑类型', '建筑结构']
        wt = csv.writer(f)
        wt.writerow(table_label)
    # 按城市列表里的值进行第一轮地址拼接
    for i in cityList:
        url = 'https://' + i + '.lianjia.com/ershoufang/'
        # url = 'https://xm.lianjia.com/ershoufang/'
        # 第二轮城市地址拼接
        all_url = Get_url(url)
        # 将所有url地址和请求头传入方法内进行采集
        Get_house_url(all_url, headers)


from django.http import HttpResponse, JsonResponse
from django.shortcuts import render

# Create your views here.
from django.views.decorators.clickjacking import xframe_options_exempt
from django.views.decorators.csrf import csrf_exempt
import numpy as np

from rj import models
from rj.utils import crawl, analyse


def index(request):
    return render(request, 'Index.html')


def spider_view(request):
    return render(request, 'Spider.html')


# 词云
def wordcloud_view(request):
    return render(request, 'WordCloud.html')


@xframe_options_exempt  # 使iframe可用
@csrf_exempt  # 取消csrf限制
def crawlData_view(request):
    # if len(models.house.objects.all()) < 5000:
    #     crawl.start()
    crawl.start()
    return JsonResponse({"flag": "爬取成功"})


@xframe_options_exempt  # 使iframe可用
@csrf_exempt  # 取消csrf限制
def analyse_view(request):
    df = analyse.get_pandas_data()
    if request.method == 'POST':
        cityName = request.POST.get('cityName')
        all_list = analyse.get_pie(df, cityName)
        districtList, priceList = analyse.analyse_district(df, cityName)
        return JsonResponse(
            {"all_list": all_list, "cityName": cityName, 'districtList': districtList, 'priceList': priceList})
    else:
        cityName = request.GET.get('cityName')

    cityList, priceList, unitPriceList, cityNum, itemCount, maxPrice, MEcity = analyse.get_house_price(df)
    all_list = analyse.get_pie(df)
    analyse.wordCloud(df)
    return render(request, 'Analysis.html',
                  {"cityList": cityList, 'priceList': priceList, 'unitPriceList': unitPriceList, 'cityNum': cityNum,
                   'itemCount': itemCount, 'maxPrice': maxPrice, 'all_list': all_list, 'MEcity': MEcity})

五、 总结

1.     经过对链家二手房数据的分析与可视化，可以看出，北京是爬取城市里最贵的，漳州相较于其他城市而言相对比较便宜。

2.     在这次实验过程中掌握了爬虫这门技术，并对大数据分析也有了更进一步的了解，通过这次的实验我发下编程不要急于求成，在后期的实验数据处理、结果分析阶段，如何将具体的实验数据整理、分析，从中提炼出对我们的分析有用的数据，是对我分析、研究能力的考验。通过这次的实验、研究，我在创新能力、动手能力、以及专业知识等方面都有不同程度的收获。