GIS+=地理信息+大数据——Windows部署Pandas环境及代码測试验证

--------------------------------------------------------------------------------------

Blog:    http://blog.csdn.net/chinagissoft

QQ群：16403743

宗旨：专注于"GIS+"前沿技术的研究与交流。将云计算技术、大数据技术、容器技术、物联网与GIS进行深度融合，探讨"GIS+"技术和行业解决方式

转载说明：文章同意转载，但必须以链接方式注明源地址，否则追究法律责任!

--------------------------------------------------------------------------------------


题记


Python Data Analysis Library 或 pandas 是基于NumPy 的一种工具，该工具是为了解决数据分析任务而创建的。Pandas 纳入了大量库和一些标准的数据模型。提供了高效地操作大型数据集所需的工具。pandas提供了大量能使我们高速便捷地处理数据的函数和方法。

你非常快就会发现，它是使Python成为强大而高效的数据分析环境的重要因素之中的一个。



本篇文章主要介绍一下在Windows环境下关于Pandas的部署。主要是一般在Windows作为开发环境。调试代码也比較方便，因为Linux我们直接使用pip install pandas安装就可以，在Windows环境下可能还须要留意一些相关内容。


系统环境

  • Windows 10
  • Python 3.4（因为我须要使用GIScript2015。须要Python3.4支持。所以用户使用默认的Python2.6.7也能够）
  • VS2012（可选）
  • PyCharm（Python 开发的IDE，可选）


部署步骤

1、假设你安装了PyCharm。在安装不论什么Python相关的软件包都比較方便

打开PyCharm，点击文件菜单。选择setting，选择Project interprete，选择某个Pythonproject。点击右上角绿色+号。能够选择相关的Python库进行安装。比如Pandas



这样的方式比較方便。只是在安装过程中也提示了相关错误：Microsoft Visual C++ 10.0 is required Unable to find vcvarsall.bat



解决的方法：

这是由于须要依赖于VC++2010库，假设你安装了VS201X系列IDE，你能够使用例如以下方式解决：

加入VS2010的环境变量，相应你安装相关VS系列环境变量。加入VS90COMNTOOLSkeyword，然后设置相关的值就可以。

假设你使用 python 2.x

    • Visual Studio 2010 (VS10):SET VS90COMNTOOLS=%VS100COMNTOOLS%
    • Visual Studio 2012 (VS11):SET VS90COMNTOOLS=%VS110COMNTOOLS%
    • Visual Studio 2013 (VS12):SET VS90COMNTOOLS=%VS120COMNTOOLS%
    • Visual Studio 2015 (VS15):SET VS90COMNTOOLS=%VS140COMNTOOLS%

假设你使用 python 3.x 

    • Visual Studio 2010 (VS10):SET VS100COMNTOOLS=%VS100COMNTOOLS%
    • Visual Studio 2012 (VS11):SET VS100COMNTOOLS=%VS110COMNTOOLS%
    • Visual Studio 2013 (VS12):SET VS100COMNTOOLS=%VS120COMNTOOLS%
    • Visual Studio 2015 (VS15):SET VS100COMNTOOLS=%VS140COMNTOOLS%

假设部署环境没有安装不论什么VS环境。建议安装一个VS2010Express1。

http://download.microsoft.com/download/1/E/5/1E5F1C0A-0D5B-426A-A603-1798B951DDAE/VS2010Express1.iso


2、当然，假设你没有安装PyCharm，你能够在Windows环境下使用pip进行安装


a：下载pip部署包

https://pypi.python.org/pypi/pip#downloads

随意解压到某个位置

b：安装pip

C:\Users\Administrator>python c:\Python34\pip-8.0.2\setup.py install
running install
running bdist_egg
running egg_info
creating pip.egg-info
writing requirements to pip.egg-info\requires.txt
writing dependency_links to pip.egg-info\dependency_links.txt
writing pip.egg-info\PKG-INFO
writing entry points to pip.egg-info\entry_points.txt
writing top-level names to pip.egg-info\top_level.txt
writing manifest file 'pip.egg-info\SOURCES.txt'
warning: manifest_maker: standard file 'setup.py' not found

reading manifest file 'pip.egg-info\SOURCES.txt'
writing manifest file 'pip.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
warning: install_lib: 'build\lib' does not exist -- no Python modules to install

creating build
creating build\bdist.win-amd64
creating build\bdist.win-amd64\egg
creating build\bdist.win-amd64\egg\EGG-INFO
copying pip.egg-info\PKG-INFO -> build\bdist.win-amd64\egg\EGG-INFO
copying pip.egg-info\SOURCES.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying pip.egg-info\dependency_links.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying pip.egg-info\entry_points.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying pip.egg-info\not-zip-safe -> build\bdist.win-amd64\egg\EGG-INFO
copying pip.egg-info\requires.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying pip.egg-info\top_level.txt -> build\bdist.win-amd64\egg\EGG-INFO
creating dist
creating 'dist\pip-8.0.2-py3.4.egg' and adding 'build\bdist.win-amd64\egg' to it
removing 'build\bdist.win-amd64\egg' (and everything under it)
Processing pip-8.0.2-py3.4.egg
creating c:\python34\lib\site-packages\pip-8.0.2-py3.4.egg
Extracting pip-8.0.2-py3.4.egg to c:\python34\lib\site-packages
Adding pip 8.0.2 to easy-install.pth file
Installing pip-script.py script to C:\Python34\Scripts
Installing pip.exe script to C:\Python34\Scripts
Installing pip3.4-script.py script to C:\Python34\Scripts
Installing pip3.4.exe script to C:\Python34\Scripts
Installing pip3-script.py script to C:\Python34\Scripts
Installing pip3.exe script to C:\Python34\Scripts

Installed c:\python34\lib\site-packages\pip-8.0.2-py3.4.egg
Processing dependencies for pip==8.0.2
Finished processing dependencies for pip==8.0.2

C:\Users\Administrator>

c：加入环境变量，将C:\Python34\Scripts;加入到环境变量

C:\Users\Administrator>path
PATH=C:\Python34\Scripts;C:\Python34\DLLs\Bin;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Python34\;C:\instantclient_11_2;C:\app\Administrator\product\11.2.0\dbhome_1\bin;C:\Program Files (x86)\Common Files\NetSarang;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\strawberry\c\bin;C:\strawberry\perl\bin;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v1.0\;C:\Program Files (x86)\Windows Kits\8.0\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\SSH Communications Security\SSH Secure Shell;C:\Windows\SysWOW64;C:\Program Files\TortoiseSVN\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\SSH Communications Security\SSH Secure Shell

d：验证pip安装

C:\Users\Administrator>pip list
numpy (1.10.4)
pandas (0.17.1)
pip (8.0.2)
python-dateutil (2.4.2)
pytz (2015.7)
setuptools (12.0.5)
six (1.10.0)
wheel (0.29.0)

e：安装pandas

C:\Users\Administrator>pip install pandas
Requirement already satisfied (use --upgrade to upgrade): pandas in c:\python34\lib\site-packages
Requirement already satisfied (use --upgrade to upgrade): pytz>=2011k in c:\python34\lib\site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil>=2 in c:\python34\lib\site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.0 in c:\python34\lib\site-packages (from pandas)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in c:\python34\lib\site-packages (from python-dateutil>=2->pandas)

3、当然，你也能够去pandas的官网下载软件包

https://pypi.python.org/pypi/pandas/0.17.1/#downloads


验证pandas

前面说了那么多部署，我们看看假设来运行一个简单的pandas代码验证一下。

因为Pandas能够用于大数据量的分析，当然特别适合于csv的文本数据。你能够先使用csv数据打开。


測试环境

    • CPU：Intel(R) Core(TM) i7-4710MQ CPU @2.5GHz 
    • 内存：16GB DDR3
    • 硬盘：C盘（SSD）。其它盘为普通硬盘


Pandas提供了IO工具能够将大文件分块读取，完整载入约1500万条数据须要57秒左右。

__author__ = 'Administrator'


import pandas as pd
from datetime import datetime
if __name__ == '__main__':
    starttime = datetime.now()
    reader = pd.read_csv('c:\\1.csv', iterator=True)
    try:
        df = reader.get_chunk(5000000)
        print(df.describe())
        endtime = datetime.now()
        costtime=(endtime-starttime).seconds
        print('cost time:'+repr(costtime))
    except StopIteration:
        print ("Iteration is stopped.")




当然。也能够通过使用不同分块大小来读取再调用 pandas.concat 连接DataFrame。chunkSize设置在150万条左右測试。

__author__ = 'Administrator'


import pandas as pd
from datetime import datetime
if __name__ == '__main__':
    starttime = datetime.now()
    reader = pd.read_csv('c:\\1.csv', iterator=True)
    loop = True
    chunkSize = 1500000
    chunks = []
    while loop:
        try:
            chunk = reader.get_chunk(chunkSize)
            chunks.append(chunk)
        except StopIteration:
            loop = False
            print ("Iteration is stopped.")
    df = pd.concat(chunks, ignore_index=True)
    print(df.describe())
    endtime = datetime.now()
    costtime=(endtime-starttime).seconds
    print('cost time:'+repr(costtime))

測试结果

C:\Python34\python.exe D:/GIScipt/Zagi/JobWorker/Core/Sys/big.py

             rate_code  passenger_count  trip_time_in_secs    trip_distance  \
count  14776615.000000  14776615.000000    14776615.000000  14776615.000000   
mean          1.034273         1.697372         683.423593         2.770976   
std           0.338771         1.365396         494.406260         3.305923   
min           0.000000         0.000000           0.000000         0.000000   
25%           1.000000         1.000000         360.000000         1.000000   
50%           1.000000         1.000000         554.000000         1.700000   
75%           1.000000         2.000000         885.000000         3.060000   
max         210.000000       255.000000       10800.000000       100.000000   

       pickup_longitude  pickup_latitude  dropoff_longitude  dropoff_latitude  
count   14776615.000000  14776615.000000    14776529.000000   14776529.000000  
mean         -72.636340        40.014399         -72.594427         39.992189  
std           10.138193         7.789904          10.288603          7.537067  
min        -2771.285400     -3547.920700       -2350.955600      -3547.920700  
25%          -73.991882        40.735512         -73.991211         40.734684  
50%          -73.981659        40.753147         -73.980125         40.753620  
75%          -73.966843        40.767288         -73.963898         40.768192  
max          112.404180      3310.364500        2228.737500       3477.105500  
cost time:62


Pandas很多其它了解：



參考文献

使用Python Pandas处理亿级数据：http://www.justinablog.com/archives/1357




posted @ 2017-06-22 09:38  jzdwajue  阅读(246)  评论(0)    收藏  举报