Python数据分析-导入外部数据
Python数据分析-导入外部数据
1. 主要内容
- 导入.xls或.xlsx文件（重点）
- 导入.csv文件
- 导入.txt文本文件
- 导入HTML网页
2. 导入.xls或.xlsx文件
导入.xls或.xlsx文件主要使用Pandas的read_excel()方法，语法如下：
pandas.read_excel(io, sheet_name=0, *, header=0, names=None, index_col=None, usecols=None, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skiprows=None, nrows=None, na_values=None, keep_default_na=True, na_filter=True, verbose=False, parse_dates=False, date_parser=_NoDefault.no_default, date_format=None, thousands=None, decimal='.', comment=None, skipfooter=0, storage_options=None, dtype_backend=_NoDefault.no_default, engine_kwargs=None) ## 说明：主要常用前两个参数
参数说明：
支持从本地文件系统或URL读取的xls、xlsx、xlsm、xlsb、odf、ods和odt文件扩展名。支持读取单个工作表或工作表列表的选项。
- io：str, bytes, ExcelFile, xlrd.Book, path object, or file-like object
Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected.
- sheet_name：str, int, list, or None, default 0
Strings are used for sheet names. Integers are used in zero-indexed sheet positions (chart sheets do not count as a sheet position). Lists of strings/integers are used to request multiple sheets. Specify to get all worksheets.
- Defaults to 0：1st sheet as a DataFrame
- 1：2nd sheet as a DataFrame
- "Sheet1"：Load sheet with name “Sheet1”
- [0, 1, "Sheet5"]： Load first, second and sheet named “Sheet5” as a dict of DataFrame
- None：All worksheets.
- header：int, list of int, default 0
Row (0-indexed) to use for the column labels of the parsed DataFrame. If a list of integers is passed those row positions will be combined into a MultiIndex. Use None if there is no header.
- names：array-like, default None
List of column names to use. If file contains no header row, then you should explicitly pass header=None.
- index_col：int, str, list of int, default None
Column (0-indexed) to use as the row labels of the DataFrame. Pass None if there is no such column. If a list is passed, those columns will be combined into a MultiIndex. If a subset of data is selected with usecols, index_col is based on the subset.
Missing values will be forward filled to allow roundtripping with to_excel for merged_cells=True. To avoid forward filling the missing values use set_index after reading the data instead of index_col.
- usecols：str, list-like, or callable, default None
- If None, then parse all columns.
- If str, then indicates comma separated list of Excel column letters and column ranges (e.g. “A:E” or “A,C,E:F”). Returns a subset of the columns according to behavior above.
- dtypeType ：name or dict of column -> type, default None
Data type for data or columns. E.g. {‘a’: np.float64, ‘b’: np.int32} Use object to preserve data as stored in Excel and not interpret dtype, which will necessarily result in object dtype. If converters are specified, they will be applied INSTEAD of dtype conversion. None
- engine：{‘openpyxl’, ‘calamine’, ‘odf’, ‘pyxlsb’, ‘xlrd’}, default None
If io is not a buffer or path, this must be set to identify io. Engine compatibility :
- openpyxl supports newer Excel file formats.
- calamine supports Excel (.xls, .xlsx, .xlsm, .xlsb) and OpenDocument (.ods) file formats.
- odf supports OpenDocument file formats (.odf, .ods, .odt).
- pyxlsb supports Binary Excel files.
- xlrd supports old-style Excel files (.xls).
When engine=None, the following logic will be used to determine the engine:
- If path_or_buffer is an OpenDocument format (.odf, .ods, .odt), then odf will be used.
- Otherwise if path_or_buffer is an xls format, xlrd will be used.
- Otherwise if path_or_buffer is in xlsb format, pyxlsb will be used.
- Otherwise openpyxl will be used.
- converters：dict, default None
Dict of functions for converting values in certain columns. Keys can either be integers or column labels, values are functions that take one input argument, the Excel cell content, and return the transformed content.
- true_values：list, default None
Values to consider as True.
- false_values：list, default None
Values to consider as False.
- skiprows：list-like, int, or callable, optional
Line numbers to skip (0-indexed) or number of lines to skip (int) at the start of the file. If callable, the callable function will be evaluated against the row indices, returning True if the row should be skipped and False otherwise. lambda x: x in [0, 2]
- nrows：int, default None
Number of rows to parse.
- na_values：scalar, str, list-like, or dict, default None
Additional strings to recognize as NA/NaN. If dict passed, specific per-column NA values. By default the following values are interpreted as NaN: ‘’, ‘#N/A’, ‘#N/A N/A’, ‘#NA’, ‘-1.#IND’, ‘-1.#QNAN’, ‘-NaN’, ‘-nan’, ‘1.#IND’, ‘1.#QNAN’, ‘<NA>’, ‘N/A’, ‘NA’, ‘NULL’, ‘NaN’, ‘None’, ‘n/a’, ‘nan’, ‘null’.
- keep_default_na：bool, default True
Whether or not to include the default NaN values when parsing the data. Depending on whether na_values is passed in, the behavior is as follows:
- If keep_default_na is True, and na_values are specified, na_values is appended to the default NaN values used for parsing.
- If keep_default_na is True, and na_values are not specified, only the default NaN values are used for parsing.
- If keep_default_na is False, and na_values are specified, only the NaN values specified na_values are used for parsing.
- If keep_default_na is False, and na_values are not specified, no strings will be parsed as NaN.
Note that if na_filter is passed in as False, the keep_default_na and na_values parameters will be ignored.
- na_filter：bool, default True
Detect missing value markers (empty strings and the value of na_values). In data without any NAs, passing na_filter=False can
- verbose：bool, default False
Indicate number of NA values placed in non-numeric columns.
- parse_dates：bool, list-like, or dict, default False
The behavior is as follows:
- bool. If True -> try parsing the index.
- list of int or names. e.g. If [1, 2, 3] -> try parsing columns 1, 2, 3 each as a separate date column.
- list of lists. e.g. If [[1, 3]] -> combine columns 1 and 3 and parse as a single date column.
- dict, e.g. {‘foo’ : [1, 3]} -> parse columns 1, 3 as date and call result ‘foo’
If a column or index contains an unparsable date, the entire column or index will be returned unaltered as an object data type. If you don`t want to parse some cells as date just change their type in Excel to “Text”. For non-standard datetime parsing, use pd.to_datetimeafter pd.read_excel.
Note: A fast-path exists for iso8601-formatted dates.
- date_parser：function, optional
Function to use for converting a sequence of string columns to an array of datetime instances. The default uses dateutil.parser.parser to do the conversion. Pandas will try to call date_parser in three different ways, advancing to the next if an exception occurs:
- date_format：str or dict of column -> format, default None
If used in conjunction with parse_dates, will parse dates according to this format. For anything more complex, please read in as object and then apply to_datetime()as-needed.
- thousands：str, default None
Thousands separator for parsing string columns to numeric. Note that this parameter is only necessary for columns stored as TEXT in Excel, any numeric columns will automatically be parsed, regardless of display format.
- decimal：str, default ‘.’
Character to recognize as decimal point for parsing string columns to numeric. Note that this parameter is only necessary for columns stored as TEXT in Excel, any numeric columns will automatically be parsed, regardless of display format.(e.g. use ‘,’ for European data).
- comment：str, default None
Comments out remainder of line. Pass a character or characters to this argument to indicate comments in the input file. Any data between the comment string and the end of the current line is ignored.
- skipfooter：int, default 0
Rows at the end to skip (0-indexed).
- storage_options：dict, optional
Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc.
- dtype_backend：{‘numpy_nullable’, ‘pyarrow’}, default ‘numpy_nullable’
Back-end data type applied to the resultant DataFrame (still experimental). Behaviour is as follows:
- "numpy_nullable": returns nullable-dtype-backed DataFrame (default).
- "pyarrow": returns pyarrow-backed nullable ArrowDtype DataFrame.
- engine_kwargs：dict, optional
Arbitrary keyword arguments passed to excel engine.
返回值：
DataFrame or dict of DataFrames.
代码示例：
- 指定io、Index和sheet_name参数
1 pd.read_excel('tmp.xlsx', index_col=0) 2 3 ### 结果 4 Name Value 5 0 string1 1 6 1 string2 2 7 2 #Comment 3
1 pd.read_excel(open('tmp.xlsx', 'rb'),sheet_name='Sheet3') 2 3 ### 结果 4 Unnamed: 0 Name Value 5 0 0 string1 1 6 1 1 string2 2 7 2 2 #Comment 3
- 指定Index和header
1 pd.read_excel('tmp.xlsx', index_col=None, header=None) 2 3 ### 结果 4 0 1 2 5 0 NaN Name Value 6 1 0.0 string1 1 7 2 1.0 string2 2 8 3 2.0 #Comment 3
- 显式指定Column参数
1 >>> pd.read_excel('tmp.xlsx', index_col=0, dtype={'Name': str, 'Value': float}) 2 3 ### 结果 4 Name Value 5 0 string1 1.0 6 1 string2 2.0 7 2 #Comment 3.0
- 显式指定True、False和NA值以及数千个分隔符（提供您想要的字符串或字符串列表的值!）
1 >>> pd.read_excel('tmp.xlsx', index_col=0, na_values=['string1', 'string2']) 2 3 ### 结果 4 Name Value 5 0 NaN 1 6 1 NaN 2 7 2 #Comment 3
- 可以使用注释kwarg跳过excel输入文件中的注释行。
1 >>> pd.read_excel('tmp.xlsx', index_col=0, comment='#') 2 3 ### 结果 4 Name Value 5 0 string1 1.0 6 1 string2 2.0 7 2 None NaN
3. 导入.csv或.txt文件
导入.csv文件主要使用Pandas的read_csv()方法，语法如下：
pandas.read_csv(filepath_or_buffer, *, sep=_NoDefault.no_default, delimiter=None, header='infer', names=_NoDefault.no_default, index_col=None, usecols=None, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skipinitialspace=False, skiprows=None, skipfooter=0, nrows=None, na_values=None, keep_default_na=True, na_filter=True, verbose=_NoDefault.no_default, skip_blank_lines=True, parse_dates=None, infer_datetime_format=_NoDefault.no_default, keep_date_col=_NoDefault.no_default, date_parser=_NoDefault.no_default, date_format=None, dayfirst=False, cache_dates=True, iterator=False, chunksize=None, compression='infer', thousands=None, decimal='.', lineterminator=None, quotechar='"', quoting=0, doublequote=True, escapechar=None, comment=None, encoding=None, encoding_errors='strict', dialect=None, on_bad_lines='error', delim_whitespace=_NoDefault.no_default, low_memory=True, memory_map=False, float_precision=None, storage_options=None, dtype_backend=_NoDefault.no_default)
参数说明：
与read_excel()方法类似，相关参数说明可前往pandas官网查询。
4. 导入HTML网页
pandas中的read_html()函数是将HTML的表格转换为DataFrame或list的一种快速方便的方法，这个函数对于快速合并来自不同网页上的表格非常有用。
pandas.read_html(io, *, match='.+', flavor=None, header=None, index_col=None, skiprows=None, attrs=None, parse_dates=False, thousands=',', encoding=None, decimal='.', converters=None, na_values=None, keep_default_na=True, displayed_only=True, extract_links=None, dtype_backend=_NoDefault.no_default, storage_options=None)
参数说明：
与read_excel()方法类似，相关参数说明可前往pandas官网查询。
时间：2024年2月1日
浙公网安备 33010602011771号