Pandas 基本使用
基本
创建 Dataframe
import pandas as pd
df = pd.DataFrame(
{
"Name": [
"Braund, Mr. Owen Harris",
"Allen, Mr. William Henry",
"Bonnell, Miss. Elizabeth",
"John, Mr. Peter Parker"
],
"Age": [22, 35, 58, 24],
"Sex": ["male", "male", "female", "male"],
"Fare": [1500, 1600, 1550, 3345],
"Pclass": [2, 2, 3, 3],
"Location": [
"China",
"America",
"Africa",
"Japan"
]
}
)
Name Age Sex Fare Pclass Location
0 Braund, Mr. Owen Harris 22 male 1500 2 China
1 Allen, Mr. William Henry 35 male 1600 2 America
2 Bonnell, Miss. Elizabeth 58 female 1550 3 Africa
3 John, Mr. Peter Parker 24 male 3345 3 Japan
创建 Series
ages = pd.Series([22, 35, 58], name="Age")
0 22
1 35
2 58
Name: Age, dtype: int64
选取一列
df["Age"]
0 22
1 35
2 58
3 24
Name: Age, dtype: int64
选取多列
df[["Name", "Age"]]
预览前几行
df.head() # 默认5行
df.head(8) # 可指定行数
预览后几行
df.tail() # 默认5行
df.tail(8) # 可指定行数
查看每一列数据类型
df.dtypes
Name object
Age int64
Sex object
Fare int64
Pclass int64
Location object
dtype: object
查看行列数
df.shape
(4, 6)
求最大值
仅对数字类型数据有效
df["Age"].max()
罗列基本统计信息
df.describe()
Age Fare Pclass
count 4.00000 4.000000 4.00000
mean 34.75000 1998.750000 2.50000
std 16.52019 898.428025 0.57735
min 22.00000 1500.000000 2.00000
25% 23.50000 1537.500000 2.00000
50% 29.50000 1575.000000 2.50000
75% 40.75000 2036.250000 3.00000
max 58.00000 3345.000000 3.00000
罗列全部数据类型
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 6 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Name 4 non-null object
1 Age 4 non-null int64
2 Sex 4 non-null object
3 Fare 4 non-null int64
4 Pclass 4 non-null int64
5 Location 4 non-null object
dtypes: int64(3), object(3)
memory usage: 320.0+ bytes
遍历数据集
for idx, sub_series in df.iterrows():
print(idx) # 行下标
print(sub_series) # 一行数据: numpy.Series
文件
读取文件
df = pd.read_csv("path.csv")
df = pd.read_csv("path.csv", index_col=0, parse_dates=True)
# index_col=0 以第 0 列 为索引
# parse_dates=True 将时间类型的列转义为时间戳类型
将数据导出为 Excel
df.to_excel(
"data.xlsx",
sheet_name="some_data",
index=False
)
筛选
大于某值
df[
df["Age"] > 35
]
Name Age Sex Fare Pclass Location
2 Bonnell, Miss. Elizabeth 58 female 1550 3 Africa
指定枚举值列表
df[
df["Fare"].isin(
[1500, 1600]
)
]
Name Age Sex Fare Pclass Location
0 Braund, Mr. Owen Harris 22 male 1500 2 China
1 Allen, Mr. William Henry 35 male 1600 2 America
多个条件并集
df[
(df["Pclass"] == 2) | (df["Pclass"] == 3)
]
# 只能用 | 或 &，不能用 or 或 and
Name Age Sex Fare Pclass Location
0 Braund, Mr. Owen Harris 22 male 1500 2 China
1 Allen, Mr. William Henry 35 male 1600 2 America
2 Bonnell, Miss. Elizabeth 58 female 1550 3 Africa
3 John, Mr. Peter Parker 24 male 3345 3 Japan
筛选非空值
df[
df["Age"].notna()
]
筛选后返回指定列
adult_names = df.loc[df["Age"] > 35, "Name"]
指定行区间、列区间
df.iloc[2:3, 1:4]
Age Sex Fare
2 58 female 1550
查看对限定条件的满足情况
0 False
1 False
2 True
3 False
Name: Age, dtype: bool
指定列的值选取子数据集
df.loc[df["Name"]=="Tom"]
正则筛选数据
match_result_series = series_data.str.contains(r".*TOM.*", regex=True).value_counts()
"""
计算出匹配了正则的个数, 包括匹配和不匹配的个数
返回为 Series, 有两行, True 和 False
如果没有匹配的项, True 没有, 要注意捕获 KeyError 的异常
"""
match_result_series[True] # 计算出匹配了正则的个数
更改
更改数据
df["Age"] = "Mike"
创建新列
df["new_age"] = df["Age"] * 1.882
df["age_fare"] = (df["Age"] / df["Fare"])
Name Age Sex Fare Pclass Location new_age age_fare
0 Braund, Mr. Owen Harris 22 male 1500 2 China 41.404 0.014667
1 Allen, Mr. William Henry 35 male 1600 2 America 65.870 0.021875
2 Bonnell, Miss. Elizabeth 58 female 1550 3 Africa 109.156 0.037419
3 John, Mr. Peter Parker 24 male 3345 3 Japan 45.168 0.007175
重命名列名
df_renamed = df.rename(
columns={
"Age": "X_Age",
"Fare": "X_Fare",
"Name": "X_Name",
}
)
将列名全转为小写
df_lower = df.rename(
columns=str.lower
)
在末尾新增一行
df.loc[df.index] = ["Tom", 20, "male", 1800, 3, "Japan"]
行去重
df.drop_duplicates(inplace=True) # inplace 为 True 会把原数据集修改
数据分析
求平均值
df["Age"].mean()
求中位数
df["Age"].median()
求最值
df["Age"].max()
df["Age"].min()
求偏度
df["Age"].skew()
一次性计算多列
df[
["Age", "Fare"]
].median()
一次性计算多列的多种统计类型
df.agg(
{
"Age": ["min", "max", "median", "skew"],
"Fare": ["min", "max", "median", "mean"]
}
)
Age Fare
min 22.000000 1500.00
max 58.000000 3345.00
median 29.500000 1575.00
skew 1.368208 NaN
mean NaN 1998.75
对全部数据分组
df.groupby("Age").mean()
Fare Pclass
Age
22 1500.0 2.0
24 3345.0 3.0
35 1600.0 2.0
58 1550.0 3.0
对指定几列分组
df[
["Fare", "Age"]
].groupby("Age").mean()
Fare
Age
22 1500.0
24 3345.0
35 1600.0
58 1550.0
分组后再筛选
df.groupby("Sex")["Age"].mean()
Sex
female 58.0
male 27.0
Name: Age, dtype: float64
df.groupby(
["Sex", "Pclass"]
)["Fare"].mean()
Sex Pclass
female 3 1550.0
male 2 1550.0
3 3345.0
Name: Fare, dtype: float64
枚举值计数
# 常规方法
df.groupby("Pclass")["Pclass"].count()
# 内置方法
df["Pclass"].value_counts()
Pclass
2 2
3 2
Name: Pclass, dtype: int64
排序
指定某列排序
默认升序
df.sort_values(
by="Age"
).head()
Name Age Sex Fare Pclass Location
0 Braund, Mr. Owen Harris 22 male 1500 2 China
3 John, Mr. Peter Parker 24 male 3345 3 Japan
1 Allen, Mr. William Henry 35 male 1600 2 America
2 Bonnell, Miss. Elizabeth 58 female 1550 3 Africa
降序
df.sort_values(
by="Age",
ascending=False
).head()
Name Age Sex Fare Pclass Location
2 Bonnell, Miss. Elizabeth 58 female 1550 3 Africa
1 Allen, Mr. William Henry 35 male 1600 2 America
3 John, Mr. Peter Parker 24 male 3345 3 Japan
0 Braund, Mr. Owen Harris 22 male 1500 2 China
按照行标签排序
df.sort_index().head()
Name Age Sex Fare Pclass Location
0 Braund, Mr. Owen Harris 22 male 1500 2 China
1 Allen, Mr. William Henry 35 male 1600 2 America
2 Bonnell, Miss. Elizabeth 58 female 1550 3 Africa
3 John, Mr. Peter Parker 24 male 3345 3 Japan
调整表格结构
宽表变窄表
宽表就是把字段全都堆在一个表中，没有用多对多活其他更符合数据库三范式的方式加以设计。感官上看字段非常多，数据非常冗余，但是理解起来简单，一看就懂。宽表的存在是为了用空间换时间。
窄表是相对于宽表而言，更符合逻辑。是为了表述某个维度的信息，从宽表中提取出若干列，组成一个新的表，便于做分析。
一个宽表的示例如下：
df = pd.DataFrame(
{
"phone": [
"iPhone 13 Pro",
"iPhone 13 Pro",
"iPhone 13 Pro",
"iPhone 13 Pro",
"Huawei Mate 40 Pro",
"Huawei Mate 40 Pro",
"Huawei Mate 40 Pro",
"Huawei Mate 40 Pro",
],
"store": [
"jingdong",
"tianmao",
"taobao",
"pinduoduo",
"jingdong",
"tianmao",
"taobao",
"pinduoduo"
],
"price": [
7999,
8399,
6569,
6754,
7688,
5499,
6799,
6588
]
}
)
这个表格的数据描述了京东、拼多多、淘宝、和天猫四家平台的 iphone 13 Pro 和 Huawei Mate 40 Pro 的价格数据（数据是瞎写的），可以看到 phone 和 store 这两列稍显冗余，重复内容过多。
phone store price
0 iPhone 13 Pro jingdong 7999
1 iPhone 13 Pro tianmao 8399
2 iPhone 13 Pro taobao 6569
3 iPhone 13 Pro pinduoduo 6754
4 Huawei Mate 40 Pro jingdong 7688
5 Huawei Mate 40 Pro tianmao 5499
6 Huawei Mate 40 Pro taobao 6799
7 Huawei Mate 40 Pro pinduoduo 6588
现在想要提取一个窄表，以 phone 为纵向索引，store 为横向字段，主体显示的值为 price。
df.pivot(
index="sex", # 索引
columns="id", # 列名
values="price" # 值
)
store jingdong pinduoduo taobao tianmao
phone
Huawei Mate 40 Pro 7688 6588 6799 5499
iPhone 13 Pro 7999 6754 6569 8399
对每行每列添加总计的数据。
df.pivot_table(
index="phone",
columns="store",
values="price",
aggfunc="mean", # 函数名称
margins=True # 显示开关
)
store jingdong pinduoduo taobao tianmao All
phone
Huawei Mate 40 Pro 7688.0 6588 6799 5499 6643.500
iPhone 13 Pro 7999.0 6754 6569 8399 7430.250
All 7843.5 6671 6684 6949 7036.875
重置索引
对于这个 DataFrame，当前的索引列是 phone 的数据。
store jingdong pinduoduo taobao tianmao
phone
Huawei Mate 40 Pro 7688 6588 6799 5499
iPhone 13 Pro 7999 6754 6569 8399
将其索引重置为数字序号：
df.pivot(
index="phone",
columns="store",
values="price"
).reset_index()
store phone jingdong pinduoduo taobao tianmao
0 Huawei Mate 40 Pro 7688 6588 6799 5499
1 iPhone 13 Pro 7999 6754 6569 8399
如果要丢弃原来的 phone 索引，加上 drop 参数：
df.pivot(
index="phone",
columns="store",
values="price"
).reset_index(drop=True)
store jingdong pinduoduo taobao tianmao
0 7688 6588 6799 5499
1 7999 6754 6569 8399
联结 datafame
有这样两个结构相似的 dataframe：
df1 = pd.DataFrame(
{
"Name": [
"Braund, Mr. Owen Harris",
"Allen, Mr. William Henry"
],
"Age": [22, 35],
"Sex": ["male", "male"],
"Fare": [1500, 1600],
"Pclass": [2, 2],
"Location": [
"China",
"America"
]
}
)
第一个表格是一个中国人和一个美国人的信息：
Name Age Sex Fare Pclass Location
0 Braund, Mr. Owen Harris 22 male 1500 2 China
1 Allen, Mr. William Henry 35 male 1600 2 America
df2 = pd.DataFrame(
{
"Name": [
"Bonnell, Miss. Elizabeth",
"John, Mr. Peter Parker"
],
"Age": [58, 24],
"Sex": ["female", "male"],
"Fare": [1550, 3345],
"Pclass": [3, 3],
"Location": [
"Africa",
"Japan"
]
}
)
第二个表格是日本人和非洲人的信息：
Name Age Sex Fare Pclass Location
0 Bonnell, Miss. Elizabeth 58 female 1550 3 Africa
1 John, Mr. Peter Parker 24 male 3345 3 Japan
将他们合并成一个 dataframe：
pd.concat(
[df1, df2],
axis=0 # 0 表示上下合并，1 表示左右合并
)
Name Age Sex Fare Pclass Location
0 Braund, Mr. Owen Harris 22 male 1500 2 China
1 Allen, Mr. William Henry 35 male 1600 2 America
0 Bonnell, Miss. Elizabeth 58 female 1550 3 Africa
1 John, Mr. Peter Parker 24 male 3345 3 Japan
合并 dataframe
对于以下两个 dataframe：
df1 = pd.DataFrame(
{
"Name": [
"Braund, Mr. Owen Harris",
"Allen, Mr. William Henry"
],
"Age": [22, 35]
}
)
Name Age
0 Braund, Mr. Owen Harris 22
1 Allen, Mr. William Henry 35
df2 = pd.DataFrame(
{
"Age": [22, 35],
"Sex": ["female", "male"],
"Fare": [1550, 3345],
"Pclass": [3, 3],
"Location": [
"Africa",
"Japan"
]
}
)
Age Sex Fare Pclass Location
0 22 female 1550 3 Africa
1 35 male 3345 3 Japan
他们有相同的一列 Age，可以以 Age 为基准，将两个 dataframe 合并：
df = pd.merge(
df1,
df2,
how="left",
on="Age" # 以 Age 为基准
)
Name Age Sex Fare Pclass Location
0 Braund, Mr. Owen Harris 22 female 1550 3 Africa
1 Allen, Mr. William Henry 35 male 3345 3 Japan
