⚡ S6E9 电动汽车购买：用数据配方做可视化 EDA
⚡ S6E9 电动汽车购买：用数据配方做可视化 EDA
大家好！👋
在本 notebook 中，我将一步一步地梳理 Predicting Electric Vehicle Purchases 的数据。
计划如下：
- 初步查看各列（比赛数据 vs. 原始数据集）
- 看看哪些特征真正影响购买率
- 检查原始数据背后的两个隐藏公式，以及一个零训练的公式能达到什么水平
- 看看数值列中一些树模型喜欢的怪异之处
评估指标是 Will_Buy_EV 上的 ROC-AUC。
import glob
import warnings
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import roc_auc_score
warnings.filterwarnings("ignore")
sns.set_theme(style="whitegrid", palette="Set2")
0. 加载数据
除了比赛文件之外，我还附上了生成比赛数据所用的原始数据集。
# Find the competition files wherever Kaggle mounted them
train_path = [p for p in glob.glob("/kaggle/input/**/train.csv", recursive=True) if "s6e9" in p or "playground" in p][0]
test_path = train_path.replace("train.csv", "test.csv")
# The original (small) dataset
orig_path = glob.glob("/kaggle/input/**/EV_Adoption_and_Range_Anxiety_Dataset.csv", recursive=True)[0]
train = pd.read_csv(train_path)
test = pd.read_csv(test_path)
orig = pd.read_csv(orig_path)
# Numeric target: 1 = will buy an EV
train["y"] = (train["Will_Buy_EV"] == "Yes").astype(int)
orig["y"] = (orig["Will_Buy_EV"] == "Yes").astype(int)
print(f"train : {train.shape}")
print(f"test : {test.shape}")
print(f"original : {orig.shape}")
print()
print(f"purchase rate in train : {train['y'].mean():.2%}")
print(f"purchase rate in original : {orig['y'].mean():.2%}")
train : (668665, 16)
test : (286571, 14)
original : (10000, 16)
purchase rate in train : 17.46%
purchase rate in original : 17.50%
train.head()
id Age Annual_Income_USD Daily_Commute_km Number_of_Cars_Owned \
0 0 66 92887.0 23.4 2
1 1 38 30000.0 5.0 1
2 2 26 94389.0 36.8 1
3 3 66 73580.0 23.7 2
4 4 54 57898.0 50.8 1
Charging_Stations_Near_Home Charging_Stations_Near_Work \
0 3 7
1 2 2
2 8 15
3 6 9
4 2 3
Environmental_Concern_Level Gender City_Type Current_Car_Type \
0 1.0 Male Suburban Sedan
1 4.0 Male Rural SUV
2 5.0 Female Urban Sedan
3 3.0 Male Suburban Hatchback
4 3.0 Male Suburban Hatchback
Home_Charging_Possible Subsidy_Available Range_Anxiety_Level Will_Buy_EV y
0 Yes No Low No 0
1 Yes No Low No 0
2 No Yes Low Yes 1
3 Yes No Low No 0
4 Yes No Low No 0
1. 数据概览
比赛数据没有缺失值。
原始数据集在 income、commute 和 environmental concern 上约有 2% 的缺失。
共有 7 个数值列和 6 个类别列。
numeric_cols = [
"Age",
"Annual_Income_USD",
"Daily_Commute_km",
"Number_of_Cars_Owned",
"Charging_Stations_Near_Home",
"Charging_Stations_Near_Work",
"Environmental_Concern_Level",
]
categorical_cols = [
"Gender",
"City_Type",
"Current_Car_Type",
"Home_Charging_Possible",
"Subsidy_Available",
"Range_Anxiety_Level",
]
print("missing values in train :", int(train.isna().sum().sum()))
orig_missing = orig.isna().sum()
print("missing values in original :", orig_missing[orig_missing > 0].to_dict())
missing values in train : 0
missing values in original : {'Annual_Income_USD': 178, 'Daily_Commute_km': 181, 'Environmental_Concern_Level': 184}
我们来比较一下 train、test 和原始数据的数值分布。
fig, axes = plt.subplots(2, 4, figsize=(20, 8))
for ax, col in zip(axes.ravel(), numeric_cols):
ax.hist(train[col], bins=60, density=True, alpha=0.6, label="train")
ax.hist(test[col], bins=60, density=True, alpha=0.4, label="test")
# Original data as a black outline
ax.hist(orig[col].dropna(), bins=60, density=True, histtype="step", lw=2, color="black", label="original")
ax.set_title(col)
axes[0, 0].legend()
axes.ravel()[-1].axis("off")
plt.tight_layout()
plt.show()
train 和 test 几乎完全重叠，并且都紧密贴合原始分布。👍
2. 什么在影响购买率？
首先看类别列。
fig, axes = plt.subplots(2, 3, figsize=(18, 9))
for ax, col in zip(axes.ravel(), categorical_cols):
# Purchase rate and group size for each category
stats = train.groupby(col)["y"].agg(["mean", "size"]).sort_values("mean")
ax.bar(stats.index.astype(str), stats["mean"], color=sns.color_palette("Set2")[: len(stats)])
# Label each bar with its rate and row count
for i, (rate, size) in enumerate(zip(stats["mean"], stats["size"])):
ax.text(i, rate, f"{rate:.1%}\nn={size:,}", ha="center", va="bottom", fontsize=9)
ax.set_title(f"Purchase rate by {col}")
ax.set_ylim(0, stats["mean"].max() * 1.3)
plt.tight_layout()
plt.show()
Subsidy 和 range anxiety 明显占主导地位。
Gender、city type 和 car type 单独来看几乎不重要。
接下来看数值侧：income、environmental concern 和 age。
fig, axes = plt.subplots(1, 3, figsize=(18, 4.5))
# (column, number of quantile bins) - None means "use the raw values"
plot_specs = [
("Annual_Income_USD", 40),
("Environmental_Concern_Level", None),
("Age", 15),
]
for ax, (col, n_bins) in zip(axes, plot_specs):
if n_bins is None:
groups = train[col]
else:
groups = pd.qcut(train[col], n_bins, duplicates="drop")
rate = train.groupby(groups, observed=True)["y"].mean()
ax.plot(range(len(rate)), rate.values, marker="o")
ax.set_title(f"Purchase rate vs {col}")
# Show only a handful of tick labels so they stay readable
step = max(1, len(rate) // 8)
ax.set_xticks(range(0, len(rate), step))
ax.set_xticklabels([str(label)[:12] for label in rate.index[::step]], rotation=30)
plt.tight_layout()
plt.show()
Income 和 environmental concern 稳步推高购买率。
Age 基本持平。
3. 数据背后的配方 🧪
原始数据集遵循两个简单的公式（Chris Deotte 的 EDA 中有完整推导）：
- worry score = commute − 5 × (chargers near home) − 5 × (chargers near work) − 150 × (home charging) + noise
- 然后切分为 Low / Medium / High 三档 range anxiety
- buy score = 1.2 × income / 100k + 0.6 × concern + 2 × subsidy − 1 × [Medium anxiety] − 3 × [High anxiety] + noise
- 若分数高于 5.5，则该人会购买
我们把它们写成简单的函数。
def buy_score(df):
"""Purchase score from the original data recipe (without the random noise)."""
score = (
1.2 * df["Annual_Income_USD"] / 100_000
+ 0.6 * df["Environmental_Concern_Level"]
+ 2.0 * (df["Subsidy_Available"] == "Yes")
- 1.0 * (df["Range_Anxiety_Level"] == "Medium")
- 3.0 * (df["Range_Anxiety_Level"] == "High")
)
return score
def worry_score(df):
"""Range-anxiety score from the original data recipe (without the random noise)."""
score = (
df["Daily_Commute_km"]
- 5 * df["Charging_Stations_Near_Home"]
- 5 * df["Charging_Stations_Near_Work"]
- 150 * (df["Home_Charging_Possible"] == "Yes")
)
return score
train["buy_score"] = buy_score(train)
train["worry_score"] = worry_score(train)
auc = roc_auc_score(train["y"], train["buy_score"])
print(f"AUC of the buy-score formula (no training at all): {auc:.5f}")
AUC of the buy-score formula (no training at all): 0.93769
fig, axes = plt.subplots(1, 2, figsize=(16, 4.5))
# Left: purchase rate along the buy score
score_bins = pd.cut(train["buy_score"], np.arange(0, 10.01, 0.25))
rate = train.groupby(score_bins, observed=True)["y"].mean()
axes[0].plot([interval.mid for interval in rate.index], rate.values, marker="o")
axes[0].axvline(5.5, ls="--", color="black")
axes[0].set_title("Purchase rate along the buy score")
axes[0].set_xlabel("buy score")
# Right: range anxiety levels along the worry score
for level in ["Low", "Medium", "High"]:
values = train.loc[train["Range_Anxiety_Level"] == level, "worry_score"]
axes[1].hist(values, bins=80, alpha=0.6, density=True, label=level)
for cut in (-25, 75):
axes[1].axvline(cut, ls="--", color="black")
axes[1].set_title("Range anxiety along the worry score")
axes[1].legend()
plt.tight_layout()
plt.show()
一个无需训练的公式就已经达到约 0.938 AUC。🤯
调优良好的 GBDT 大约落在 0.946 左右，所以我们的模型其实是在争夺最后千分之几的差距。
4. 数值列中的怪异之处
数据生成器复用了原始数据集中的 income 值：
- 在恰好 $30,000 处有一个尖峰（原始数据中的收入下限）
- train 中几乎每个 income 值都原封不动地出现在原始文件中
original_incomes = set(orig["Annual_Income_USD"].dropna())
share_in_orig = train["Annual_Income_USD"].isin(original_incomes).mean()
print(f"train incomes that appear verbatim in the original : {share_in_orig:.1%}")
print(f"rows with income == 30,000 : {(train['Annual_Income_USD'] == 30_000).mean():.2%}")
print(f"rows with commute == 5.0 km : {(train['Daily_Commute_km'] == 5).mean():.2%}")
train incomes that appear verbatim in the original : 97.9%
rows with income == 30,000 : 9.21%
rows with commute == 5.0 km : 21.58%
income_counts = train["Annual_Income_USD"].value_counts()
fig, axes = plt.subplots(1, 2, figsize=(16, 4.5))
# Left: how often does each exact income value repeat?
axes[0].hist(np.log10(income_counts.values), bins=50)
axes[0].set_title("How often each exact income value repeats (log10 count)")
# Right: does the repeat count relate to the purchase rate?
income_freq = train["Annual_Income_USD"].map(income_counts)
freq_bins = pd.qcut(income_freq, 10, duplicates="drop")
rate = train.groupby(freq_bins, observed=True)["y"].mean()
axes[1].bar(range(len(rate)), rate.values)
axes[1].set_xticks(range(len(rate)))
axes[1].set_xticklabels([str(interval) for interval in rate.index], rotation=30)
axes[1].set_title("Purchase rate by income-value frequency")
plt.tight_layout()
plt.show()
精确的 income 值会重复多次，而重复的次数本身携带信息。
这就是为什么频率编码和对精确 income 值做 out-of-fold 目标编码在这场比赛中对 GBDT 有帮助。
要点总结
- 正如配方所说，Subsidy、range anxiety、income 和 environmental concern 承载了几乎全部信号。
- 仅凭这个配方就能得到约 0.938 AUC，因此它是一个很好的起始特征。
- 把 income 既当作数值又当作类别来处理（精确值、取整分桶），并且始终用 out-of-fold 方式进行编码。
感谢阅读！如果对你有帮助，欢迎点赞（upvote）。🙌
浙公网安备 33010602011771号