【机器学习实战】-- Titanic 数据集（8）-- Boosting提升 (XGBOOST)

1. 写在前面: 

本篇属于实战部分，更注重于算法在实际项目中的应用。如需对感知机算法本身有进一步的了解，可参考以下链接，在本人学习的过程中，起到了很大的帮助：

【1】XGBoost算法原理小结 https://www.cnblogs.com/pinard/p/10979808.html

【2】XGBoost Parameters https://xgboost.readthedocs.io/en/latest/parameter.html

【3】XGBoost Python API Reference https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn

 

2. 数据集：

数据集地址：https://www.kaggle.com/c/titanic

Titanic数据集是Kaggle上参与人数最多的项目之一。数据本身简单小巧，适合初学者上手，深入了解比较各个机器学习算法。

数据集包含11个变量：PassengerID、Pclass、Name、Sex、Age、SibSp、Parch、Ticket、Fare、Cabin、Embarked，通过这些数据来预测乘客在Titanic事故中是否幸存下来。

 

3. 算法简介：

XGBoost实际上是对GBDT的优化，GBDT利用了泰勒一阶展开，拟合负梯度；而XGBoost则对损失函数泰勒二阶展开，一次求解出决策树最优的所有J个叶子节点区域$R_{tj}$和每个叶子节点区域的最优解$c_{tj}$。

 

4. 实战：

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from sklearn.preprocessing import MinMaxScaler, StandardScaler, OneHotEncoder, OrdinalEncoder
from sklearn.impute import SimpleImputer
from sklearn.model_selection import StratifiedKFold, GridSearchCV, train_test_split, cross_validate
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier, RandomForestClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, auc
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.utils import class_weight

import xgboost as xgb


class DataFrameSelector(BaseEstimator, TransformerMixin):
    def __init__(self, attribute_name):
        self.attribute_name = attribute_name

    def fit(self, x, y=None):
        return self

    def transform(self, x):
        return x[self.attribute_name].values


# Load data
data = pd.read_csv('train.csv')

data_x = data.drop('Survived', axis=1)
data_y = data['Survived']

# Data cleaning
cat_attribs = ['Pclass', 'Sex', 'Embarked']
dis_attribs = ['SibSp', 'Parch']
con_attribs = ['Age', 'Fare']

# encoder: OneHotEncoder()、OrdinalEncoder()
cat_pipeline = Pipeline([
    ('selector', DataFrameSelector(cat_attribs)),
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder()),
])

dis_pipeline = Pipeline([
    ('selector', DataFrameSelector(dis_attribs)),
    ('scaler', MinMaxScaler()),
    ('imputer', SimpleImputer(strategy='most_frequent')),
])

con_pipeline = Pipeline([
    ('selector', DataFrameSelector(con_attribs)),
    ('scaler', MinMaxScaler()),
    ('imputer', SimpleImputer(strategy='mean')),
])

full_pipeline = FeatureUnion(
    transformer_list=[
        ('con_pipeline', con_pipeline),
        ('dis_pipeline', dis_pipeline),
        ('cat_pipeline', cat_pipeline),
    ]
)

data_x_cleaned = full_pipeline.fit_transform(data_x)

X_train, X_test, y_train, y_test = train_test_split(data_x_cleaned, data_y, stratify=data_y, test_size=0.25, random_state=1992)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=2)

# XGBoost
xgb_cla = xgb.XGBClassifier(use_label_encoder=False, verbosity=1, objective='binary:logistic', random_state=1992)
cls_wt = class_weight.compute_class_weight('balanced', np.unique(y_train), y_train)  # cls_wt[1]/cls_wt[0]

param_grid = [{
        'learning_rate': [0.5],
        'n_estimators': [5],
        'max_depth': [5],
        'min_child_weight': [5],
        'gamma': [5],
        'scale_pos_weight': [1],
        'subsample': [0.8],
                }]

grid_search = GridSearchCV(xgb_cla, param_grid=param_grid, cv=cv, scoring='accuracy', n_jobs=-1, return_train_score=True)
grid_search.fit(X_train, y_train)
cv_results_grid_search = pd.DataFrame(grid_search.cv_results_).sort_values(by='rank_test_score')

predicted_train_xgb = grid_search.predict(X_train)
predicted_test_xgb = grid_search.predict(X_test)

print('------------XGBoost grid_search: Results of train------------')
print(accuracy_score(y_train, predicted_train_xgb))
print(precision_score(y_train, predicted_train_xgb))
print(recall_score(y_train, predicted_train_xgb))

print('------------XGBoost grid search: Results of test------------')
print(accuracy_score(y_test, predicted_test_xgb))
print(precision_score(y_test, predicted_test_xgb))
print(recall_score(y_test, predicted_test_xgb))

 

4.1 xgboost中自定义eval_metric：

和 Adaboost 和 GBDT 一样，也想将每一步的分步强学习器在训练集以及验证集上的表现。由于xgboost并没有直接的staged_predict方法，这就需要用到参数 “eval_metric”，来指定 evaluation metric，其自带的有 “error”，在这里我也自己写了一个 “accuracy” 的eval metric, 正好可以和 "error" 进行对比。然而在这里我发现了一个问题：我们之前设定的 objective 是 “binary:logistic”，根据官方文档：binary:logistic: logistic regression for binary classification, output probability，然而实际上它只是在使用 predict 方法时，输出的是 probability，输入到 eval_metric 中的所谓的“y_predicted”，其实是 $f(x)$，也就是未经过 transformation 的值，因此在 eval_metric 中的定义需要小心。

 

def metric_precision(y_predicted, y_true):
    label = y_true.get_label()
    predicted_binary = [1 if y_cont > 0 else 0 for y_cont in y_predicted]

    print('max_y_predicted:%f' % max(y_predicted))
    print('min_y_predicted:%f' % min(y_predicted))

    precision = precision_score(label, predicted_binary)
    return 'Precision', precision


def metric_accuracy(y_predicted, y_true):
    label = y_true.get_label()
    # predicted_binary = np.round(y_predicted)
    # y_predicted = 1 / (1 + np.exp(-y_predicted))
    predicted_binary = [1 if y_cont > 0 else 0 for y_cont in y_predicted]

    print('max_y_predicted:%f' % max(y_predicted))
    print('min_y_predicted:%f' % min(y_predicted))

    accuracy = accuracy_score(label, predicted_binary)
    return 'Accuracy', accuracy


def metric_recall(y_predicted, y_true):
    label = y_true.get_label()
    predicted_binary = [1 if y_cont > 0 else 0 for y_cont in y_predicted]

    print('max_y_predicted:%f' % max(y_predicted))
    print('min_y_predicted:%f' % min(y_predicted))

    recall = recall_score(label, predicted_binary)
    return 'Recall', recall


xgb_cla2 = grid_search.best_estimator_
temp2 = xgb_cla2.predict(X_train)
temp2_pro = xgb_cla2.predict_proba(X_train)
xgb_cla2.fit(X_train, y_train, eval_set=[(X_train, y_train), (X_test, y_test)], eval_metric=['error'])

xgb_cla2.fit(X_train, y_train, eval_set=[(X_train, y_train), (X_test, y_test)], eval_metric=metric_accuracy)

plt.figure()
plt.plot(xgb_cla2.evals_result()['validation_1']['Accuracy'], label='test')
plt.plot(xgb_cla2.evals_result()['validation_0']['Accuracy'], label='train')

plt.legend()
plt.grid(axis='both')

在训练集上，经过grid_search, n_estiamtors(sklearn api) / num_boost_round(learning api) 为5最优。可以发现在测试集上，同样是在5时取得最优解，后续 test_accuracy 不再提升，而 train_accuracy 进一步提升，逐渐过拟合。

4.2 结果分析：

和前几篇一样，将 XGBoost 的最优参数用于预测集，并将结果上传kaggle，结果如下（注这里的训练集只是全体训练集，包括代码中的 train 和 test）：

可以看到其效果和AdaBoost、GBDT差不多，略差于RF(随机森林)。

	训练集 accuracy	训练集 precision	训练集 recall	预测集 accuracy（需上传kaggle获取结果）
朴素贝叶斯最优解	0.790	0.731	0.716	0.756
感知机	0.771	0.694	0.722	0.722
逻辑回归	0.807	0.781	0.690	0.768
线性SVM	0.801	0.772	0.684	0.773
rbf核SVM	0.834	0.817	0.731	0.785
AdaBoost	0.844	0.814	0.769	0.789
GBDT	0.843	0.877	0.687	0.778
RF	0.820	0.917	0.585	0.792
XGBoost	0.831	0.847	0.681	0.780