数据竞赛实战(3)——公共自行车使用量预测

前言

1,背景介绍

  公共自行车低碳,环保,健康,并且解决了交通中“最后一公里”的痛点,在全国各个城市越来越受欢迎。本次练习的数据取自于两个城市某街道上的几处公共自行车停车桩。我们希望根据时间,天气等信息,预测出该街区在一小时内的被借取的公共自行车的数量。

2,任务类型

  回归

3,数据文件说明

train.csv        训练集     文件大小为273KB

test.csv               预测集    文件大小为179KB

sample_submit.csv    提交示例      文件大小为97KB

4,数据变量说明 

  训练集中共有10000条样本,预测集中有7000条样本 

 

 

5,评估方法

  评价方法为RMSE(Root of Mean Squared Error)

 6,完整代码,请移步小编的GitHub

  传送门:请点击我

 

数据预处理

1,观察数据有没有缺失值

print(train.info())


<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 7 columns):
city          10000 non-null int64
hour          10000 non-null int64
is_workday    10000 non-null int64
weather       10000 non-null int64
temp_1        10000 non-null float64
temp_2        10000 non-null float64
wind          10000 non-null int64
dtypes: float64(2), int64(5)
memory usage: 547.0 KB
None

  我们可以看到,共有10000个观测值,没有缺失值。

2,观察每个变量的基础描述信息

print(train.describe())


               city          hour      ...             temp_2          wind
count  10000.000000  10000.000000      ...       10000.000000  10000.000000
mean       0.499800     11.527500      ...          15.321230      1.248600
std        0.500025      6.909777      ...          11.308986      1.095773
min        0.000000      0.000000      ...         -15.600000      0.000000
25%        0.000000      6.000000      ...           5.800000      0.000000
50%        0.000000     12.000000      ...          16.000000      1.000000
75%        1.000000     18.000000      ...          24.800000      2.000000
max        1.000000     23.000000      ...          46.800000      7.000000

[8 rows x 7 columns]

  通过观察可以得出一些猜测,如城市0 和城市1基本可以排除南方城市;整个观测记录时间跨度较长,还可能包含了一个长假期数据等等。

3,查看相关系数

  (为了方便查看,绝对值低于0.2的就用nan替代)

    corr = feature_data.corr()
    corr[np.abs(corr) < 0.2] = np.nan
    print(corr)


            city  hour  is_workday  weather    temp_1    temp_2  wind
city         1.0   NaN         NaN      NaN       NaN       NaN   NaN
hour         NaN   1.0         NaN      NaN       NaN       NaN   NaN
is_workday   NaN   NaN         1.0      NaN       NaN       NaN   NaN
weather      NaN   NaN         NaN      1.0       NaN       NaN   NaN
temp_1       NaN   NaN         NaN      NaN  1.000000  0.987357   NaN
temp_2       NaN   NaN         NaN      NaN  0.987357  1.000000   NaN
wind         NaN   NaN         NaN      NaN       NaN       NaN   1.0

  从相关性角度来看,用车的时间和当时的气温对借取数量y有较强的关系;气温和体感气温显强正相关(共线性),这个和常识一致。

模型训练及其结果展示

1,标杆模型:简单线性回归模型

  该模型预测结果的RMSE为:39.132

# -*- coding: utf-8 -*-

# 引入模块
from sklearn.linear_model import LinearRegression
import pandas as pd

# 读取数据
train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")
submit = pd.read_csv("sample_submit.csv")

# 删除id
train.drop('id', axis=1, inplace=True)
test.drop('id', axis=1, inplace=True)

# 取出训练集的y
y_train = train.pop('y')

# 建立线性回归模型
reg = LinearRegression()
reg.fit(train, y_train)
y_pred = reg.predict(test)

# 若预测值是负数,则取0
y_pred = map(lambda x: x if x >= 0 else 0, y_pred)

# 输出预测结果至my_LR_prediction.csv
submit['y'] = y_pred
submit.to_csv('my_LR_prediction.csv', index=False)

  

2,决策树回归模型

  该模型预测结果的RMSE为:28.818

# -*- coding: utf-8 -*-

# 引入模块
from sklearn.tree import DecisionTreeRegressor
import pandas as pd

# 读取数据
train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")
submit = pd.read_csv("sample_submit.csv")

# 删除id
train.drop('id', axis=1, inplace=True)
test.drop('id', axis=1, inplace=True)

# 取出训练集的y
y_train = train.pop('y')

# 建立最大深度为5的决策树回归模型
reg = DecisionTreeRegressor(max_depth=5)
reg.fit(train, y_train)
y_pred = reg.predict(test)

# 输出预测结果至my_DT_prediction.csv
submit['y'] = y_pred
submit.to_csv('my_DT_prediction.csv', index=False)

  

3,Xgboost回归模型

  该模型预测结果的RMSE为:18.947

# -*- coding: utf-8 -*-

# 引入模块
from xgboost import XGBRegressor
import pandas as pd

# 读取数据
train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")
submit = pd.read_csv("sample_submit.csv")

# 删除id
train.drop('id', axis=1, inplace=True)
test.drop('id', axis=1, inplace=True)

# 取出训练集的y
y_train = train.pop('y')

# 建立一个默认的xgboost回归模型
reg = XGBRegressor()
reg.fit(train, y_train)
y_pred = reg.predict(test)

# 输出预测结果至my_XGB_prediction.csv
submit['y'] = y_pred
submit.to_csv('my_XGB_prediction.csv', index=False)

  

 

4,Xgboost回归模型调参过程

  Xgboost的相关博客:请点击我

  参数调优的方法步骤一般情况如下:

  • 1,选择较高的学习速率(learning rate)。一般情况下,学习速率的值为0.1。但是对于不同的问题,理想的学习速率有时候会在0.05到0.3之间波动。选择对应于此学习速率的理想决策树数量。 Xgboost有一个很有用的函数“cv”,这个函数可以在每一次迭代中使用交叉验证,并返回理想的决策树数量。

  • 2,对于给定的学习速率和决策树数量,进行决策树特定参数调优(max_depth,min_child_weight,gamma,subsample,colsample_bytree)。在确定一棵树的过程中,我们可以选择不同的参数。

  • 3,Xgboost的正则化参数的调优。(lambda,alpha)。这些参数可以降低模型的复杂度,从而提高模型的表现。

  • 4,降低学习速率,确定理想参数。

 

5,Xgboost使用GridSearchCV调参过程

5.1,Xgboost 的默认参数如下(在sklearn库中的默认参数):

def __init__(self, max_depth=3, learning_rate=0.1, n_estimators=100,
                 silent=True, objective="rank:pairwise", booster='gbtree',
                 n_jobs=-1, nthread=None, gamma=0, min_child_weight=1, max_delta_step=0,
                 subsample=1, colsample_bytree=1, colsample_bylevel=1,
                 reg_alpha=0, reg_lambda=1, scale_pos_weight=1,
                 base_score=0.5, random_state=0, seed=None, missing=None, **kwargs):

  

5.2,首先调n_estimators

def xgboost_parameter_tuning(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    param_test1 = {
        'n_estimators': range(100, 1000, 100)
    }
    gsearch1 = GridSearchCV(estimator= xgb.XGBRegressor(
        learning_rate=0.1, max_depth=5,
        min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,
        nthread=4, scale_pos_weight=1, seed=27),
        param_grid=param_test1, iid=False, cv=5
    )


    gsearch1.fit(X_train, y_train)
    return gsearch1.best_params_, gsearch1.best_score_

  得到结果如下(所以我们选择树的个数为200):

{'n_estimators': 200}
0.9013685759002941

  

5.3,调参 max_depth和min_child_weight

  (树的最大深度,缺省值为3,范围是[1, 正无穷),树的深度越大,则对数据的拟合程度越高,但是通常取值为3-10)

  (孩子节点中的最小的样本权重和,如果一个叶子节点的样本权重和小于min_child_weight则拆分过程结果)

  下面我们对这两个参数调优,是因为他们对最终结果由很大的影响,所以我直接小范围微调。

def xgboost_parameter_tuning2(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)

    param_test2 = {
        'max_depth': range(3, 10, 1),
        'min_child_weight': range(1, 6, 1),
    }
    gsearch1 = GridSearchCV(estimator= xgb.XGBRegressor(
        learning_rate=0.1, n_estimators=200
    ), param_grid=param_test2, cv=5)


    gsearch1.fit(X_train, y_train)
    return gsearch1.best_params_, gsearch1.best_score_

  得到的结果如下:

{'max_depth': 5, 'min_child_weight': 5}
0.9030852081699604

  我们对于数值进行较大跨度的48种不同的排列组合,可以看出理想的max_depth值为5,理想的min_child_weight值为5。

5.4,gamma参数调优

  (gamma值使得算法更加conservation,且其值依赖于loss function,在模型中应该调参)

  在已经调整好其他参数的基础上,我们可以进行gamma参数的调优了。Gamma参数取值范围可以很大,我这里把取值范围设置为5,其实我们也可以取更精确的gamma值。

def xgboost_parameter_tuning3(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)

    param_test3 = {
        'gamma': [i/10.0 for i in range(0, 5)]
    }
    gsearch1 = GridSearchCV(estimator=xgb.XGBRegressor(
        learning_rate=0.1, n_estimators=200, max_depth=5, min_child_weight=5
    ), param_grid=param_test3, cv=5)


    gsearch1.fit(X_train, y_train)
    return gsearch1.best_params_, gsearch1.best_score_

  

结果如下:

{'gamma': 0.0}
0.9024876500236406

  

5.5,调整subsample 和 colsample_bytree 参数

  (subsample 用于训练模型的子样本占整个样本集合的比例,如果设置0.5则意味着XGBoost将随机的从整个样本集合中抽取出百分之50的子样本建立模型,这样能防止过拟合,取值范围为(0, 1])

  (在建立树的时候对特征采样的比例,缺省值为1,物质范围为(0, 1])

  下一步是尝试不同的subsample 和colsample_bytree 参数。我们分两个阶段来进行这个步骤。这两个步骤都取0.6,0.7,0.8,0.9 作为起始值。

def xgboost_parameter_tuning4(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)

    param_test4 = {
        'subsample': [i / 10.0 for i in range(6, 10)],
        'colsample_bytree': [i / 10.0 for i in range(6, 10)]
    }
    gsearch1 = GridSearchCV(estimator=xgb.XGBRegressor(
        learning_rate=0.1, n_estimators=200, max_depth=5, min_child_weight=5, gamma=0
    ), param_grid=param_test4, cv=5)


    gsearch1.fit(X_train, y_train)
    return gsearch1.best_params_, gsearch1.best_score_

  

  结果如下:

{'colsample_bytree': 0.9, 'subsample': 0.8}
0.9039011907271065

  

5.6,正则化参数调优

  由于gamma函数提供了一种更加有效的降低过拟合的方法,大部分人很少会用到这个参数,但是我们可以尝试用一下这个参数。

def xgboost_parameter_tuning5(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)

    param_test5 = {
        'reg_alpha': [0, 0.001, 0.005, 0.01, 0.05]
    }
    gsearch1 = GridSearchCV(estimator=xgb.XGBRegressor(
        learning_rate=0.1, n_estimators=200, max_depth=5, min_child_weight=5, gamma=0.0,
        colsample_bytree=0.9, subsample=0.8), param_grid=param_test5, cv=5)


    gsearch1.fit(X_train, y_train)
    return gsearch1.best_params_, gsearch1.best_score_

  结果如下:

{'reg_alpha': 0.01}
0.899800819611995

5.6,汇总出我们搜索到的最佳参数,然后训练

  代码如下:

def xgboost_train(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    params = {
        'learning_rate': 0.1,
        'n_estimators': 200,
        'max_depth': 5,
        'min_child_weight': 5,
        'gamma': 0.0,
        'colsample_bytree': 0.9,
        'subsample': 0.8,
        'reg_alpha': 0.01,

    }
    model = xgb.XGBRegressor(**params)
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)

    submit = pd.read_csv(submitfile)
    submit['y'] = model.predict(test_feature)
    submit.to_csv('my_xgboost_prediction1.csv', index=False)

  

  我们可以对比上面的结果,最终的结果为15.208,比直接使用xgboost提高了3.92.

 

最终所有代码总结如下:

#_*_coding:utf-8_*_
import numpy as np
import pandas as pd


def load_data(trainfile, testfile):
    traindata = pd.read_csv(trainfile)
    testdata = pd.read_csv(testfile)
    print(traindata.shape)   #(10000, 9)
    print(testdata.shape)    #(7000, 8)
    # print(traindata)
    print(type(traindata))
    feature_data = traindata.iloc[:, 1:-1]
    label_data = traindata.iloc[:, -1]
    test_feature = testdata.iloc[:, 1:]
    return feature_data, label_data, test_feature

def xgboost_train(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    params = {
        'learning_rate': 0.1,
        'n_estimators': 200,
        'max_depth': 5,
        'min_child_weight': 5,
        'gamma': 0.0,
        'colsample_bytree': 0.9,
        'subsample': 0.8,
        'reg_alpha': 0.01,

    }
    model = xgb.XGBRegressor()
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)

    submit = pd.read_csv(submitfile)
    submit['y'] = model.predict(test_feature)
    submit.to_csv('my_xgboost_prediction.csv', index=False)


def xgboost_parameter_tuning1(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)

    param_test1 = {
        'n_estimators': range(100, 1000, 100)
    }
    gsearch1 = GridSearchCV(estimator= xgb.XGBRegressor(
        learning_rate=0.1, max_depth=5,
        min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,
        nthread=4, scale_pos_weight=1, seed=27),
        param_grid=param_test1, iid=False, cv=5
    )


    gsearch1.fit(X_train, y_train)
    return gsearch1.best_params_, gsearch1.best_score_

def xgboost_parameter_tuning2(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)

    param_test2 = {
        'max_depth': range(3, 10, 1),
        'min_child_weight': range(1, 6, 1),
    }
    gsearch1 = GridSearchCV(estimator= xgb.XGBRegressor(
        learning_rate=0.1, n_estimators=200
    ), param_grid=param_test2, cv=5)


    gsearch1.fit(X_train, y_train)
    return gsearch1.best_params_, gsearch1.best_score_

def xgboost_parameter_tuning3(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)

    param_test3 = {
        'gamma': [i/10.0 for i in range(0, 5)]
    }
    gsearch1 = GridSearchCV(estimator=xgb.XGBRegressor(
        learning_rate=0.1, n_estimators=200, max_depth=5, min_child_weight=5
    ), param_grid=param_test3, cv=5)


    gsearch1.fit(X_train, y_train)
    return gsearch1.best_params_, gsearch1.best_score_

def xgboost_parameter_tuning4(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)

    param_test4 = {
        'subsample': [i / 10.0 for i in range(6, 10)],
        'colsample_bytree': [i / 10.0 for i in range(6, 10)]
    }
    gsearch1 = GridSearchCV(estimator=xgb.XGBRegressor(
        learning_rate=0.1, n_estimators=200, max_depth=5, min_child_weight=5,gamma=0.0
    ), param_grid=param_test4, cv=5)


    gsearch1.fit(X_train, y_train)
    return gsearch1.best_params_, gsearch1.best_score_

def xgboost_parameter_tuning5(feature_data, label_data, test_feature, submitfile):
    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)

    param_test5 = {
        'reg_alpha': [0, 0.001, 0.005, 0.01, 0.05]
    }
    gsearch1 = GridSearchCV(estimator=xgb.XGBRegressor(
        learning_rate=0.1, n_estimators=200, max_depth=5, min_child_weight=5, gamma=0.0,
        colsample_bytree=0.9, subsample=0.8), param_grid=param_test5, cv=5)


    gsearch1.fit(X_train, y_train)
    return gsearch1.best_params_, gsearch1.best_score_

if __name__ == '__main__':
    trainfile = 'data/train.csv'
    testfile = 'data/test.csv'
    submitfile = 'data/sample_submit.csv'
    feature_data, label_data, test_feature = load_data(trainfile, testfile)
    xgboost_train(feature_data, label_data, test_feature, submitfile)

 

6,随机森林回归模型 

  该模型预测结果的RMSE为:18.947

#_*_coding:utf-8_*_
import numpy as np
import pandas as pd


def load_data(trainfile, testfile):
    traindata = pd.read_csv(trainfile)
    testdata = pd.read_csv(testfile)
    feature_data = traindata.iloc[:, 1:-1]
    label_data = traindata.iloc[:, -1]
    test_feature = testdata.iloc[:, 1:]
    return feature_data, label_data, test_feature

def random_forest_train(feature_data, label_data, test_feature, submitfile):
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)

    model = RandomForestRegressor()
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)

    submit = pd.read_csv(submitfile)
    submit['y'] = model.predict(test_feature)
    submit.to_csv('my_random_forest_prediction.csv', index=False)


if __name__ == '__main__':
    trainfile = 'data/train.csv'
    testfile = 'data/test.csv'
    submitfile = 'data/sample_submit.csv'
    feature_data, label_data, test_feature = load_data(trainfile, testfile)
    random_forest_train(feature_data, label_data, test_feature, submitfile)

  

7,随机森林回归模型调参过程

  随机森林的相关博客:请点击我

  首先,我们看一下随机森林的调参过程

 

  • 1,首先先调即不会增加模型复杂度,又对模型影响最大的参数n_estimators(学习曲线)
  • 2,找到最佳值后,调max_depth(单个网格搜索,也可以使用学习曲线)
  •   (一般根据数据的大小来进行一个探视,当数据集很小的时候,可以采用1~10,或者1~20这样的试探,但是对于大型数据来说骂我们应该尝试30~50层深度(或许更深))
  • 3,接下来依次对各个参数进行调参
  •   (注意,对于大型数据集,max_leaf_nodes可以尝试从1000来构建,先输入1000,每100个叶子一个区间,再逐渐缩小范围;对于min_samples_split和min_samples_leaf,一般从他们的最小值开始向上增加10 或者20,面对高维度高样本数据,如果不放心可以直接50+,对于大型数据可能需要200~300的范围,如果调整的时候发现准确率无论如何都上不来,可以大胆放心的调试一个很大的数据,大力限制模型的复杂度。)

 7.1  使用gridsearchcv探索n_estimators的最佳值

def random_forest_parameter_tuning1(feature_data, label_data, test_feature):
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    param_test1 = {
        'n_estimators': range(10, 71, 10)
    }
    model = GridSearchCV(estimator=RandomForestRegressor(
        min_samples_split=100, min_samples_leaf=20, max_depth=8, max_features='sqrt',
        random_state=10), param_grid=param_test1, cv=5
    )
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)
    return model.best_score_, model.best_params_

  结果如下:

{'n_estimators': 70}
0.6573670183811001

  这样我们得到了最佳的弱学习器迭代次数,为70.。

7.2  对决策树最大深度 max_depth 和内部节点再划分所需要的最小样本数求最佳值

   我们首先得到了最佳弱学习器迭代次数,接着我们对决策树最大深度max_depth和内部节点再划分所需要最小样本数min_samples_split进行网格搜索。

def random_forest_parameter_tuning2(feature_data, label_data, test_feature):
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    param_test2 = {
        'max_depth': range(3, 14, 2),
        'min_samples_split': range(50, 201, 20)
    }
    model = GridSearchCV(estimator=RandomForestRegressor(
        n_estimators=70, min_samples_leaf=20, max_features='sqrt', oob_score=True,
        random_state=10), param_grid=param_test2, cv=5
    )
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)
    return model.best_score_, model.best_params_

  结果为:

{'max_depth': 13, 'min_samples_split': 50}
0.7107311632187736

  

  对于内部节点再划分所需要最小样本数min_samples_split,我们暂时不能一起定下来,因为这个还和决策树其他的参数存在关联。

7.3  求内部节点再划分所需要的最小样本数min_samples_split和叶子节点最小样本数min_samples_leaf的最佳参数

  下面我们对内部节点在划分所需要最小样本数min_samples_split和叶子节点最小样本数min_samples_leaf一起调参。

def random_forest_parameter_tuning3(feature_data, label_data, test_feature):
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    param_test3 = {
        'min_samples_split': range(10, 90, 20),
        'min_samples_leaf': range(10, 60, 10),
    }
    model = GridSearchCV(estimator=RandomForestRegressor(
        n_estimators=70, max_depth=13, max_features='sqrt', oob_score=True,
        random_state=10), param_grid=param_test3, cv=5
    )
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)
    return model.best_score_, model.best_params_

  结果如下:

{'min_samples_leaf': 10, 'min_samples_split': 10}
0.7648492269870218

  

7.4  求最大特征数max_features的最佳参数

def random_forest_parameter_tuning4(feature_data, label_data, test_feature):
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    param_test3 = {
        'max_features': range(3, 9, 2),
    }
    model = GridSearchCV(estimator=RandomForestRegressor(
        n_estimators=70, max_depth=13, min_samples_split=10, min_samples_leaf=10, oob_score=True,
        random_state=10), param_grid=param_test3, cv=5
    )
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)
    return model.best_score_, model.best_params_

  结果如下:

{'max_features': 7}
0.881211719251515

  

7.5  汇总出我们搜索到的最佳参数,然后训练

def random_forest_train(feature_data, label_data, test_feature, submitfile):
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    params = {
        'n_estimators': 70,
        'max_depth': 13,
        'min_samples_split': 10,
        'min_samples_leaf': 10,
        'max_features': 7
    }
    model = RandomForestRegressor(**params)
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)

    submit = pd.read_csv(submitfile)
    submit['y'] = model.predict(test_feature)
    submit.to_csv('my_random_forest_prediction1.csv', index=False)

  最终计算得到的结果如下:

  我们发现,经过调参,结果由17.144 优化到16.251,效果相对Xgboost来说,不是很大。所以最终我们选择Xgboost算法。

7.6  所有代码如下:

#_*_coding:utf-8_*_
import numpy as np
import pandas as pd


def load_data(trainfile, testfile):
    traindata = pd.read_csv(trainfile)
    testdata = pd.read_csv(testfile)
    feature_data = traindata.iloc[:, 1:-1]
    label_data = traindata.iloc[:, -1]
    test_feature = testdata.iloc[:, 1:]
    return feature_data, label_data, test_feature

def random_forest_train(feature_data, label_data, test_feature, submitfile):
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    params = {
        'n_estimators': 70,
        'max_depth': 13,
        'min_samples_split': 10,
        'min_samples_leaf': 10,
        'max_features': 7
    }
    model = RandomForestRegressor(**params)
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)

    submit = pd.read_csv(submitfile)
    submit['y'] = model.predict(test_feature)
    submit.to_csv('my_random_forest_prediction1.csv', index=False)

def random_forest_parameter_tuning1(feature_data, label_data, test_feature):
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    param_test1 = {
        'n_estimators': range(10, 71, 10)
    }
    model = GridSearchCV(estimator=RandomForestRegressor(
        min_samples_split=100, min_samples_leaf=20, max_depth=8, max_features='sqrt',
        random_state=10), param_grid=param_test1, cv=5
    )
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)
    return model.best_score_, model.best_params_

def random_forest_parameter_tuning2(feature_data, label_data, test_feature):
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    param_test2 = {
        'max_depth': range(3, 14, 2),
        'min_samples_split': range(50, 201, 20)
    }
    model = GridSearchCV(estimator=RandomForestRegressor(
        n_estimators=70, min_samples_leaf=20, max_features='sqrt', oob_score=True,
        random_state=10), param_grid=param_test2, cv=5
    )
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)
    return model.best_score_, model.best_params_

def random_forest_parameter_tuning3(feature_data, label_data, test_feature):
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    param_test3 = {
        'min_samples_split': range(10, 90, 20),
        'min_samples_leaf': range(10, 60, 10),
    }
    model = GridSearchCV(estimator=RandomForestRegressor(
        n_estimators=70, max_depth=13, max_features='sqrt', oob_score=True,
        random_state=10), param_grid=param_test3, cv=5
    )
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)
    return model.best_score_, model.best_params_

def random_forest_parameter_tuning4(feature_data, label_data, test_feature):
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    from sklearn.model_selection import GridSearchCV

    X_train, X_test, y_train, y_test = train_test_split(feature_data, label_data, test_size=0.23)
    param_test4 = {
        'max_features': range(3, 9, 2)
    }
    model = GridSearchCV(estimator=RandomForestRegressor(
        n_estimators=70, max_depth=13, min_samples_split=10, min_samples_leaf=10, oob_score=True,
        random_state=10), param_grid=param_test4, cv=5
    )
    model.fit(X_train, y_train)
    # 对测试集进行预测
    y_pred = model.predict(X_test)
    # 计算准确率
    MSE = mean_squared_error(y_test, y_pred)
    RMSE = np.sqrt(MSE)
    print(RMSE)
    return model.best_score_, model.best_params_


if __name__ == '__main__':
    trainfile = 'data/train.csv'
    testfile = 'data/test.csv'
    submitfile = 'data/sample_submit.csv'
    feature_data, label_data, test_feature = load_data(trainfile, testfile)
    random_forest_train(feature_data, label_data, test_feature, submitfile)

  

参考文献:https://www.jianshu.com/p/748b6c35773d

posted @ 2019-04-29 19:46  战争热诚  阅读(3883)  评论(2编辑  收藏  举报