XGBoost

1、The principles of XGBooost:

1.1 The final model：

The final model forms used by XGBoost, GBDT, and AdaBoost are the same, where m represents the weak model obtained in the m-th iteration:

$$y_{x}=\sum_{m}^{M}\hat{f}_{m}(x)$$　

1.2 Target Function

$$L^{(m)}=\sum_{i}^{N}l(y_{i},\hat{y}_{i}^{m-1}+\hat{f}_{m}(x_{i}))+\Omega(f_{m})+C$$

Let T be the number of leaf nodes and $\omega$ be the weight of the leaf nodes. Then the regularization term is:

$$\Omega(f_{m})=\gamma T+\frac{1}{2}\lambda\sum_{j}^{T}\omega_{j}^{2}$$

1.3 Performing a second-order Taylor expansion on the objective function (loss):

second-order Taylor expansion：$$f(x+\Delta x)\approx f(x)+f{}'(x)\Delta x+\frac{1}{2}f{}''(x)\Delta x^{2}$$

Let：

$$g_{i}=\partial_{\hat{y}^{m-1}}l(y_{i},\hat{y}^{m-1})$$

$$h_{i}=\partial_{\hat{y}^{m-1}}^{2}l(y_{i},\hat{y}^{m-1})$$

So：$$L^{(m)}=\sum_{i}^{N}l(y_{i},\hat{y}_{i}^{m-1}+\hat{f}_{m}(x_{i}))+\Omega(f_{m})+C\\
\approx \sum_{i}^{N}[l(y_{i},\hat{y}_{i}^{m-1})+g_{i}\hat{f}_{m}(x_{i})+\frac{1}{2}h_{i}\hat{f}_{m}^{2}(x_{i})]+\Omega(f_{m})+C$$

 

Because $l(y_{i},\hat{y}_{i}^{m-1})$ is a constant term, therefore the objective function can also be expressed as: 

$$Obj^{m} \approx \sum_{i}^{N}[g_{i}\hat{f}_{m}(x_{i})+\frac{1}{2}h_{i}\hat{f}_{m}^{2}(x_{i})]+\Omega(f_{m})\\$$

$$\approx \sum_{i}^{N}[g_{i}\omega_{q(x_{i})}+\frac{1}{2}h_{i}\omega_{q(x_{i})}^{2}]+\gamma T+\frac{1}{2}\lambda\sum_{j}^{T}\omega_{j}^{2}\\
\approx \sum_{j}^{T}[\sum_{i\in I_{j}}^{}g_{i}\omega_{j}+\frac{1}{2}\sum_{i\in I_{j}}^{}h_{i}\omega_{j}^{2}]+\gamma T+\frac{1}{2}\lambda\sum_{j}^{T}\omega_{j}^{2}\\
\approx \sum_{j}^{T}[G_{j}\omega_{j}+\frac{1}{2}H_{j}\omega_{j}^{2}]+\gamma T+\frac{1}{2}\lambda\sum_{j}^{T}\omega_{j}^{2}\\
\approx \sum_{j}^{J}[G_{j}\omega_{j}+\frac{1}{2}(H_{j}+\lambda)\omega_{j}^{2})]+\gamma T $$

 

where：

$$G_{j}=\sum_{i\in I_{j}}^{}g_{i}$$

$$H_{j}=\sum_{i\in I_{j}}^{}h_{i}$$

1.4 Let partial dirivativee of $\omega_{j}$，be 0，the weight of the leaf node is obtained as:

$$\omega_{j}=-\frac{G_{i}}{H_{j}+\lambda}$$

1.5 Substituting the extremum point into the objective function yields:

$$Obj^{m}=-\frac{1}{2}\sum_{j}^{J}\frac{G_{j}^{2}}{H_{j}+\lambda}+\gamma T$$

1.6 Selection of split points (G and H without subscripts represent the values when no split is performed):

$$L_{split}=\frac{1}{2}[\frac{G_{L}^{2}}{H_{L}+\lambda}+\frac{G_{R}^{2}}{H_{R}+\lambda}-\frac{G^{2}}{H+\lambda}]$$