Sklearn-源码解析-书-v1-0-十二-

Sklearn 源码解析（书）v1.0（十二）

├── Predictor.predict

├── _compute_gradients_hessians

sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py

├── BaseHistGradientBoosting.init

├── BaseHistGradientBoosting.fit

├── BaseHistGradientBoosting._fit_stages

├── BaseHistGradientBoosting.predict

├── BaseHistGradientBoosting.staged_predict

├── BaseHistGradientBoosting._raw_predict

├── BaseHistGradientBoosting._predict_stages

├── HistGradientBoostingClassifier.init

├── HistGradientBoostingRegressor.init

sklearn/ensemble/_stacking.py

├── StackingClassifier.init

├── StackingClassifier.fit

├── StackingClassifier.predict

├── StackingClassifier.predict_proba

├── StackingRegressor.init

├── StackingRegressor.fit

├── StackingRegressor.predict

├── _fit_base_estimators

├── _get_cv_predictions

sklearn/ensemble/_voting.py

├── VotingClassifier.init

├── VotingClassifier.fit

├── VotingClassifier.predict

├── VotingClassifier.predict_proba

├── VotingRegressor.init

├── VotingRegressor.fit

├── VotingRegressor.predict

sklearn/ensemble/_weight_boosting.py

├── AdaBoostClassifier.init

├── AdaBoostClassifier.fit

├── AdaBoostClassifier.predict

├── AdaBoostClassifier.predict_proba

├── AdaBoostRegressor.init

├── AdaBoostRegressor.fit

├── AdaBoostRegressor.predict

├── _samme_proba

sklearn/ensemble/_iforest.py

├── IsolationForest.init

├── IsolationForest.fit

├── IsolationForest.decision_function

├── IsolationForest.predict

├── IsolationForest._compute_chunked_score_samples

├── IsolationForest._average_path_length

├── _parallel_decision_function

sklearn/ensemble/init.py

├── all

25.4 Bagging 与随机森林 —— 自助采样下的“并行智慧”

Bagging（Bootstrap Aggregating）是集成学习中最简单却极其强大的并行范式。其核心思想是：通过自助采样生成多个训练子集，并行训练同类型的基学习器，最后通过投票或平均聚合预测结果。scikit-learn 在 sklearn/ensemble/_bagging.py 中实现了通用的 BaseBagging 基类，并在 sklearn/ensemble/_forest.py 中针对决策树构建了专门的 BaseForest 体系，进一步衍生出随机森林与极端随机树。

25.4.1 类型定义详解

25.4.1.1 BaseBagging 基类

class BaseBagging(BaseEnsemble, metaclass=ABCMeta):
    """Base class for Bagging meta-estimator."""
    _parameter_constraints: dict = {
        "estimator": [HasMethods(["fit", "predict"]), None],
        "n_estimators": [Interval(Integral, 1, None, closed="left")],
        "max_samples": [
            None,
            Interval(Integral, 1, None, closed="left"),
            Interval(RealNotInt, 0, 1, closed="right"),
        ],
        "max_features": [
            Interval(Integral, 1, None, closed="left"),
            Interval(RealNotInt, 0, 1, closed="right"),
        ],
        "bootstrap": ["boolean"],
        "bootstrap_features": ["boolean"],
        "oob_score": ["boolean"],
        "warm_start": ["boolean"],
        "n_jobs": [None, Integral],
        "random_state": ["random_state"],
        "verbose": ["verbose"],
    }

这个参数约束字典清晰地描述了 Bagging 的配置空间：estimator 指定基学习器（默认决策树），n_estimators 控制集成规模，max_samples 与 max_features 分别控制样本与特征的子采样比例，bootstrap 与 bootstrap_features 决定是否有放回采样，oob_score 开启袋外评估，warm_start 支持增量训练。

25.4.1.2 BaseForest 森林基类

class BaseForest(MultiOutputMixin, BaseEnsemble, metaclass=ABCMeta):
    _parameter_constraints: dict = {
        "n_estimators": [Interval(Integral, 1, None, closed="left")],
        "bootstrap": ["boolean"],
        "oob_score": ["boolean", callable],
        "n_jobs": [Integral, None],
        "random_state": ["random_state"],
        "verbose": ["verbose"],
        "warm_start": ["boolean"],
        "max_samples": [
            None,
            Interval(RealNotInt, 0.0, None, closed="neither"),
            Interval(Integral, 1, None, closed="left"),
        ],
    }

BaseForest 继承自 BaseEnsemble，专门为树集成定制：固定基学习器为决策树，暴露 criterion、max_depth 等树特有参数，并增加了 apply、decision_path 等树结构分析方法。

25.4.2 逐行解析关键函数

25.4.2.1 BaseBagging.fit：自助采样与并行训练主流程

源码路径：sklearn/ensemble/_bagging.py - BaseBagging.fit()（第 200-350 行）

def fit(self, X, y, sample_weight=None, **fit_params):
    # ① 输入验证与元数据路由
    _raise_for_params(fit_params, self, "fit")
    X, y = validate_data(self, X, y, accept_sparse=["csr", "csc"], dtype=None,
                         ensure_all_finite=False, multi_output=True)
    if sample_weight is not None:
        sample_weight = _check_sample_weight(sample_weight, X, dtype=None)
        if not self.bootstrap:
            warn(f"sample_weight 推荐配合 bootstrap=True 使用")
    return self._fit(X, y, max_samples=self.max_samples, sample_weight=sample_weight, **fit_params)

这段代码完成了输入校验、样本权重检查，并委托给 _fit 执行核心训练逻辑。

def _fit(self, X, y, max_samples=None, max_depth=None, check_input=True,
         sample_weight=None, **fit_params):
    random_state = check_random_state(self.random_state)
    n_samples = X.shape[0]
    self._n_samples = n_samples
    y = self._validate_y(y)
    self._validate_estimator(self._get_estimator())
    # 元数据路由处理
    if _routing_enabled():
        routed_params = process_routing(self, "fit", **fit_params)
    else:
        routed_params = Bunch()
        routed_params.estimator = Bunch(fit=fit_params)
    # 验证 max_samples 与 max_features
    if max_samples is None:
        max_samples = self.max_samples
    max_samples = _get_n_samples_bootstrap(X.shape[0], max_samples, sample_weight)
    if not self.bootstrap and max_samples > X.shape[0]:
        raise ValueError("无放回采样时 max_samples 不能超过 n_samples")
    self._max_samples = max_samples
    # max_features 验验证
    if isinstance(self.max_features, numbers.Integral):
        max_features = self.max_features
    elif isinstance(self.max_features, float):
        max_features = int(self.max_features * self.n_features_in_)
    if max_features > self.n_features_in_:
        raise ValueError("max_features 必须 <= n_features")
    max_features = max(1, int(max_features))
    self._max_features = max_features
    # 存储 sample_weight 供后续 OOB 使用
    self._sample_weight = sample_weight
    # OOB 与 warm_start 冲突检查
    if not self.bootstrap and self.oob_score:
        raise ValueError("OOB 估计仅在 bootstrap=True 时可用")
    if self.warm_start and self.oob_score:
        raise ValueError("OOB 估计仅在 warm_start=False 时可用")
    if hasattr(self, "oob_score_") and self.warm_start:
        del self.oob_score_
    # warm_start 增量训练
    if not self.warm_start or not hasattr(self, "estimators_"):
        self.estimators_ = []
        self.estimators_features_ = []
    n_more_estimators = self.n_estimators - len(self.estimators_)
    if n_more_estimators < 0:
        raise ValueError("warm_start 时 n_estimators 必须 >= 现有估计器数量")
    elif n_more_estimators == 0:
        warn("Warm-start 未增加 n_estimators，不训练新树")
        return self
    # 并行构建估计器
    n_jobs, n_estimators, starts = _partition_estimators(n_more_estimators, self.n_jobs)
    total_n_estimators = sum(n_estimators)
    if self.warm_start and len(self.estimators_) > 0:
        random_state.randint(MAX_INT, size=len(self.estimators_))
    seeds = random_state.randint(MAX_INT, size=n_more_estimators)
    self._seeds = seeds
    all_results = Parallel(n_jobs=n_jobs, verbose=self.verbose, **self._parallel_args())(
        delayed(_parallel_build_estimators)(
            n_estimators[i], self, X, y, sample_weight,
            seeds[starts[i]:starts[i+1]], total_n_estimators,
            verbose=self.verbose, check_input=check_input,
            fit_params=routed_params.estimator.fit,
        ) for i in range(n_jobs)
    )
    # 归约结果
    self.estimators_ += list(itertools.chain.from_iterable(t[0] for t in all_results))
    self.estimators_features_ += list(itertools.chain.from_iterable(t[1] for t in all_results))
    if self.oob_score:
        self._set_oob_score(X, y)
    return self

核心流程总结：

参数验证：校验 max_samples、max_features，处理 sample_weight 与 bootstrap 的兼容性。
随机种子管理：为每个基学习器生成独立随机种子，支持 warm_start 的随机状态衔接。
并行训练：使用 joblib.Parallel 分发 _parallel_build_estimators 任务，每个任务构建一批估计器。
结果归约：收集各进程返回的估计器与特征索引，追加到 estimators_ 与 estimators_features_。
OOB 评估：若启用 oob_score，调用 _set_oob_score 计算袋外分数。

这段代码实现了 Bagging 的完整训练流水线：自助采样索引生成、基学习器并行拟合、OOB 评估集成。

25.4.2.2 _parallel_build_estimators：单批次估计器构建工作函数

源码路径：sklearn/ensemble/_bagging.py - _parallel_build_estimators()（第 350-450 行）

def _parallel_build_estimators(n_estimators, ensemble, X, y, sample_weight,
                               seeds, total_n_estimators, verbose, check_input, fit_params):
    n_samples, n_features = X.shape
    max_features = ensemble._max_features
    max_samples = ensemble._max_samples
    bootstrap = ensemble.bootstrap
    bootstrap_features = ensemble.bootstrap_features
    has_check_input = has_fit_parameter(ensemble.estimator_, "check_input")
    requires_feature_indexing = bootstrap_features or max_features != n_features
    consumes_sample_weight = _consumes_sample_weight(ensemble.estimator_)
    estimators = []
    estimators_features = []
    for i in range(n_estimators):
        if verbose > 1:
            print(f"Building estimator {i+1} of {n_estimators} (total {total_n_estimators})...")
        random_state = seeds[i]
        estimator = ensemble._make_estimator(append=False, random_state=random_state)
        if has_check_input:
            estimator_fit = partial(estimator.fit, check_input=check_input)
        else:
            estimator_fit = estimator.fit
        # 生成特征与样本采样索引
        features, indices = _generate_bagging_indices(
            random_state, bootstrap_features, bootstrap,
            n_features, n_samples, max_features, max_samples, sample_weight
        )
        fit_params_ = fit_params.copy()
        # 两种行采样方式：样本权重法 vs 索引法
        if consumes_sample_weight:
            # ① 样本权重法：通过 bincount 将索引转为权重，更节省内存
            indices_as_sample_weight = np.bincount(indices, minlength=n_samples)
            fit_params_["sample_weight"] = indices_as_sample_weight
            X_ = X[:, features] if requires_feature_indexing else X
            estimator_fit(X_, y, **fit_params_)
        else:
            # ② 索引法：直接切片数据
            y_ = _safe_indexing(y, indices)
            X_ = _safe_indexing(X, indices)
            fit_params_ = _check_method_params(X, params=fit_params_, indices=indices)
            if requires_feature_indexing:
                X_ = X_[:, features]
            estimator_fit(X_, y_, **fit_params_)
        estimators.append(estimator)
        estimators_features.append(features)
    return estimators, estimators_features

关键设计点：

双采样策略：若基学习器支持 sample_weight，使用权重法避免数据复制（内存高效）；否则退回索引切片法。
特征子采样：bootstrap_features 控制特征是否有放回采样，max_features 限制特征数量。
随机隔离：每个估计器使用独立随机种子，保证采样独立性。

25.4.2.3 BaseBagging._set_oob_score：袋外评估计算

源码路径：sklearn/ensemble/_bagging.py - BaseBagging._set_oob_score()（第 450-550 行，以 BaggingClassifier 为例）

def _set_oob_score(self, X, y):
    n_samples = y.shape[0]
    n_classes_ = self.n_classes_
    predictions = np.zeros((n_samples, n_classes_))
    for estimator, samples, features in zip(
        self.estimators_, self.estimators_samples_, self.estimators_features_
    ):
        mask = ~indices_to_mask(samples, n_samples)  # OOB 掩码
        if hasattr(estimator, "predict_proba"):
            predictions[mask, :] += estimator.predict_proba((X[mask, :])[:, features])
        else:
            p = estimator.predict((X[mask, :])[:, features])
            j = 0
            for i in range(n_samples):
                if mask[i]:
                    predictions[i, p[j]] += 1
                    j += 1
    if (predictions.sum(axis=1) == 0).any():
        warn("部分样本无 OOB 预测，可能估计器过少")
    oob_decision_function = predictions / predictions.sum(axis=1)[:, np.newaxis]
    oob_score = accuracy_score(y, np.argmax(predictions, axis=1))
    self.oob_decision_function_ = oob_decision_function
    self.oob_score_ = oob_score

原理：每棵树约有 36.8% 的样本未被采样（OOB 样本），聚合这些树对各自 OOB 样本的预测，得到无偏的泛化误差估计。分类任务累加概率或投票，回归任务（BaggingRegressor._set_oob_score）累加预测值取平均，最后计算准确率或 R² 分数。

25.4.2.4 BaseForest._parallel_build_trees：森林并行构建树

源码路径：sklearn/ensemble/_forest.py - _parallel_build_trees()（第 300-400 行）

def _parallel_build_trees(tree, bootstrap, X, y, sample_weight, tree_idx, n_trees,
                          verbose=0, class_weight=None, n_samples_bootstrap=None,
                          missing_values_in_feature_mask=None):
    if verbose > 1:
        print(f"building tree {tree_idx+1} of {n_trees}")
    if bootstrap:
        n_samples = X.shape[0]
        indices = _generate_sample_indices(random_state, n_samples, n_samples_bootstrap, sample_weight)
        sample_weight_tree = np.bincount(indices, minlength=n_samples)
        if class_weight == "balanced_subsample":
            expanded_class_weight = compute_sample_weight("balanced", y, indices=indices)
            sample_weight_tree = sample_weight_tree * expanded_class_weight
        tree._fit(X, y, sample_weight=sample_weight_tree, check_input=False,
                  missing_values_in_feature_mask=missing_values_in_feature_mask)
    else:
        tree._fit(X, y, sample_weight=sample_weight, check_input=False,
                  missing_values_in_feature_mask=missing_values_in_feature_mask)
    return tree

与 Bagging 的区别：

直接操作 DecisionTree 的 _fit 方法，传递 class_weight 与 missing_values_in_feature_mask。
使用 np.bincount 将 Bootstrap 索引转为样本权重，避免数据复制。
class_weight="balanced_subsample" 在每棵树的 Bootstrap 样本上单独计算类别权重。

25.4.2.5 ForestClassifier.predict_proba：概率平均聚合

源码路径：sklearn/ensemble/_forest.py - ForestClassifier.predict_proba()（第 480-550 行）

def predict_proba(self, X):
    check_is_fitted(self)
    X = self._validate_X_predict(X)
    n_jobs, _, _ = _partition_estimators(self.n_estimators, self.n_jobs)
    all_proba = [
        np.zeros((X.shape[0], j), dtype=np.float64) for j in np.atleast_1d(self.n_classes_)
    ]
    lock = threading.Lock()
    Parallel(n_jobs=n_jobs, verbose=self.verbose, require="sharedmem")(
        delayed(_accumulate_prediction)(e.predict_proba, X, all_proba, lock) for e in self.estimators_
    )
    for proba in all_proba:
        proba /= len(self.estimators_)
    if len(all_proba) == 1:
        return all_proba[0]
    else:
        return all_proba

并行聚合机制：使用线程锁保护共享数组 all_proba，每个线程累加一棵树的 predict_proba 结果，最后取平均。这种共享内存并行模式利用了决策树预测释放 GIL 的特性，避免了进程间通信开销。

25.4.3 RandomForest 与 ExtraTrees 的差异化配置

| 特性 | RandomForest | ExtraTrees |

|------|--------------|------------|

| 基学习器 | DecisionTreeClassifier/Regressor | ExtraTreeClassifier/Regressor |

| bootstrap | True (默认) | False (默认，全量样本) |

| splitter | 'best' (最佳分裂) | 'random' (随机阈值) |

| max_features | "sqrt" (分类) / 1.0 (回归) | "sqrt" (分类) / 1.0 (回归) |

| 随机性来源 | 样本 Bootstrap + 特征子采样 | 样本全量 + 特征子采样 + 随机分裂阈值 |

源码对比：

# 第 25 章 —— RandomForestClassifier.__init__ (sklearn/ensemble/_forest.py 第 550-650 行)
def __init__(self, n_estimators=100, *, criterion="gini", max_depth=None,
             min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0,
             max_features="sqrt", max_leaf_nodes=None, min_impurity_decrease=0.0,
             bootstrap=True, oob_score=False, n_jobs=None, random_state=None,
             verbose=0, warm_start=False, class_weight=None, ccp_alpha=0.0,
             max_samples=None, monotonic_cst=None):
    super().__init__(estimator=DecisionTreeClassifier(), n_estimators=n_estimators,
        estimator_params=("criterion", "max_depth", ..., "monotonic_cst"),
        bootstrap=bootstrap, oob_score=oob_score, n_jobs=n_jobs, random_state=random_state,
        verbose=verbose, warm_start=warm_start, class_weight=class_weight, max_samples=max_samples)
    self.criterion = criterion
    self.max_depth = max_depth
    ...

# 第 25 章 —— ExtraTreesClassifier.__init__ (sklearn/ensemble/_forest.py 第 650-750 行)
def __init__(self, n_estimators=100, *, criterion="gini", max_depth=None,
             min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0,
             max_features="sqrt", max_leaf_nodes=None, min_impurity_decrease=0.0,
             bootstrap=False, oob_score=False, n_jobs=None, random_state=None,
             verbose=0, warm_start=False, class_weight=None, ccp_alpha=0.0,
             max_samples=None, monotonic_cst=None):
    super().__init__(estimator=ExtraTreeClassifier(), n_estimators=n_estimators,
        estimator_params=("criterion", "max_depth", ..., "monotonic_cst"),
        bootstrap=bootstrap, oob_score=oob_score, n_jobs=n_jobs, random_state=random_state,
        verbose=verbose, warm_start=warm_start, class_weight=class_weight, max_samples=max_samples)
    self.criterion = criterion
    ...

设计权衡：RandomForest 通过 Bootstrap + 最佳分裂获得低偏差、中等方差；ExtraTrees 通过全量样本 + 随机分裂获得更高偏差、更低方差，训练更快（无需寻找最佳分裂点），适合高维稀疏数据。

25.4.4 数据流图

graph TD A[fit(X, y)] --> B[_fit: 参数验证] B --> C[生成随机种子 seeds] C --> D[_partition_estimators 划分任务] D --> E[Parallel(_parallel_build_estimators)] E --> F1[估计器 1...k] E --> F2[估计器 k+1...2k] E --> Fn[估计器 ...] F1 --> G[收集 estimators_ & estimators_features_] F2 --> G Fn --> G G --> H{oob_score?} H -->|Yes| I[_set_oob_score: 计算 OOB 预测与分数] H -->|No| J[返回 self] I --> J

25.5 梯度提升决策树 (GBDT) —— 串行纠错的“助推引擎”

传统 GBDT（GradientBoostingClassifier/Regressor）在 sklearn/ensemble/_gb.py 与 sklearn/ensemble/_gradient_boosting.pyx 中实现，采用分阶段加法建模：每轮拟合一棵回归树去拟合当前模型的负梯度（伪残差），再通过线搜索确定最优步长更新预测值。

25.5.1 类型定义详解

25.5.1.1 BaseGradientBoosting 基类

class BaseGradientBoosting(BaseEnsemble, metaclass=ABCMeta):
    _parameter_constraints: dict = {
        **DecisionTreeRegressor._parameter_constraints,
        "learning_rate": [Interval(Real, 0.0, None, closed="left")],
        "n_estimators": [Interval(Integral, 1, None, closed="left")],
        "criterion": [StrOptions({"squared_error"}), Hidden(StrOptions({"deprecated", "friedman_mse"}))],
        "subsample": [Interval(Real, 0.0, 1.0, closed="right")],
        "verbose": ["verbose"],
        "warm_start": ["boolean"],
        "validation_fraction": [Interval(Real, 0.0, 1.0, closed="neither")],
        "n_iter_no_change": [Interval(Integral, 1, None, closed="left"), None],
        "tol": [Interval(Real, 0.0, None, closed="left")],
    }

关键参数：learning_rate 缩放每棵树的贡献，subsample 引入随机性（随机梯度提升），validation_fraction 与 n_iter_no_change 支持早停。

25.5.1.2 损失函数抽象

_LOSSES = {
    "squared_error": HalfSquaredError,
    "absolute_error": AbsoluteError,
    "huber": HuberLoss,
    "quantile": PinballLoss,
    "log_loss": HalfBinomialLoss,  # 分类
    "exponential": ExponentialLoss,  # 分类
}

损失类需实现 gradient、loss、update_terminal_regions、fit_intercept_only 等接口，支持二阶优化。

25.5.2 逐行解析关键函数

25.5.2.1 BaseGradientBoosting.fit：分阶段拟合主循环

源码路径：sklearn/ensemble/_gb.py - BaseGradientBoosting.fit()（第 500-800 行）

def fit(self, X, y, sample_weight=None, monitor=None):
    if not self.warm_start:
        self._clear_state()
    # 校验 criterion 废弃参数
    if self.criterion != "deprecated":
        warnings.warn("criterion 参数已废弃", FutureWarning)
    # 数据验证
    X, y = validate_data(self, X, y, accept_sparse=["csr", "csc", "coo"], dtype=DTYPE, multi_output=True)
    sample_weight_is_none = sample_weight is None
    sample_weight = _check_sample_weight(sample_weight, X)
    y = self._encode_y(y=y, sample_weight=sample_weight if not sample_weight_is_none else None)
    self._set_max_features()
    self._loss = self._get_loss(sample_weight=sample_weight)
    # 早停数据分割
    if self.n_iter_no_change is not None:
        stratify = y if is_classifier(self) else None
        X_train, X_val, y_train, y_val, sw_train, sw_val = train_test_split(
            X, y, sample_weight, random_state=self.random_state,
            test_size=self.validation_fraction, stratify=stratify)
    else:
        X_train, y_train, sw_train = X, y, sample_weight
        X_val = y_val = sw_val = None
    n_samples = X_train.shape[0]
    # 首次 fit 或 warm_start
    if not self._is_fitted():
        self._init_state()
        if self.init_ == "zero":
            raw_predictions = np.zeros((n_samples, self.n_trees_per_iteration_), dtype=np.float64)
        else:
            if sample_weight_is_none:
                self.init_.fit(X_train, y_train)
            else:
                self.init_.fit(X_train, y_train, sample_weight=sw_train)
            raw_predictions = _init_raw_predictions(X_train, self.init_, self._loss, is_classifier(self))
        begin_at_stage = 0
        self._rng = check_random_state(self.random_state)
    else:
        if self.n_estimators < self.estimators_.shape[0]:
            raise ValueError("warm_start 时 n_estimators 不能减少")
        begin_at_stage = self.estimators_.shape[0]
        X_train = check_array(X_train, dtype=DTYPE, order="C", accept_sparse="csr", ensure_all_finite=False)
        raw_predictions = self._raw_predict(X_train)
        self._resize_state()
    # 迭代拟合阶段
    n_stages = self._fit_stages(X_train, y_train, raw_predictions, sw_train, self._rng,
                                X_val, y_val, sw_val, begin_at_stage, monitor)
    # 裁剪数组
    if n_stages != self.estimators_.shape[0]:
        self.estimators_ = self.estimators_[:n_stages]
        self.train_score_ = self.train_score_[:n_stages]
        if hasattr(self, "oob_improvement_"):
            self.oob_improvement_ = self.oob_improvement_[:n_stages]
            self.oob_scores_ = self.oob_scores_[:n_stages]
            self.oob_score_ = self.oob_scores_[-1]
    self.n_estimators_ = n_stages
    return self

核心流程：

状态初始化/恢复：_init_state 分配 estimators_、train_score_ 等数组；warm_start 时 _resize_state 扩容。
初始预测：_init_raw_predictions 使用 init 估计器（默认 DummyClassifier/Regressor）生成初始值，通过链接函数映射到原始预测空间。
阶段迭代：委托 _fit_stages 执行提升循环。

25.5.2.2 _fit_stages：提升迭代核心

源码路径：sklearn/ensemble/_gb.py - BaseGradientBoosting._fit_stages()（第 800-1100 行）

def _fit_stages(self, X, y, raw_predictions, sample_weight, random_state,
                X_val, y_val, sample_weight_val, begin_at_stage=0, monitor=None):
    n_samples = X.shape[0]
    do_oob = self.subsample < 1.0
    sample_mask = np.ones((n_samples,), dtype=bool)
    n_inbag = max(1, int(self.subsample * n_samples))
    # 详细输出
    if self.verbose:
        verbose_reporter = VerboseReporter(verbose=self.verbose)
        verbose_reporter.init(self, begin_at_stage)
    X_csc = csc_matrix(X) if issparse(X) else None
    X_csr = csr_matrix(X) if issparse(X) else None
    # 早停损失历史
    if self.n_iter_no_change is not None:
        loss_history = np.full(self.n_iter_no_change, np.inf)
        y_val_pred_iter = self._staged_raw_predict(X_val, check_input=False)
    # 兼容旧版损失因子
    if isinstance(self._loss, (HalfSquaredError, HalfBinomialLoss)):
        factor = 2
    else:
        factor = 1
    # 提升循环
    for i in range(begin_at_stage, self.n_estimeters):
        # 子采样
        if do_oob:
            sample_mask = _random_sample_mask(n_samples, n_inbag, random_state)
            y_oob_masked = y[~sample_mask]
            sw_oob_masked = sample_weight[~sample_mask]
            if i == 0:
                initial_loss = factor * self._loss(y_true=y_oob_masked, raw_prediction=raw_predictions[~sample_mask], sample_weight=sw_oob_masked)
        # 拟合单阶段
        raw_predictions = self._fit_stage(i, X, y, raw_predictions, sample_weight, sample_mask, random_state, X_csc, X_csr)
        # 记录损失
        if do_oob:
            self.train_score_[i] = factor * self._loss(y_true=y[sample_mask], raw_prediction=raw_predictions[sample_mask], sample_weight=sample_weight[sample_mask])
            self.oob_scores_[i] = factor * self._loss(y_true=y_oob_masked, raw_prediction=raw_predictions[~sample_mask], sample_weight=sw_oob_masked)
            prev_loss = initial_loss if i == 0 else self.oob_scores_[i-1]
            self.oob_improvement_[i] = prev_loss - self.oob_scores_[i]
            self.oob_score_ = self.oob_scores_[-1]
        else:
            self.train_score_[i] = factor * self._loss(y_true=y, raw_prediction=raw_predictions, sample_weight=sample_weight)
        if self.verbose > 0:
            verbose_reporter.update(i, self)
        if monitor is not None and monitor(i, self, locals()):
            break
        # 早停检查
        if self.n_iter_no_change is not None:
            val_loss = factor * self._loss(y_val, next(y_val_pred_iter), sample_weight_val)
            if np.any(val_loss + self.tol < loss_history):
                loss_history[i % len(loss_history)] = val_loss
            else:
                break
    return i + 1

关键机制：

随机子采样：subsample < 1 时每轮随机抽取 n_inbag 样本，实现随机梯度提升，sample_mask 标记 in-bag 样本。
OOB 评估：利用 out-of-bag 样本计算 oob_scores_ 与 oob_improvement_，无需额外验证集。
早停：n_iter_no_change 监控验证损失（或训练损失），连续多轮无显著改善则停止。
稀疏矩阵优化：预转换 CSC/CSR 格式供树构建与预测使用。

25.5.2.3 _fit_stage：单阶段多树拟合

def _fit_stage(self, i, X, y, raw_predictions, sample_weight, sample_mask, random_state, X_csc=None, X_csr=None):
    original_y = y
    if isinstance(self._loss, HuberLoss):
        set_huber_delta(self._loss, y_true=y, raw_prediction=raw_predictions, sample_weight=sample_weight)
    # 计算负梯度
    neg_gradient = -self._loss.gradient(y_true=y, raw_prediction=raw_predictions, sample_weight=None)
    if neg_gradient.ndim == 1:
        neg_g_view = neg_gradient.reshape((-1, 1))
    else:
        neg_g_view = neg_gradient
    for k in range(self.n_trees_per_iteration_):
        if self._loss.is_multiclass:
            y = np.array(original_y == k, dtype=np.float64)  # OvR 二值化
        # 构建回归树拟合负梯度
        tree = DecisionTreeRegressor(criterion="squared_error", splitter="best", max_depth=self.max_depth,
            min_samples_split=self.min_samples_split, min_samples_leaf=self.min_samples_leaf,
            min_weight_fraction_leaf=self.min_weight_fraction_leaf, min_impurity_decrease=self.min_impurity_decrease,
            max_features=self.max_features, max_leaf_nodes=self.max_leaf_nodes, random_state=random_state,
            ccp_alpha=self.ccp_alpha)
        if self.subsample < 1.0:
            sample_weight = sample_weight * sample_mask.astype(np.float64)
        X_fit = X_csc if X_csc is not None else X
        tree.fit(X_fit, neg_g_view[:, k], sample_weight=sample_weight, check_input=False)
        # 更新叶值（线搜索）
        X_update = X_csr if X_csr is not None else X
        _update_terminal_regions(self._loss, tree.tree_, X_update, y, neg_g_view[:, k], raw_predictions,
                                 sample_weight, sample_mask, learning_rate=self.learning_rate, k=k)
        self.estimators_[i, k] = tree
    return raw_predictions

要点：

负梯度计算：调用损失函数的 gradient 方法，多输出/多分类时形状为 (n_samples, K)。
多分类处理：HalfMultinomialLoss 时 n_trees_per_iteration_ = n_classes，每轮构建 K 棵树，针对每个类别拟合负梯度（OvR 策略）。
叶值更新：_update_terminal_regions 执行线搜索，针对不同损失函数有专门实现。

25.5.2.4 Cython 优化预测器：Predictor 类

源码路径：sklearn/ensemble/_gradient_boosting.pyx - Predictor 与 update_terminal_regions（第 200-650 行）

cdef class Predictor:
    cdef public int n_classes
    cdef public int n_trees
    cdef public double learning_rate
    cdef public double init_predictor
    cdef public Tree *trees
    cdef public object loss
    cdef __cinit__(self, int n_classes, int n_trees, double learning_rate, double init_predictor):
        self.n_classes = n_classes
        self.n_trees = n_trees
        self.learning_rate = learning_rate
        self.init_predictor = init_predictor
        self.trees = <Tree *> malloc(n_trees * sizeof(Tree))

    cpdef predict(self, double[:, :] X, double[:, :] out) nogil:
        """单样本遍历树，累加预测值"""
        cdef int i, j, k
        cdef Tree *tree
        cdef Node *node
        for i in range(X.shape[0]):
            for k in range(self.n_trees):
                out[i, k] = self.init_predictor
            for j in range(self.n_trees):
                tree = self.trees + j
                node = tree.nodes
                while node.left_child != TREE_LEAF:
                    if X[i, node.feature] <= node.threshold:
                        node = tree.nodes + node.left_child
                    else:
                        node = tree.nodes + node.right_child
                out[i, j % self.n_classes] += self.learning_rate * node.value[0, 0]

    cpdef update_terminal_regions(self, Tree *tree, double[:, :] X, double[:] y,
                                  double[:] raw_prediction, double learning_rate, int k) nogil:
        """遍历叶节点，计算最优常数值更新"""
        cdef int n_samples = X.shape[0]
        cdef int leaf
        cdef double sum_grad, sum_hess, update
        # 计算每个样本落入的叶节点
        cdef int[:] terminal_regions = np.zeros(n_samples, dtype=int)
        for i in range(n_samples):
            node = tree.nodes
            while node.left_child != TREE_LEAF:
                if X[i, node.feature] <= node.threshold:
                    node = tree.nodes + node.left_child
                else:
                    node = tree.nodes + node.right_child
            terminal_regions[i] = node - tree.nodes
        # 按叶节点聚合梯度/海森
        for leaf in range(tree.node_count):
            if tree.nodes[leaf].left_child == TREE_LEAF:
                sum_grad = 0.0
                sum_hess = 0.0
                count = 0
                for i in range(n_samples):
                    if terminal_regions[i] == leaf:
                        sum_grad += ...  # 根据损失类型计算
                        sum_hess += ...
                        count += 1
                if count > 0:
                    update = -sum_grad / (sum_hess + 1e-15)  # 牛顿步
                    tree.nodes[leaf].value[0, 0] = update * learning_rate

Cython 加速点：

无 GIL 并行：nogil 释放全局解释器锁，利用多核并行预测。
内存视图：double[:, :] X 直接访问 NumPy 数组缓冲区，避免 Python 对象开销。
指针运算：Tree *trees、Node *node 直接操作 C 结构体，树遍历极快。
原地更新：update_terminal_regions 直接修改树节点的 value 字段，无需 Python 层拷贝。

25.5.2.5 _update_terminal_regions_regression/classification：叶值更新逻辑

源码路径：sklearn/ensemble/_gb.py - _update_terminal_regions()（第 450-650 行，Python 实现）

def _update_terminal_regions(loss, tree, X, y, neg_gradient, raw_prediction,
                             sample_weight, sample_mask, learning_rate=0.1, k=0):
    terminal_regions = tree.apply(X)
    if not isinstance(loss, HalfSquaredError):
        masked_terminal_regions = terminal_regions.copy()
        masked_terminal_regions[~sample_mask] = -1
        if isinstance(loss, HalfBinomialLoss):
            def compute_update(y_, indices, neg_gradient, raw_prediction, k):
                neg_g = neg_gradient.take(indices, axis=0)
                prob = y_ - neg_g
                numerator = np.average(neg_g, weights=sw)
                denominator = np.average(prob * (1 - prob), weights=sw)
                return _safe_divide(numerator, denominator)
        elif isinstance(loss, HalfMultinomialLoss):
            def compute_update(...):
                # 多分类 softmax 牛顿步
                numerator = np.average(neg_g, weights=sw) * (K - 1) / K
                denominator = np.average(prob * (1 - prob), weights=sw)
                return _safe_divide(numerator, denominator)
        elif isinstance(loss, ExponentialLoss):
            def compute_update(...):
                # AdaBoost 指数损失
                hessian = neg_g.copy()
                hessian[y_ == 0] *= -1
                return _safe_divide(np.average(neg_g, weights=sw), np.average(hessian, weights=sw))
        else:
            def compute_update(...):
                return loss.fit_intercept_only(y_true=y_ - raw_prediction[indices, k], sample_weight=sw)
        # 更新每个叶节点
        for leaf in np.nonzero(tree.children_left == TREE_LEAF)[0]:
            indices = np.nonzero(masked_terminal_regions == leaf)[0]
            y_ = y.take(indices, axis=0)
            sw = None if sample_weight is None else sample_weight[indices]
            update = compute_update(y_, indices, neg_gradient, raw_prediction, k)
            tree.value[leaf, 0, 0] = update
    # 更新原始预测
    raw_prediction[:, k] += learning_rate * tree.value[:, 0, 0].take(terminal_regions, axis=0)

数学原理：对于平方损失，叶值即为负梯度均值（无需线搜索）；对于对数损失/指数损失，使用一步牛顿-拉夫逊迭代近似最优叶值：

\[w_{leaf} = \frac{\sum_i g_i}{\sum_i h_i} \]

其中 \(g_i\) 为负梯度，\(h_i\) 为海森矩阵对角元。

25.5.3 数据流图

graph TD A[fit(X, y)] --> B[_init_state: 分配数组] B --> C[_init_raw_predictions: 初始模型] C --> D[_fit_stages 循环] D --> E[subsample < 1? 生成 sample_mask] E --> F[_loss.gradient: 计算负梯度] F --> G[for k in n_trees_per_iteration] G --> H[DecisionTreeRegressor.fit 负梯度] H --> I[_update_terminal_regions: 叶值线搜] I --> J[raw_predictions += learning_rate * tree.predict] J --> K[记录 train_score_/oob_score_] K --> L{早停条件?} L -->|否| D L -->|是| M[裁剪数组, 返回 self]

25.6 直方图梯度提升：分箱与树生长 —— 高效 GBDT 的“现代引擎”

HistGradientBoosting（HGBT）在 sklearn/ensemble/_hist_gradient_boosting/ 目录下实现，通过特征分箱、直方图加速分裂寻优、最佳优先树生长等工程优化，在大规模数据上比传统 GBDT 快一个数量级，且原生支持缺失值与类别特征。

25.6.1 类型定义详解

25.6.1.1 _BinMapper：分箱映射器

class _BinMapper(TransformerMixin, BaseEstimator):
    def __init__(self, n_bins=256, subsample=int(2e5), is_categorical=None,
                 known_categories=None, random_state=None, n_threads=None):
        self.n_bins = n_bins
        self.subsample = subsample
        self.is_categorical = is_categorical
        self.known_categories = known_categories
        self.random_state = random_state
        self.n_threads = n_threads

核心属性：

bin_thresholds_：每个特征的分箱阈值（连续特征）或类别映射（类别特征）。
n_bins_non_missing_：每个特征实际使用的非缺失值分箱数。
missing_values_bin_idx_ = n_bins - 1：缺失值专用分箱索引。

25.6.1.2 TreeGrower：树生长控制器

class TreeGrower:
    def __init__(self, X_binned, gradients, hessians, max_leaf_nodes=None, max_depth=None,
                 min_samples_leaf=20, min_gain_to_split=0.0, min_hessian_to_split=1e-3,
                 n_bins=256, n_bins_non_missing=None, has_missing_values=False,
                 is_categorical=None, monotonic_cst=None, interaction_cst=None,
                 l2_regularization=0.0, feature_fraction_per_split=1.0,
                 rng=np.random.default_rng(), shrinkage=1.0, n_threads=None):
        # 组件组合
        self.histogram_builder = HistogramBuilder(X_binned, n_bins, gradients, hessians,
                                                  hessians_are_constant, n_threads)
        self.splitter = Splitter(X_binned, n_bins_non_missing, missing_values_bin_idx,
                                 has_missing_values, is_categorical, monotonic_cst,
                                 l2_regularization, min_hessian_to_split,
                                 min_samples_leaf, min_gain_to_split,
                                 hessians_are_constant, feature_fraction_per_split,
                                 rng, n_threads)
        # 状态
        self.root = None
        self.splittable_nodes = []  # 优先队列
        self.finalized_leaves = []

设计模式：组合模式——TreeGrower 组合 HistogramBuilder（统计构建）与 Splitter（分裂寻优），自身负责树生长策略（最佳优先、深度/叶数约束）。

25.6.1.3 Splitter 与 HistogramBuilder：分裂寻优核心

# 第 25 章 —— histogram.pyx
cdef class HistogramBuilder:
    cdef build_histograms_brute(self, sample_indices, allowed_features):
        # O(n_samples) 扫描构建直方图
    cdef build_histograms_subtraction(self, parent_histograms, sibling_histograms, allowed_features):
        # O(n_bins) 减法技巧：hist(child) = hist(parent) - hist(sibling)

# 第 25 章 —— splitting.pyx
cdef class Splitter:
    cdef find_node_split(self, n_samples, histograms, sum_gradients, sum_hessians,
                         value, lower_bound, upper_bound, allowed_features):
        # 遍历特征与分箱，计算增益，返回最佳 SplitInfo

25.6.2 逐行解析关键函数

25.6.2.1 _BinMapper.fit：分箱阈值计算

源码路径：sklearn/ensemble/_hist_gradient_boosting/binning.py - _BinMapper.fit()（第 100-200 行）

def fit(self, X, y=None):
    if not (3 <= self.n_bins <= 256):
        raise ValueError("n_bins 必须在 [3, 256] 范围内")
    X = check_array(X, dtype=[X_DTYPE], ensure_all_finite=False)
    max_bins = self.n_bins - 1
    rng = check_random_state(self.random_state)
    if self.subsample is not None and X.shape[0] > self.subsample:
        subset = rng.choice(X.shape[0], self.subsample, replace=False)
        X = X.take(subset, axis=0)
    # 处理分类特征标记
    if self.is_categorical is None:
        self.is_categorical_ = np.zeros(X.shape[1], dtype=np.uint8)
    else:
        self.is_categorical_ = np.asarray(self.is_categorical, dtype=np.uint8)
    # 并行计算连续特征分箱阈值
    non_cat_thresholds = Parallel(n_jobs=self.n_threads, backend="threading")(
        delayed(_find_binning_thresholds)(X[:, f_idx], max_bins)
        for f_idx in range(n_features) if not self.is_categorical_[f_idx]
    )
    # 组装阈值
    for f_idx in range(n_features):
        if self.is_categorical_[f_idx]:
            thresholds = known_categories[f_idx]  # 类别特征直接用已知类别
            n_bins_non_missing[f_idx] = thresholds.shape[0]
            self.bin_thresholds_[f_idx] = thresholds
        else:
            self.bin_thresholds_[f_idx] = non_cat_thresholds[non_cat_idx]
            n_bins_non_missing[f_idx] = self.bin_thresholds_[f_idx].shape[0] + 1
            non_cat_idx += 1
    self.n_bins_non_missing_ = np.array(n_bins_non_missing, dtype=np.uint32)
    self.missing_values_bin_idx_ = self.n_bins - 1
    return self

分箱策略：

连续特征：若唯一值数 ≤ max_bins，用中点分箱；否则按分位数分箱（np.percentile），保证每箱样本数近似均衡。
类别特征：直接使用 known_categories 中的编码值作为分箱边界，无需计算分位数。
缺失值专用箱：最后一个分箱（索引 n_bins-1）始终保留给缺失值。

25.6.2.2 _find_binning_thresholds：单特征分位数分箱

def _find_binning_thresholds(col_data, max_bins):
    missing_mask = np.isnan(col_data)
    if missing_mask.any():
        col_data = col_data[~missing_mask]
    col_data = np.sort(col_data)
    distinct_values = np.unique(col_data).astype(X_DTYPE)
    if len(distinct_values) <= max_bins:
        midpoints = (distinct_values[:-1] + distinct_values[1:]) * 0.5
    else:
        percentiles = np.linspace(0, 100, num=max_bins + 1)[1:-1]
        midpoints = np.percentile(col_data, percentiles, method="midpoint").astype(X_DTYPE)
    np.clip(midpoints, a_min=None, a_max=ALMOST_INF, out=midpoints)
    return midpoints

25.6.2.3 TreeGrower.grow：最佳优先树生长

源码路径：sklearn/ensemble/_hist_gradient_boosting/grower.py - TreeGrower.grow()（第 300-500 行）

def grow(self):
    while self.splittable_nodes:
        self.split_next()
    self._apply_shrinkage()

def split_next(self):
    node = heappop(self.splittable_nodes)  # 取增益最大的节点
    # 分裂样本索引
    sample_indices_left, sample_indices_right, right_child_pos = \
        self.splitter.split_indices(node.split_info, node.sample_indices)
    # 创建子节点
    left_child_node = TreeNode(depth=depth, sample_indices=sample_indices_left,
                               partition_start=node.partition_start,
                               partition_stop=node.partition_start + right_child_pos,
                               sum_gradients=node.split_info.sum_gradient_left,
                               sum_hessians=node.split_info.sum_hessian_left,
                               value=node.split_info.value_left)
    right_child_node = TreeNode(...)
    # 单调约束传递
    if self.with_monotonic_cst:
        mid = (left_child_node.value + right_child_node.value) / 2
        if monotonic_cst == POS:
            left.set_children_bounds(node.children_lower_bound, mid)
            right.set_children_bounds(mid, node.children_upper_bound)
        elif NEG: ...
    # 计算子节点直方图（减法技巧）
    if should_split_left or should_split_right:
        smallest_child, largest_child = (left, right) if n_left < n_right else (right, left)
        smallest_child.histograms = self.histogram_builder.compute_histograms_brute(...)
        largest_child.histograms = self.histogram_builder.compute_histograms_subtraction(...)
        # 推入优先队列
        if should_split_left: self._compute_best_split_and_push(left_child_node)
        if should_split_right: self._compute_best_split_and_push(right_child_node)
    self.n_nodes += 2
    return left_child_node, right_child_node

最佳优先策略：使用堆（heapq）维护可分裂节点，每次分裂增益最大的节点，而非逐层生长。这使得在叶节点数受限（max_leaf_nodes=31）时，树结构更优。

25.6.2.4 Splitter.find_node_split：分裂寻优核心

源码路径：sklearn/ensemble/_hist_gradient_boosting/splitting.pyx - Splitter.find_node_split()（第 200-400 行）

cdef void _find_best_bin_to_split_left_to_right(Splitter self, feature_idx, has_missing_values,
        histograms, n_samples, sum_gradients, sum_hessians, value, monotonic_cst,
        lower_bound, upper_bound, split_info_struct *split_info) nogil:
    # 左到右扫描：缺失值分入右子节点
    for bin_idx in range(end):
        hist = histograms[feature_idx, bin_idx]
        n_samples_left += hist.count
        n_samples_right = n_samples - n_samples_left
        if self.hessians_are_constant:
            sum_hessian_left += hist.count
        else:
            sum_hessian_left += hist.sum_hessians
        sum_hessian_right = sum_hessians - sum_hessian_left
        sum_gradient_left += hist.sum_gradients
        sum_gradient_right = sum_gradients - sum_gradient_left
        # 约束检查
        if n_samples_left < self.min_samples_leaf: continue
        if sum_hessian_left < self.min_hessian_to_split: continue
        # 增益计算
        gain = _split_gain(sum_gradient_left, sum_hessian_left, sum_gradient_right, sum_hessian_right,
                           loss_current_node, monotonic_cst, lower_bound, upper_bound,
                           self.l2_regularization)
        if gain > best_gain and gain > self.min_gain_to_split:
            best_gain = gain
            best_bin_idx = bin_idx
            best_sum_gradient_left = sum_gradient_left
            ...
    if found_better_split:
        split_info.gain = best_gain
        split_info.bin_idx = best_bin_idx
        split_info.missing_go_to_left = False  # 缺失值去右
        ...

双向扫描处理缺失值：

左→右扫描：缺失值默认分入右子节点。
右→左扫描（仅当特征有缺失值时）：缺失值分入左子节点。
取增益最大的方向，实现缺失值自动分流（类似 XGBoost/LightGBM）。

类别特征分裂：使用位集枚举子集分裂（_bitset.pyx），对有序类别值按梯度/海森比值排序后扫描，将类别分裂转化为有序分裂问题。

25.6.2.5 HistogramBuilder.compute_histograms_brute/subtraction：直方图构建

源码路径：sklearn/ensemble/_hist_gradient_boosting/histogram.pyx（第 100-250 行）

cpdef void _build_histogram(const int feature_idx,
        const unsigned int [::1] sample_indices,
        const X_BINNED_DTYPE_C [::1] binned_feature,
        const G_H_DTYPE_C [::1] ordered_gradients,
        const G_H_DTYPE_C [::1] ordered_hessians,
        hist_struct [:, ::1] out) nogil:
    # 循环展开优化缓存命中
    for i in range(0, unrolled_upper, 4):
        bin_0 = binned_feature[sample_indices[i]]
        bin_1 = binned_feature[sample_indices[i + 1]]
        bin_2 = binned_feature[sample_indices[i + 2]]
        bin_3 = binned_feature[sample_indices[i + 3]]
        out[feature_idx, bin_0].sum_gradients += ordered_gradients[i]
        out[feature_idx, bin_1].sum_gradients += ordered_gradients[i + 1]
        out[feature_idx, bin_2].sum_gradients += ordered_gradients[i + 2]
        out[feature_idx, bin_3].sum_gradients += ordered_gradients[i + 3]
        # 同理累加 hessians 和 count
        ...

减法技巧：

cpdef void _subtract_histograms(const int feature_idx, unsigned int n_bins,
        hist_struct [:, ::1] hist_a, hist_struct [:, ::1] hist_b) nogil:
    for i in range(n_bins):
        hist_a[feature_idx, i].sum_gradients -= hist_b[feature_idx, i].sum_gradients
        hist_a[feature_idx, i].sum_hessians -= hist_b[feature_idx, i].sum_hessians
        hist_a[feature_idx, i].count -= hist_b[feature_idx, i].count

利用 hist(parent) = hist(left) + hist(right)，只需对样本少的子节点暴力构建直方图，另一个子节点通过减法获得，复杂度从 O(n_samples) 降为 O(n_bins)。

25.6.3 数据流图

graph TD A[BaseHistGradientBoosting.fit] --> B[_BinMapper.fit: 计算分箱阈值] B --> C[_BinMapper.transform: 训练数据分箱] C --> D[_fit_stages 迭代] D --> E[_loss.gradient/hessian: 计算梯度海森] E --> F[TreeGrower.grow] F --> G[Splitter.find_node_split: 直方图分裂寻优] G --> H[HistogramBuilder: 构建/减法直方图] H --> I[创建子节点, 入堆] I --> J{叶数/深度达标?} J -->|否| G J -->|是| K[make_predictor: 生成 TreePredictor] K --> L[_update_raw_predictions: 更新原始预测] L --> M[早停检查] M -->|继续| D M -->|停止| N[返回 self]

25.7 直方图梯度提升：预测与梯度更新 —— 高效 GBDT 的“核心循环”

25.7.1 TreePredictor：高效预测器结构

源码路径：sklearn/ensemble/_hist_gradient_boosting/predictor.py 与 _predictor.pyx

class TreePredictor:
    def __init__(self, nodes, binned_left_cat_bitsets, raw_left_cat_bitsets):
        self.nodes = nodes  # 扁平数组: feature_idx, bin_threshold, left, right, value, is_leaf, ...
        self.binned_left_cat_bitsets = binned_left_cat_bitsets
        self.raw_left_cat_bitsets = raw_left_cat_bitsets

    def predict_binned(self, X, missing_values_bin_idx, n_threads):
        out = np.empty(X.shape[0], dtype=Y_DTYPE)
        _predict_from_binned_data(self.nodes, X, self.binned_left_cat_bitsets,
                                  missing_values_bin_idx, n_threads, out)
        return out

Cython 预测内核 (_predictor.pyx)：

cdef inline Y_DTYPE_C _predict_one_from_binned_data(
        node_struct [:] nodes,
        const X_BINNED_DTYPE_C [:, :] binned_data,
        const BITSET_INNER_DTYPE_C [:, :] binned_left_cat_bitsets,
        const int row, const uint8_t missing_values_bin_idx) nogil:
    cdef node_struct node = nodes[0]
    cdef unsigned int node_idx = 0
    cdef X_BINNED_DTYPE_C data_val
    while True:
        if node.is_leaf:
            return node.value
        data_val = binned_data[row, node.feature_idx]
        if data_val == missing_values_bin_idx:
            node_idx = node.left if node.missing_go_to_left else node.right
        elif node.is_categorical:
            if in_bitset_2d_memoryview(binned_left_cat_bitsets, data_val, node.bitset_idx):
                node_idx = node.left
            else:
                node_idx = node.right
        else:
            node_idx = node.left if data_val <= node.bin_threshold else node.right
        node = nodes[node_idx]

设计亮点：

扁平数组存储：树节点压缩为结构体数组，消除指针追踪开销，缓存友好。
分箱索引直接比较：预测时直接比较 uint8 分箱索引，避免浮点运算。
位集加速类别分裂：in_bitset_2d_memoryview 单指令判断类别是否属于左子集。
并行预测：prange 并行处理样本，nogil 释放 GIL。

25.7.2 BaseHistGradientBoosting._fit_stages：训练主控流程

源码路径：sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py - _fit_stages()（第 500-700 行）

def _fit_stages(self, X_binned_train, y_train, raw_predictions, sample_weight_train,
                rng, X_binned_val, y_val, sample_weight_val, begin_at_stage, monitor):
    n_samples = X_binned_train.shape[0]
    gradient, hessian = self._loss.init_gradient_and_hessian(n_samples=n_samples, dtype=G_H_DTYPE, order="F")
    for iteration in range(begin_at_stage, self.max_iter):
        # 更新梯度/海森
        if self._loss.constant_hessian:
            self._loss.gradient(y_train, raw_predictions, sample_weight_train, gradient_out=gradient, n_threads=n_threads)
        else:
            self._loss.gradient_hessian(y_train, raw_predictions, sample_weight_train,
                                        gradient_out=gradient, hessian_out=hessian, n_threads=n_threads)
        # 多输出/多分类循环
        for k in range(self.n_trees_per_iteration_):
            grower = TreeGrower(X_binned=X_binned_train, gradients=g_view[:, k], hessians=h_view[:, k], ...)
            grower.grow()
            if not self._loss.differentiable:
                _update_leaves_values(self._loss, grower, y_train, raw_predictions[:, k], sample_weight_train)
            predictor = grower.make_predictor(self._bin_mapper.bin_thresholds_)
            self._predictors[-1].append(predictor)
            _update_raw_predictions(raw_predictions[:, k], grower, n_threads)
        # 早停检查
        if self.do_early_stopping_:
            should_early_stop = self._check_early_stopping_...
            if should_early_stop: break
    return self

25.7.3 损失函数与梯度计算

源码路径：sklearn/ensemble/_hist_gradient_boosting/common.pyx - _compute_gradients_hessians 与 _loss_function

cpdef void _compute_gradients_hessians(loss, y_true, raw_prediction, sample_weight,
                                       gradient_out, hessian_out, n_threads) nogil:
    # 并行逐样本计算一二阶导数
    for i in prange(n_samples, schedule='static', num_threads=n_threads):
        gradient_out[i] = loss.gradient(y_true[i], raw_predication[i])
        if not loss.constant_hessian:
            hessian_out[i] = loss.hessian(y_true[i], raw_prediction[i])

支持的损失族：

二分类：HalfBinomialLoss（Logistic Loss）
多分类：HalfMultinomialLoss（Softmax + CrossEntropy）
回归：HalfSquaredError (MSE)、HalfPoissonLoss、 HalfGammaLoss、 PinballLoss (Quantile)
非可微损失：AbsoluteError (MAE) 通过 fit_intercept_only 线搜索（中位数/分位数）

25.8 直方图梯度提升：拟合与早停 —— 高效 GBDT 的“训练主控”

25.8.1 HistGradientBoostingClassifier/Regressor：用户接口

源码路径：sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py - HistGradientBoostingClassifier.__init__ 与 fit（第 50-350 行）

class HistGradientBoostingClassifier(BaseHistGradientBoosting):
    def __init__(self, loss="log_loss", *, learning_rate=0.1, max_iter=100,
                 max_leaf_nodes=31, max_depth=None, min_samples_leaf=20,
                 l2_regularization=0.0, max_features=1.0, max_bins=255,
                 categorical_features="from_dtype", monotonic_cst=None,
                 interaction_cst=None, warm_start=False, early_stopping="auto",
                 scoring="loss", validation_fraction=0.1, n_iter_no_change=10,
                 tol=1e-7, verbose=0, random_state=None, class_weight=None):
        super().__init__(loss=loss, learning_rate=learning_rate, max_iter=max_iter,
                         max_leaf_nodes=max_leaf_nodes, max_depth=max_depth,
                         min_samples_leaf=min_samples_leaf, l2_regularization=l2_regularization,
                         max_features=max_features, max_bins=max_bins,
                         categorical_features=categorical_features,
                         monotonic_cst=monotonic_cst, interaction_cst=interaction_cst,
                         warm_start=warm_start, early_stopping=early_stopping,
                         scoring=scoring, validation_fraction=validation_fraction,
                         n_iter_no_change=n_iter_no_change, tol=tol,
                         verbose=verbose, random_state=random_state)
        self.class_weight = class_weight

    def fit(self, X, y, sample_weight=None, *, X_val=None, y_val=None, sample_weight_val=None):
        # 处理类别特征编码、验证集分割、早停配置
        # 调用父类 fit
        return super().fit(X, y, sample_weight, X_val=X_val, y_val=y_val, sample_weight_val=sample_weight_val)

关键参数：

max_iter=100：最大提升轮数（树的数量）。
max_leaf_nodes=31：每棵树最大叶节点数（LightGBM 默认 31，平衡精度与速度）。
early_stopping="auto"：样本量 > 10000 时自动启用早停。
categorical_features="from_dtype"：自动识别 pandas/polars DataFrame 的 category 列。

25.8.2 类别特征与单调约束

类别特征原生支持：

_preprocess_X 中 OrdinalEncoder 将类别编码为 [0, n_categories)。
_BinMapper 为类别特征直接使用已知类别作为分箱边界。
Splitter._find_best_bin_to_split_category 使用位集枚举子集分裂，复杂度 O(2^K) 但 K 通常很小（≤ max_bins=255）。

单约束实现：

# 第 25 章 —— grower.py TreeGrower.split_next 中
if self.with_monotonic_cst:
    mid = (left.value + right_node.value) / 2
    if monotonic_cst == POS:
        left.set_children_bounds(node.children_lower_bound, mid)
        right.set_children_bounds(mid, node.children_upper_bound)
    elif NEG: ...
# 第 25 章 —— splitting.pyx compute_node_value 中
value = -sum_gradient / (sum_hessian + l2_regularization + 1e-15)
value = np.clip(value, lower_bound, upper_bound)

单调约束通过传递父节点的值域边界给子节点，在叶值计算时裁剪到允许范围内。

25.8.3 样本权重与类别不平衡

# 第 25 章 —— common.pyx _compute_sample_weight
cpdef void _compute_sample_weight(sample_weight, class_weight, y, n_threads) nogil:
    if class_weight == "balanced":
        # 计算类别权重
        class_weight_arr = compute_class_weight("balanced", classes, y)
        for i in range(n_samples):
            sample_weight[i] *= class_weight_arr[y[i]]
    elif sample_weight is not None:
        # 用户提供的 sample_weight 直接使用
        pass

梯度/海森计算时乘以样本权重：gradient *= sample_weight，hessian *= sample_weight，实现加权损失优化。

25.9 堆叠与投票 —— 异质模型的“协同决策”

25.9.1 StackingClassifier/Regressor：两层集成架构

源码路径：sklearn/ensemble/_stacking.py

class _BaseStacking(TransformerMixin, _BaseHeterogeneousEnsemble):
    def fit(self, X, y, **fit_params):
        # 1. 并行训练基学习器（全量数据）
        self.estimators_ = Parallel(n_jobs=self.n_jobs)(
            delayed(_fit_single_estimator)(clone(est), X, y, fit_params)
            for name, est in self.estimators if est != "drop"
        )
        # 2. 交叉验证生成元特征
        cv = check_cv(self.cv, y=y, classifier=is_classifier(self))
        predictions = Parallel(n_jobs=self.n_jobs)(
            delayed(cross_val_predict)(clone(est), X, y, cv=deepcopy(cv), method=meth, ...)
            for name, est, meth in zip(names, all_estimators, self.stack_method_)
            if est != "drop"
        )
        # 3. 拼接元特征矩阵
        X_meta = self._concatenate_predictions(X, predictions)
        # 4. 拟合元学习器
        _fit_single_estimator(self.final_estimator_, X_meta, y, fit_params)

关键设计：

元特征生成：_get_cv_predictions 使用 cross_val_predict 获取基学习器的交叉验证预测（out-of-fold），避免数据泄露。
**stack_method='auto'**：优先 predict_proba→decision_function→predict`，根据基学习器能力自动选择。
passthrough：可选将原始特征拼接到元特征矩阵，允许元学习器直接访问原始信息。

25.9.2 VotingClassifier/Regressor：平权/加权投票

源码路径：sklearn/ensemble/_voting.py

class VotingClassifier(ClassifierMixin, _BaseVoting):
    def predict(self, X):
        if self.voting == "soft":
            maj = np.argmax(self.predict_proba(X), axis=1)
        else:  # hard voting
            predictions = self._predict(X)  # shape (n_samples, n_estimators)
            maj = np.apply_along_axis(
                lambda x: np.argmax(np.bincount(x, weights=self._weights_not_none)),
                axis=1, arr=predictions)
        return self.le_.inverse_transform(maj)

    def predict_proba(self, X):
        avg = np.average(self._collect_probas(X), axis=0, weights=self._weights_not_none)
        return avg

class VotingRegressor(RegressorMixin, _BaseVoting):
    def predict(self, X):
        return np.average(self._predict(X), axis=1, weights=self._weights_not_none)

硬投票 vs 软投票：

硬投票：每个基学习器投一票，加权众数决定。
软投票：平均基学习器的预测概率，取最大概率类别。要求所有基学习器支持 predict_proba，且通常校准良好时效果更好。

25.10 AdaBoost —— 自适应的“权重调优师”

25.10.1 AdaBoostClassifier：SAMME 算法核心

源码路径：sklearn/ensemble/_weight_boosting.py - AdaBoostClassifier._boost()（第 200-400 行）

def _boost(self, iboost, X, y, sample_weight, random_state):
    estimator = self._make_estimator(random_state=random_state)
    estimator.fit(X, y, sample_weight=sample_weight)
    y_predict = estimator.predict(X)
    if iboost == 0:
        self.classes_ = getattr(estimator, "classes_", None)
        self.n_classes_ = len(self.classes_)
    incorrect = y_predict != y
    estimator_error = np.mean(np.average(incorrect, weights=sample_weight, axis=0))
    if estimator_error <= 0:
        return sample_weight, 1.0, 0.0
    if estimator_error >= 1.0 - (1.0 / n_classes):
        self.estimators_.pop(-1)
        return None, None, None
    # SAMME 权重公式
    estimator_weight = self.learning_rate * (
        np.log((1.0 - estimator_error) / estimator_error) + np.log(n_classes - 1.0)
    )
    if not iboost == self.n_estimators - 1:
        sample_weight = np.exp(
            np.log(sample_weight) + estimator_weight * incorrect * (sample_weight > 0)
        )
    return sample_weight, estimator_weight, estimator_error

SAMME 多分类推广：

二分类 AdaBoost：estimator_weight = 0.5 * log((1-err)/err)
多分类 SAMME：estimator_weight = learning_rate * (log((1-err)/err) + log(K-1))
样本权重更新：错误分类样本权重乘以 exp(estimator_weight)，正确分类样本权重不变。

概率估计 (_samme_proba)：

@staticmethod
def _samme_proba(estimator, n_classes, X):
    proba = estimator.predict_proba(X)
    proba = np.exp((n_classes - 1) * np.log(proba))  # 修正概率尺度
    return proba / proba.sum(axis=1)[:, np.newaxis]

25.10.2 AdaBoostRegressor：R2 算法（指数损失）

def _boost(self, iboost, X, y, sample_weight, random_state):
    estimator = self._make_estimator(random_state=random_state)
    bootstrap_idx = random_state.choice(n_samples, size=n_samples, replace=True, p=sample_weight)
    X_, y_ = _safe_indexing(X, bootstrap_idx), _safe_indexing(y, bootstrap_idx)
    estimator.fit(X_, y_)
    y_predict = estimator.predict(X)
    error_vect = np.abs(y_predict - y)
    error_max = error_vect[sample_weight > 0].max()
    if error_max != 0:
        error_vect /= error_max
    if self.loss == "square": error_vect **= 2
    elif self.loss == "exponential": error_vect = 1.0 - np.exp(-error_vect)
    estimator_error = (sample_weight * error_vect).sum()
    if estimator_error >= 0.5:
        return None, None, None
    beta = estimator_error / (1.0 - estimator_error)
    estimator_weight = self.learning_rate * np.log(1.0 / beta)
    sample_weight[sample_weight > 0] *= np.power(beta, (1.0 - error_vect) * self.learning_rate)
    return sample_weight, estimator_weight, estimator_error

预测聚合：加权中位数而非加权平均（鲁棒性更强）。

def _get_median_predict(self, X, limit):
    predictions = np.array([est.predict(X) for est in self.estimators_[:limit]]).T
    sorted_idx = np.argsort(predictions, axis=1)
    weight_cdf = np.cumsum(self.estimator_weights_[sorted_idx], axis=1)
    median_idx = (weight_cdf >= 0.5 * weight_cdf[:, -1][:, np.newaxis]).argmax(axis=1)
    return predictions[np.arange(n_samples), median_idx]

25.11 隔离森林 —— 检测异常的“孤岛猎人”

25.11.1 IsolationForest：隔离树构建与异常分数

源码路径：sklearn/ensemble/_iforest.py - IsolationForest.fit() 与 decision_function()（第 150-450 行）

def fit(self, X, y=None, sample_weight=None):
    X = validate_data(self, X, accept_sparse=["csc"], dtype=tree_dtype, ensure_all_finite=False)
    rnd = check_random_state(self.random_state)
    y = rnd.uniform(size=X.shape[0])  # 伪造目标值，仅用于树构建
    # max_samples 默认 min(256, n_samples)
    if self.max_samples == "auto":
        max_samples = min(256, n_samples)
    max_depth = int(np.ceil(np.log2(max(max_samples, 2))))
    super()._fit(X, y, max_samples=max_samples, max_depth=max_depth, sample_weight=sample_weight, check_input=False)
    # 预计算每棵树的平均路径长度与决策路径长度
    self._average_path_length_per_tree, self._decision_path_lengths = zip(*[
        (_average_path_length(tree.tree_.n_node_samples), tree.tree_.compute_node_depths())
        for tree in self.estimators_
    ])
    # 设置 offset_
    if self.contamination == "auto":
        self.offset_ = -0.5
    else:
        self.offset_ = np.percentile(self._score_samples(X), 100.0 * self.contamination)
    return self

隔离树配置：

基学习器：ExtraTreeRegressor(max_features=1, splitter='random') —— 单特征随机分裂。
最大深度：ceil(log2(max_samples))，对应二叉树隔离单个样本所需的最大分裂次数。

25.11.1.1 _average_path_length：路径长度期望公式

def _average_path_length(n_samples_leaf):
    n_samples_leaf = check_array(n_samples_leaf, ensure_2d=False)
    average_path_length = np.zeros(n_samples_leaf.shape)
    mask_1 = n_samples_leaf <= 1
    mask_2 = n_samples_leaf == 2
    not_mask = ~np.logical_or(mask_1, mask_2)
    average_path_length[mask_1] = 0.0
    average_path_length[mask_2] = 1.0
    # c(n) = 2H(n-1) - 2(n-1)/n, H 为调和数
    average_path_length[not_mask] = (
        2.0 * (np.log(n_samples_leaf[not_mask] - 1.0) + np.euler_gamma)
        - 2.0 * (n_samples_leaf[not_mask] - 1.0) / n_samples_leaf[not_mask]
    )
    return average_path_length.reshape(n_samples_leaf_shape)

数学原理：隔离树等价于不成功的二叉搜索树（BST）查找，平均路径长度：

\[c(n) = 2H(n-1) - \frac{2(n-1)}{n} \approx 2\ln(n-1) + 2\gamma - 2 + \frac{2}{n} \]

其中 \(\gamma \approx 0.577\) 为欧拉-马歇罗尼常数。

25.11.1.2 decision_function / score_samples：异常分数计算

def _compute_chunked_score_samples(self, X):
    n_samples = _num_samples(X)
    chunk_n_rows = get_chunk_n_rows(row_bytes=16 * self._max_features, max_n_rows=n_samples)
    slices = gen_batches(n_samples, chunk_n_rows)
    scores = np.zeros(n_samples, order="f")
    for sl in slices:
        scores[sl] = self._compute_score_samples(X[sl], subsample_features)
    return scores

def _compute_score_samples(self, X, subsample_features):
    depths = np.zeros(n_samples, order="f")
    lock = threading.Lock()
    Parallel(require="sharedmem")(
        delayed(_parallel_compute_tree_depths)(
            tree, X, features, self._decision_path_lengths[i],
            self._average_path_length_per_tree[i], depths, lock)
        for i, (tree, features) in enumerate(zip(self.estimators_, self.estimators_features_))
    )
    denominator = len(self.estimators_) * _average_path_length([self._max_samples])
    scores = 2 ** (-np.divide(depths, denominator, out=np.ones_like(depths), where=denominator!=0))
    return scores

def decision_function(self, X):
    return self.score_samples(X) - self.offset_

并行计算：分块处理大数据集，每块并行累加各树的路径长度，最后归一化为 [0,1] 区间的异常分数：

\[s(x, n) = 2^{-\frac{E[h(x)]}{c(n)}} \]

路径越短（越易隔离），分数越接近 1（异常）；路径越长，分数越接近 0（正常）。

25.12 设计中的取舍

25.12.1 为什么不用单一巨大决策树而用集成？

单树：高方差，极易过拟合，对噪声敏感。
Bagging (RF)：降低方差，并行训练快，OOB 免验证集评估。
Boosting (GBDT/HGBT)：降低偏差，串行纠错，精度通常更高，但训练不可并行，易过拟合噪声数据。
HGBT vs GBDT：HGBT 分箱近似分裂寻优，时间复杂度从 O(n_samples × n_features) 降为 O(n_bins × n_features)，大规模数据上快 10-100 倍，原生支持缺失值/类别特征/单调约束。

25.12.2 为什么 RandomForest 要用 Bootstrap 而 ExtraTrees 不用？

RF (Bootstrap=True)：样本扰动 + 特征扰动，双重随机性解相关，OOB 样本天然存在。
ET (Bootstrap=False)：全量样本 + 随机分裂阈值，随机性更强，树间相关性更低，训练更快（无 Bootstrap 采样开销），但无 OOB 评估。

25.12.3 为什么 HGBT 用分箱近似而非精确贪心分裂？

精确贪心：需排序所有特征值，寻找最佳分裂点，O(n log n) 或 O(n) 配合直方图，但内存访问不连续，难以向量化。
分箱近似：预先将特征离散化为 ≤255 个分箱，分裂寻优仅在分箱边界搜索，直方图累加梯度/海森，内存连续，极易 SIMD 并行化，且缺失值天然归入专用分箱。

25.12.4 为什么隔离森林不需要显式密度/距离计算？

核心洞见：异常样本“少且不同”，在随机分裂下更容易被早早隔离（路径短）。
无需距离度量：完全基于树结构的路径长度，天然适应高维稀疏数据，避免了“维度灾难”下距离失效的问题。
计算高效：max_samples=256 限制树规模，训练/预测均为线性时间复杂度。

25.13 动手练习

阅读随机森林的并行构建与 OOB 评估机制
- 阅读 sklearn/ensemble/_forest.py 中 BaseForest._parallel_build_trees 与 BaseBagging._set_oob_score，理解：
  1. _parallel_build_trees 如何利用 joblib 并行构建多棵树
  2. OOB 样本索引如何通过 _generate_unsampled_indices 生成
  3. _set_oob_score 如何聚合各树对 OOB 样本的预测计算 OOB 分数
- 回答问题：
  - 为什么 max_samples < 1.0 时才会有 OOB 样本？
  - OOB 分数与交叉验证分数的统计关系是什么？
对比传统 GBDT 与 HistGradientBoosting 的分裂寻优策略
- 阅读 sklearn/ensemble/_gradient_boosting.pyx 中 Predictor.update_terminal_regions 与 sklearn/ensemble/_hist_gradient_boosting/grower.py 中 TreeGrower._find_best_split，对比：
  1. 传统 GBDT 如何遍历所有特征值寻找最佳分裂点（精确贪心）
  2. HGBT 如何利用分箱直方图在离散化桶上寻找近似最优分裂
  3. HGBT 中 Splitter.find_node_split 如何处理连续特征与类别特征的分裂增益计算
- 回答问题：
  - HGBT 的分箱策略为何能大幅降低分裂寻优的时间复杂度？
  - 类别特征在 HGBT 中如何利用位集实现高效的子集分裂？
分析 StackingClassifier 的交叉验证元特征生成流程
- 阅读 sklearn/ensemble/_stacking.py 中 _get_cv_predictions 与 StackingClassifier.fit，理解：
  1. _get_cv_predictions 如何利用 cross_val_predict 生成每个基学习器的交叉验证预测
  2. 元特征矩阵如何按列拼接所有基学习器的预测（概率或决策函数值）
  3. 元学习器如何在完整训练集上拟合元特征矩阵
- 回答问题：
  - 为什么堆叠泛化必须使用交叉验证预测而非直接 fit 预测生成元特征？
  - stack_method='auto' 时如何根据基学习器能力选择 predict_proba/decision_function/predict？
探究隔离森林的路径长度与异常分数计算
- 阅读 sklearn/ensemble/_iforest.py 中 IsolationForest._compute_chunked_score_samples 与 _average_path_length，分析：
  1. _average_path_length 如何基于调和数近似计算不成功 BST 搜索的平均路径长度
  2. decision_function 如何将平均路径长度归一化为 [0,1] 区间的异常分数
  3. 并行计算 _parallel_decision_function 如何分块处理大规模数据
- 回答问题：
  - 为什么隔离森林不需要显式计算密度或距离即可检测异常？
  - max_samples 参数如何影响隔离树的深度与异常检测的敏感度？
实现自定义 AdaBoost 基学习器权重更新策略
- 参考 sklearn/ensemble/_weight_boosting.py 中 AdaBoostClassifier.fit 与 _samme_proba，设计一个变体：
  1. 继承 AdaBoostClassifier，重写 _boost 方法
  2. 实现基于梯度幅度而非分类误差的样本权重更新规则
  3. 在 predict_proba 中融合新权重计算加权概率
- 回答问题：
  - AdaBoost.R2 回归变体为何使用指数损失而非分类误差？
  - 如何验证自定义权重更新策略在噪声数据上的鲁棒性优于原版 SAMME？

25.14 本章小结

这一章中我们深入剖析了 scikit-learn 集成学习模块的核心实现。首先，我们学习了 Bagging 与随机森林的自助采样、并行训练、OOB 评估机制，对比了 RandomForest 与 ExtraTrees 在 Bootstrap、分裂策略上的差异化设计。其次，我们详细解析了传统 GBDT 的分阶段加法建模、负梯度拟合、Cython 优化的叶值线搜索与预测器结构。接着，我们重点探讨了 HistGradientBoosting 的现代工程优化：分箱映射、直方图加速分裂寻优、最佳优先树生长、缺失值自动分流、位集类别分裂、扁平化预测器等关键技术。随后，我们了解了堆叠泛化的交叉验证元特征生成与投票集成的硬/软投票机制。最后，我们剖析了 AdaBoost 的 SAMME/R2 权重自适应算法，以及隔离森林基于路径长度的无监督异常检测原理。

本章我们一起学习了以下概念：

| 概念 | 解释 |

|------|------|

| BaseBagging | Bagging 元估计器基类，实现自助采样、并行训练、OOB 评估与预测聚合 |

| BaseForest | 森林基类，管理树的并行构建、特征随机化、apply/decision_path 等共享逻辑 |

| RandomForest / ExtraTrees | 随机森林（最佳分裂+Bootstrap）与极端随机树（随机分裂+全量样本）的分类/回归实现 |

| BaseGradientBoosting | 传统 GBDT 基类，分阶段拟合负梯度，支持多种损失，依赖 Cython 优化预测器更新终端区域 |

| HistGradientBoosting | 直方图梯度提升：分箱+直方图加速分裂寻优，最佳优先生长，原生支持缺失值与类别特征 |

| TreeGrower / Splitter / HistogramBuilder | HGBT 核心组件：树生长控制器、分裂寻优器、直方图构建器，协同完成高效树构建 |

| StackingClassifier/Regressor | 堆叠泛化：交叉验证生成元特征，元学习器综合基学习器预测 |

| VotingClassifier/Regressor | 投票集成：硬投票（众数）与软投票（加权平均概率）聚合异质模型预测 |

| AdaBoost | 自适应提升：SAMME 分类与 R2 回归变体，按误差调整样本权重聚焦难分样本 |

| IsolationForest | 隔离森林：随机分裂构建隔离树，路径长度越短越异常，并行计算异常分数 |

下一章中，我们将学习特征选择与降维技术，理解如何从高维数据中提取最具判别力的特征子集，进一步提升模型性能与可解释性。

第 26 章 —— 决策树与协方差

26.1 学习目标

难度：★★★☆☆（3/5）
预备知识：Python 基础、面向对象编程与 Markdown/代码阅读基础
理解决策树的构建与剪枝，掌握分裂策略、不纯度计算及特征重要性
理解最近邻算法的度量学习与索引构建，掌握KDTree、BallTree的查询机制
理解核密度估计（KDE）与局部离群因子（LOF）在密度估计与异常检测中的应用
理解邻域成分分析（NCA）的有监督度量学习与变换机制
理解高斯混合模型的参数估计、EM算法与协方差类型选择
理解贝叶斯高斯混合模型的变分推断与模型选择机制
熟悉各种协方差估计方法及其在鲁棒性与异常检测中的应用

26.2 生活类比

想象决策树就像玩“二十个问题”猜物品：每个问题（特征）都在尽可能减剩余可能性（不纯度），直到确定答案（叶节点）。剪枝则像在问完所有问题后，发现有些问题其实没用，于是去掉它们让决策更简洁。

最近邻算法就像在陌生城市问路：你不是看地图，而是直接询问周围几个人（k个最近邻）他们怎么去目的地，然后取多数意见或加权平均。KDTree和BallTree则是提前把城市按区域划好，问路时只需查看相关区域，省去逐个问所有人的时间。

核密度估计（KDE）像在街头观察人流密度：不是只看某个点的人数，而是看其附近区域有多少人，用平滑函数（如高斯核）给近处的人更高权重，远处的少一些，这样得到的密度图更真实。LOF则进一步说：一个点是否异常，不仅看它周围是否有人少，还要看它周围的人周围是否也有人少——如果自己周围空旷但附近的人都在热闹区域，那它才是真正的异常（如孤岛上的岗哨）。

邻域成分分析（NCA）就像给地图加了放大镜：它学习一个线性变换，让同类别的点在变换后的空间里更近，不同类别的点更远，从而让最近邻分类器在变换后的空间里表现更好——这相当于先扭曲地图让同城的朋友聚在一起，异城的分散开，再用最近邻判断归属。

高斯混合模型则像调色板上的颜料混合，每种颜料代表一个高斯分量。混合权重决定每种颜料在最终混合色中的比例，均值向量对应每种颜料的基础色调（如群青、赭石或钛白），而协方差矩阵描述颜料在画布上的扩散方式：各向同性（球形）、轴对称（对角）或完全各向异性（自由形状）。EM算法则像一位调色师：先猜测每种颜料的用量（E步），根据已混合的颜色估算实际用量（M步），再根据用量调整比例，如此反复直到整体色调稳定。贝叶斯高斯混合在此基础上加入了画家的主观偏好：先验分布代表画家在混合前对每种颜料用量的期望（如认为群青用量应较多），后验分布则结合了先验与实际混合结果，更符合观测到的画作特征。变分推断则用简化的假设（如颜料用量独立）近似复杂的后验，从而便于计算。协方差估计则类似于评估画笔笔触的分散特性：经验协方差直接观测所有笔触的偏离情况，但在样本量小时易受极值影响；Ledoit-Wolf收缩在经验估计上拉向均值，减少噪声影响，类似于调色时加入中性灰以降低饱和度；图拉索正则化通过稀疏约束强调主要成分，等同于仅保留最显眼的笔触方向；鲁棒MCD估计通过忽略极端笔触来计算协方差，类似于剔除明显异常的飞溅点再评估分散度；最后，椭圆包络异常检测以鲁棒协方差为基础，设定概率等值轮廓，轮廓外点视为异常，就像在色彩分布图上标注异常点。

26.3 源码地图

sklearn/tree/_classes.py

├── BaseDecisionTree.init

├── BaseDecisionTree._fit

├── BaseDecisionTree.predict

├── BaseDecisionTree.apply

├── BaseDecisionTree.decision_path

├── BaseDecisionTree._prune_tree

├── DecisionTreeClassifier.init

├── DecisionTreeClassifier.fit

├── DecisionTreeClassifier.predict_proba

├── DecisionTreeRegressor.init

├── DecisionTreeRegressor.fit

├── DecisionTreeRegressor._compute_partial_dependence_recursion

├── ExtraTreeClassifier.init

├── ExtraTreeRegressor.init

└── BaseDecisionTree.sklearn_tags

sklearn/tree/_splitter.pyx

├── Splitter.cinit

├── Splitter.init

├── Splitter.node_reset

├── Splitter.node_split

├── node_split_best

├── node_split_random

├── BestSplitter.init

├── BestSplitter.node_split

├── BestSparseSplitter.init

├── BestSparseSplitter.node_split

├── RandomSplitter.init

├── RandomSplitter.node_split

├── RandomSparseSplitter.init

├── RandomSparseSplitter.node_split

└── ShiftMissingValuesToLeftIfRequired

sklearn/tree/_criterion.pyx

├── Criterion.cinit

├── Criterion.init

├── Criterion.init_missing

├── Criterion.reset

├── Criterion.reverse_reset

├── Criterion.update

├── Criterion.node_impurity

├── Criterion.children_impurity

├── Criterion.node_value

├── Criterion.clip_node_value

├── Criterion.middle_value

├── Criterion.check_monotonicity

├── ClassificationCriterion.cinit

├── ClassificationCriterion.init

├── ClassificationCriterion.init_missing

├── ClassificationCriterion.reset

├── ClassificationCriterion.reverse_reset

├── ClassificationCriterion.update

├── ClassificationCriterion.node_impurity

├── ClassificationCriterion.children_impurity

├── ClassificationCriterion.node_value

├── ClassificationCriterion.clip_node_value

├── ClassificationCriterion.middle_value

├── ClassificationCriterion.check_monotonicity

├── Entropy.cinit

├── Entropy.node_impurity

├── Entropy.children_impurity

├── Gini.cinit

├── Gini.node_impurity

├── Gini.children_impurity

├── RegressionCriterion.cinit

├── RegressionCriterion.init

├── RegressionCriterion.init_missing

├── RegressionCriterion.reset

├── RegressionCriterion.reverse_reset

├── RegressionCriterion.update

├── RegressionCriterion.node_impurity

├── RegressionCriterion.children_impurity

├── RegressionCriterion.node_value

├── RegressionCriterion.clip_node_value

├── RegressionCriterion.middle_value

├── RegressionCriterion.check_monotonicity

├── MSE.cinit

├── MSE.node_impurity

├── MSE.proxy_impurity_improvement

├── MSE.children_impurity

├── MAE.cinit

├── MAE.init

├── MAE.init_missing

├── MAE.reset

├── MAE.reverse_reset

├── MAE.update

├── MAE.node_value

├── MAE.middle_value

├── MAE.check_monotonicity

├── MAE.children_impurity

├── Poisson.cinit

├── Poisson.node_impurity

├── Poisson.proxy_impurity_improvement

├── Poisson.children_impurity

├── Poisson.poisson_loss

└── WeightedFenwickTree

sklearn/tree/_partitioner.pyx

├── DensePartitioner.init

├── DensePartitioner.init_node_split

├── DensePartitioner.sort_samples_and_feature_values

├── DensePartitioner.find_min_max

├── DensePartitioner.next_p

├── DensePartitioner.partition_samples

├── DensePartitioner.partition_samples_final

├── SparsePartitioner.init

├── SparsePartitioner.init_node_split

├── SparsePartitioner.sort_samples_and_feature_values

├── SparsePartitioner.find_min_max

├── SparsePartitioner.next_p

├── SparsePartitioner.partition_samples

├── SparsePartitioner.partition_samples_final

├── SparsePartitioner._partition

├── SparsePartitioner.extract_nnz

├── compare_SIZE_t

├── binary_search

├── extract_nnz_index_to_samples

├── extract_nnz_binary_search

├── sparse_swap

├── _py_sort

├── sort

├── swap

├── median3

├── introsort

├── sift_down

├── heapsort

└── ShiftMissingValuesToLeftIfRequired

sklearn/tree/_tree.pyx

├── Tree.cinit

├── Tree.dealloc

├── Tree.reduce

├── Tree.getstate

├── Tree.setstate

├── Tree._resize

├── Tree._resize_c

├── Tree._add_node

├── Tree.predict

├── Tree.apply

├── Tree._apply_dense

├── Tree._apply_sparse_csr

├── Tree.decision_path

├── Tree._decision_path_dense

├── Tree._decision_path_sparse_csr

├── Tree.compute_node_depths

├── Tree.compute_feature_importances

├── Tree._get_value_ndarray

├── Tree._get_node_ndarray

├── Tree.compute_partial_dependence

├── _check_n_classes

├── _check_value_ndarray

├── _dtype_to_dict

├── _dtype_dict_with_modified_bitness

├── _all_compatible_dtype_dicts

├── _check_node_ndarray

├── _CCPPruneController

├── _AlphaPruner

├── _PathFinder

├── CostComplexityPruningRecord

├── _cost_complexity_prune

├── _build_pruned_tree_ccp

├── ccp_pruning_path

├── BuildPrunedRecord

├── _build_pruned_tree

├── _build_pruned_tree_py

└── _build_pruned_tree_py

sklearn/tree/_utils.pyx

├── safe_realloc

├── _realloc_test

├── sizet_ptr_to_ndarray

├── rand_int

├── rand_uniform

├── log

├── WeightedFenwickTree.cinit

├── WeightedFenwickTree.reset

├── WeightedFenwickTree.dealloc

├── WeightedFenwickTree.add

├── WeightedFenwickTree.search

└── PytestWeightedFenwickTree

sklearn/tree/_export.py

├── _color_brew

├── Sentinel

├── plot_tree

├── _BaseTreeExporter

├── _DOTTreeExporter

├── _MPLTreeExporter

├── export_graphviz

├── _compute_depth

├── export_text

└── _BaseTreeExporter

sklearn/neighbors/_classification.py

├── _adjusted_metric

├── KNeighborsClassifier.init

├── KNeighborsClassifier.fit

├── KNeighborsClassifier.predict

├── KNeighborsClassifier.predict_proba

├── KNeighborsClassifier.score

├── KNeighborsClassifier.sklearn_tags

├── RadiusNeighborsClassifier.init

├── RadiusNeighborsClassifier.fit

├── RadiusNeighborsClassifier.predict

├── RadiusNeighborsClassifier.predict_proba

├── RadiusNeighborsClassifier.score

├── RadiusNeighborsClassifier.sklearn_tags

└── KNeighborsClassifier.sklearn_tags

sklearn/neighbors/_regression.py

├── KNeighborsRegressor.init

├── KNeighborsRegressor.fit

├── KNeighborsRegressor.predict

├── KNeighborsRegressor.sklearn_tags

├── RadiusNeighborsRegressor.init

├── RadiusNeighborsRegressor.fit

├── RadiusNeighborsRegressor.predict

├── RadiusNeighborsRegressor.sklearn_tags

└── KNeighborsRegressor.sklearn_tags

sklearn/neighbors/_base.py

├── _get_weights

├── _is_sorted_by_data

├── _check_precomputed

├── sort_graph_by_row_values

├── _kneighbors_from_graph

├── _radius_neighbors_from_graph

├── NeighborsBase.init

├── NeighborsBase._check_algorithm_metric

├── NeighborsBase._fit

├── NeighborsBase.sklearn_tags

├── KNeighborsMixin._kneighbors_reduce_func

├── KNeighborsMixin.kneighbors

├── KNeighborsMixin.kneighbors_graph

├── RadiusNeighborsMixin._radius_neighbors_reduce_func

├── RadiusNeighborsMixin.radius_neighbors

├── RadiusNeighborsMixin.radius_neighbors_graph

├── RadiusNeighborsMixin.sklearn_tags

└── NeighborsBase.sklearn_tags

sklearn/neighbors/_unsupervised.py

├── NearestNeighbors.init

├── NearestNeighbors.fit

└── NearestNeighbors.sklearn_tags

sklearn/neighbors/_partition_nodes.pyx

└── partition_node_indices

sklearn/neighbors/_graph.py

├── _check_params

├── _query_include_self

├── kneighbors_graph

├── radius_neighbors_graph

├── KNeighborsTransformer.init

├── KNeighborsTransformer.fit

├── KNeighborsTransformer.transform

├── KNeighborsTransformer.fit_transform

├__RadiusNeighborsTransformer.init

├── RadiusNeighborsTransformer.fit

├── RadiusNeighborsTransformer.transform

├── RadiusNeighborsTransformer.fit_transform

└── RadiusNeighborsTransformer.sklearn_tags

sklearn/neighbors/_kde.py

├── _choose_algorithm

├── KernelDensity.init

├── KernelDensity.fit

├── KernelDensity.score_samples

├── KernelDensity.score

├── KernelDensity.sample

└── KernelDensity.sklearn_tags

sklearn/neighbors/_lof.py

├── _check_novelty_fit_predict

├── LocalOutlierFactor.fit_predict

├── LocalOutlierFactor.fit

├── LocalOutlierFactor._check_novelty_predict

├── LocalOutlierFactor.predict

├── LocalOutlierFactor._predict

├── LocalOutlierFactor._check_novelty_decision_function

├── LocalOutlierFactor.decision_function

├── LocalOutlierFactor._check_novelty_score_samples

├── LocalOutlierFactor.score_samples

├── LocalOutlierFactor._local_reachability_density

└── LocalOutlierFactor.sklearn_tags

sklearn/neighbors/_nca.py

├── _parameter_constraints

├── NeighborhoodComponentsAnalysis.init

├── NeighborhoodComponentsAnalysis.fit

├── NeighborhoodComponentsAnalysis.transform

├── NeighborhoodComponentsAnalysis._initialize

├── NeighborhoodComponentsAnalysis._callback

├── NeighborhoodComponentsAnalysis._loss_grad_lbfgs

└── NeighborhoodComponentsAnalysis.sklearn_tags

sklearn/mixture/_gaussian_mixture.py

├── _check_weights

├── _check_means

├── _check_precision_positivity

├── _check_precision_matrix

├── _check_precisions_full

├── _check_precisions

├── _estimate_gaussian_covariances_full

├── _estimate_gaussian_covariances_tied

├── _estimate_gaussian_covariances_diag

├── _estimate_gaussian_covariances_spherical

├── _estimate_gaussian_parameters

├── _compute_precision_cholesky

├── _flipudlr

├── _compute_precision_cholesky_from_precisions

├── _compute_log_det_cholesky

├── _estimate_log_gaussian_prob

├── GaussianMixture.init

├── GaussianMixture._check_parameters

├── GaussianMixture._initialize_parameters

├── GaussianMixture._initialize

├── GaussianMixture._m_step

├── GaussianMixture._estimate_log_prob

├── GaussianMixture._estimate_log_weights

├── GaussianMixture._compute_lower_bound

├── GaussianMixture._get_parameters

├── GaussianMixture._set_parameters

├── GaussianMixture._n_parameters

├── GaussianMixture.bic

├── GaussianMixture.aic

├── GaussianMixture.sklearn_tags

└── GaussianMixture.sklearn_tags

sklearn/mixture/_bayesian_mixture.py

├── _log_dirichlet_norm

├── _log_wishart_norm

├── BayesianGaussianMixture.init

├── BayesianGaussianMixture._check_parameters

├── BayesianGaussianMixture._check_weights_parameters

├── BayesianGaussianMixture._check_means_parameters

├── BayesianGaussianMixture._check_precision_parameters

├── BayesianGaussianMixture._checkcovariance_prior_parameter

├── BayesianGaussianMixture._initialize

├── BayesianGaussianMixture._estimate_weights

├── BayesianGaussianMixture._estimate_means

├── BayesianGaussianMixture._estimate_precisions

├── BayesianGaussianMixture._estimate_wishart_full

├── BayesianGaussianMixture._estimate_wishart_tied

├── BayesianGaussianMixture._estimate_wishart_diag

├── BayesianGaussianMixture._estimate_wishart_spherical

├── BayesianGaussianMixture._m_step

├── BayesianGaussianMixture._estimate_log_weights

├── BayesianGaussianMixture._estimate_log_prob

├── BayesianGaussianMixture._compute_lower_bound

├── BayesianGaussianMixture._get_parameters

├── BayesianGaussianMixture._set_parameters

└── BayesianGaussianMixture.sklearn_tags

sklearn/covariance/_empirical_covariance.py

├── log_likelihood

├── empirical_covariance

├── EmpiricalCovariance.init

├── EmpiricalCovariance._set_covariance

├── EmpiricalCovariance.get_precision

├── EmpiricalCovariance.fit

├── EmpiricalCovariance.score

├── EmpiricalCovariance.error_norm

├── EmpiricalCovariance.mahalanobis

└── EmpiricalCovariance.sklearn_tags

sklearn/covariance/_shrunk_covariance.py

├── _ledoit_wolf

├── _oas

├── shrunk_covariance

├── ledoit_wolf_shrinkage

├── ledoit_wolf

├── LedoitWolf.init

├── LedoitWolf.fit

├── OAS.init

├── OAS.fit

├── _oas

└── LedoitWolf.sklearn_tags

sklearn/covariance/_robust_covariance.py

├── c_step

├── _c_step

├── _consistency_factor

├── select_candidates

├── fast_mcd

├── MinCovDet.init

├── MinCovDet.fit

├── MinCovDet.correct_covariance

├── MinCovDet.reweight_covariance

├── MinCovDet.sklearn_tags

└── MinCovDet.sklearn_tags

sklearn/covariance/_graph_lasso.py

├── GraphicalLasso.init

├── GraphicalLasso.fit

├── GraphicalLassoCV.init

├__GraphicalLassoCV.fit

├── graphical_lasso

├── graphical_lasso_path

├── _objective

├── _dual_gap

├── alpha_max

├── get_metadata_routing

└── GraphicalLassoCV.sklearn_tags

sklearn/covariance/_elliptic_envelope.py

├── EllipticEnvelope.init

├── EllipticEnvelope.fit

├── EllipticEnvelope.decision_function

├── EllipticEnvelope.score_samples

├── EllipticEnvelope.predict

├── EllipticEnvelope.score

└── EllipticEnvelope.sklearn_tags

26.4 决策树的构建与剪枝

决策树通过递归划分特征空间来构建模型，每个内部节点选择一个特征和阈值将数据分为两部分，目标是最大化不纯度下降（如基尼不纯度或信息增益）。构建过程包括特征选择、阈值搜索、停止条件判断（如最大深度、最小样本数）以及叶节点值计算。剪枝则通过移除对整体预测误差贡献小的子树来防止过拟合，常用方法包括代价复杂度剪枝（CCP）。

26.4.1 决策树的构建流程

源码路径：sklearn/tree/_classes.py - BaseDecisionTree._fit（200-300行）

def _fit(
    self,
    X,
    y,
    sample_weight=None,
    check_input=True,
    missing_values_in_feature_mask=None,
):
    random_state = check_random_state(self.random_state)

    if check_input:
        # Need to validate separately here.
        # We can't pass multi_output=True because that would allow y to be
        # csr.

        # _compute_missing_values_in_feature_mask will check for finite values and
        # compute the missing mask if the tree supports missing values
        check_X_params = dict(
            dtype=DTYPE, accept_sparse="csc", ensure_all_finite=False
        )
        check_y_params = dict(ensure_2d=False, dtype=None)
        X, y = validate_data(
            self, X, y, validate_separately=(check_X_params, check_y_params)
        )

        missing_values_in_feature_mask = (
            self._compute_missing_values_in_feature_mask(X)
        )
        if issparse(X):
            X.sort_indices()

            if X.indices.dtype != np.intc or X.indptr.dtype != np.intc:
                raise ValueError(
                    "No support for np.int64 index based sparse matrices"
                )

        if self.criterion == "poisson":
            if np.any(y < 0):
                raise ValueError(
                    "Some value(s) of y are negative which is"
                    " not allowed for Poisson regression."
                )
            if np.sum(y) <= 0:
                raise ValueError(
                    "Sum of y is not positive which is "
                    "necessary for Poisson regression."
                )

    # Determine output settings
    n_samples, self.n_features_in_ = X.shape
    is_classification = is_classifier(self)

    y = np.atleast_1d(y)
    expanded_class_weight = None

    if y.ndim == 1:
        # reshape is necessary to preserve the data contiguity against vs
        # [:, np.newaxis] that does not.
        y = np.reshape(y, (-1, 1))

    self.n_outputs_ = y.shape[1]

    if is_classification:
        check_classification_targets(y)
        y = np.copy(y)

        self.classes_ = []
        self.n_classes_ = []

        if self.class_weight is not None:
            y_original = np.copy(y)

        y_encoded = np.zeros(y.shape, dtype=int)
        for k in range(self.n_outputs_):
            classes_k, y_encoded[:, k] = np.unique(y[:, k], return_inverse=True)
            self.classes_.append(classes_k)
            self.n_classes_.append(classes_k.shape[0])
        y = y_encoded

        if self.class_weight is not None:
            expanded_class_weight = compute_sample_weight(
                self.class_weight, y_original
            )

        self.n_classes_ = np.array(self.n_classes_, dtype=np.intp)

    if getattr(y, "dtype", None) != DOUBLE or not y.flags.contiguous:
        y = np.ascontiguousarray(y, dtype=DOUBLE)

    max_depth = np.iinfo(np.int32).max if self.max_depth is None else self.max_depth

    if isinstance(self.min_samples_leaf, numbers.Integral):
        min_samples_leaf = self.min_samples_leaf
    else:  # float
        min_samples_leaf = ceil(self.min_samples_leaf * n_samples)

    if isinstance(self.min_samples_split, numbers.Integral):
        min_samples_split = self.min_samples_split
    else:  # float
        min_samples_split = ceil(self.min_samples_split * n_samples)
        min_samples_split = max(2, min_samples_split)

    min_samples_split = max(min_samples_split, 2 * min_samples_leaf)

    if isinstance(self.max_features, str):
        if self.max_features == "sqrt":
            max_features = max(1, int(np.sqrt(self.n_features_in_)))
            # // 逐行注释解释：计算sqrt特征数量
        elif self.max_features == "log2":
            max_features = max(1, int(np.log2(self.n_features_in_)))
            # // 逐行注释解释：计算log2特征数量
    elif self.max_features is None:
        max_features = self.n_features_in_
    elif isinstance(self.max_features, numbers.Integral):
        max_features = self.max_features
    else:  # float
        if self.max_features > 0.0:
            max_features = max(1, int(self.max_features * self.n_features_in_))
        else:
            max_features = 0

    self.max_features_ = max_features

    max_leaf_nodes = -1 if self.max_leaf_nodes is None else self.max_leaf_nodes

    if len(y) != n_samples:
        raise ValueError(
            "Number of labels=%d does not match number of samples=%d"
            % (len(y), n_samples)
        )

    if sample_weight is not None:
        sample_weight = _check_sample_weight(sample_weight, X, dtype=DOUBLE)

    if expanded_class_weight is not None:
        if sample_weight is not None:
            sample_weight = sample_weight * expanded_class_weight
        else:
            sample_weight = expanded_class_weight

    # Set min_weight_leaf from min_weight_fraction_leaf
    if sample_weight is None:
        min_weight_leaf = self.min_weight_fraction_leaf * n_samples
    else:
        min_weight_leaf = self.min_weight_fraction_leaf * np.sum(sample_weight)

    # Build tree
    criterion = self.criterion
    if not isinstance(criterion, Criterion):
        if is_classification:
            criterion = CRITERIA_CLF[self.criterion](
                self.n_outputs_, self.n_classes_
            )
        else:
            criterion = CRITERIA_REG[self.criterion](self.n_outputs_, n_samples)
    else:
        # Make a deepcopy in case the criterion has mutable attributes that
        # might be shared and modified concurrently during parallel fitting
        criterion = copy.deepcopy(criterion)

    SPLITTERS = SPARSE_SPLITTERS if issparse(X) else DENSE_SPLITTERS

    splitter = self.splitter
    if self.monotonic_cst is None:
        monotonic_cst = None
    else:
        if self.n_outputs_ > 1:
            raise ValueError(
                "Monotonicity constraints are not supported with multiple outputs."
            )
        # Check to correct monotonicity constraint' specification,
        # by applying element-wise logical conjunction
        # Note: we do not cast `np.asarray(self.monotonic_cst, dtype=np.int8)`
        # straight away here so as to generate error messages for invalid
        # values using the original values prior to any dtype related conversion.
        monotonic_cst = np.asarray(self.monotonic_cst)
        if monotonic_cst.shape[0] != X.shape[1]:
            raise ValueError(
                "monotonic_cst has shape {} but the input data "
                "X has {} features.".format(monotonic_cst.shape[0], X.shape[1])
            )
        valid_constraints = np.isin(monotonic_cst, (-1, 0, 1))
        if not np.all(valid_constraints):
            unique_constaints_value = np.unique(monotonic_cst)
            raise ValueError(
                "monotonic_cst must be None or an array-like of -1, 0 or 1, but"
                f" got {unique_constaints_value}"
            )
        monotonic_cst = np.asarray(monotonic_cst, dtype=np.int8)
        if is_classifier(self):
            if self.n_classes_[0] > 2:
                raise ValueError(
                    "Monotonicity constraints are not supported with multiclass "
                    "classification"
                )
                # // 逐行注释解释：多类不支持单调性约束
            # Binary classification trees are built by constraining probabilities
            # of the *negative class* in order to make the implementation similar
            # to regression trees.
            # Since self.monotonic_cst encodes constraints on probabilities of the
            # *positive class*, all signs must be flipped.
            monotonic_cst *= -1
            # // 逐行注释解释：二分类需反转单调性方向

    if not isinstance(self.splitter, Splitter):
        splitter = SPLITTERS[self.splitter](
            criterion,
            self.max_features_,
            min_samples_leaf,
            min_weight_leaf,
            random_state,
            monotonic_cst,
        )

    if is_classifier(self):
        self.tree_ = Tree(self.n_features_in_, self.n_classes_, self.n_outputs_)
    else:
        self.tree_ = Tree(
            self.n_features_in_,
            # TODO: tree shouldn't need this in this case
            np.array([1] * self.n_outputs_, dtype=np.intp),
            self.n_outputs_,
        )

    # Use BestFirst if max_leaf_nodes given; use DepthFirst otherwise
    if max_leaf_nodes < 0:
        builder = DepthFirstTreeBuilder(
            splitter,
            min_samples_split,
            min_samples_leaf,
            min_weight_leaf,
            max_depth,
            self.min_impurity_decrease,
        )
    else:
        builder = BestFirstTreeBuilder(
            splitter,
            min_samples_split,
            min_samples_leaf,
            min_weight_leaf,
            max_depth,
            max_leaf_nodes,
            self.min_impurity_decrease,
        )

    builder.build(self.tree_, X, y, sample_weight, missing_values_in_feature_mask)

    if self.n_outputs_ == 1 and is_classifier(self):
        self.n_classes_ = self.n_classes_[0]
        self.classes_ = self.classes_[0]

    self._prune_tree()

    return self

这段代码实现了决策树的构建流程：它先验证输入数据，处理类别编码和样本权重，然后根据max_features计算实际要考虑的特征数量，初始化损失函数和分裂器，选择深度优先或最佳优先构建策略，最后调用构建器生成树并执行代价复杂度剪枝。

源码路径：sklearn/tree/_tree.pyx - DepthFirstTreeBuilder.build（100-200行）

    cpdef build(
        self,
        Tree tree,
        object X,
        const float64_t[:, ::1] y,
        const float64_t[:] sample_weight=None,
        const uint8_t[::1] missing_values_in_feature_mask=None,
    ):
        """Build a decision tree from the training set (X, y)."""

        # check input
        X, y, sample_weight = self._check_input(X, y, sample_weight)

        # Initial capacity
        cdef intp_t init_capacity

        if tree.max_depth <= 10:
            init_capacity = <intp_t> (2 ** (tree.max_depth + 1)) - 1
        else:
            init_capacity = 2047

        tree._resize(init_capacity)

        # Parameters
        cdef Splitter splitter = self.splitter
        cdef intp_t max_depth = self.max_depth
        cdef intp_t min_samples_leaf = self.min_samples_leaf
        cdef float64_t min_weight_leaf = self.min_weight_leaf
        cdef intp_t min_samples_split = self.min_samples_split
        cdef float64_t min_impurity_decrease = self.min_impurity_decrease

        # Recursive partition (without actual recursion)
        splitter.init(X, y, sample_weight, missing_values_in_feature_mask)

        cdef intp_t start
        cdef intp_t end
        cdef intp_t depth
        cdef intp_t parent
        cdef bint is_left
        cdef intp_t n_node_samples = splitter.n_samples
        cdef float64_t weighted_n_node_samples
        cdef SplitRecord split
        cdef intp_t node_id

        cdef float64_t middle_value
        cdef float64_t left_child_min
        cdef float64_t left_child_max
        cdef float64_t right_child_min
        cdef float64_t right_child_max
        cdef bint is_leaf
        cdef bint first = 1
        cdef intp_t max_depth_seen = -1
        cdef int rc = 0

        cdef stack[StackRecord] builder_stack
        cdef StackRecord stack_record

        cdef ParentInfo parent_record
        _init_parent_record(&parent_record)

        with nogil:
            # push root node onto stack
            builder_stack.push({
                "start": 0,
                "end": n_node_samples,
                "depth": 0,
                "parent": _TREE_UNDEFINED,
                "is_left": 0,
                "impurity": INFINITY,
                "n_constant_features": 0,
                "lower_bound": -INFINITY,
                "upper_bound": INFINITY,
            })

            while not builder_stack.empty():
                stack_record = builder_stack.top()
                builder_stack.pop()

                start = stack_record.start
                end = stack_record.end
                depth = stack_record.depth
                parent = stack_record.parent
                is_left = stack_record.is_left
                parent_record.impurity = stack_record.impurity
                parent_record.n_constant_features = stack_record.n_constant_features
                parent_record.lower_bound = stack_record.lower_bound
                parent_record.upper_bound = stack_record.upper_bound

                n_node_samples = end - start
                splitter.node_reset(start, end, &weighted_n_node_samples)

                is_leaf = (depth >= max_depth or
                           n_node_samples < min_samples_split or
                           n_node_samples < 2 * min_samples_leaf or
                           weighted_n_node_samples < 2 * min_weight_leaf)

                if first:
                    parent_record.impurity = splitter.node_impurity()
                    first = 0

                # impurity == 0 with tolerance due to rounding errors
                is_leaf = is_leaf or parent_record.impurity <= EPSILON

                if not is_leaf:
                    splitter.node_split(
                        &parent_record,
                        &split,
                    )
                    # If EPSILON=0 in the below comparison, float precision
                    # issues stop splitting, producing trees that are
                    # dissimilar to v0.18
                    is_leaf = (is_leaf or split.pos >= end or
                               (split.improvement + EPSILON <
                                min_impurity_decrease))

                node_id = tree._add_node(parent, is_left, is_leaf, split.feature,
                                         split.threshold, parent_record.impurity,
                                         n_node_samples, weighted_n_node_samples,
                                         split.missing_go_to_left)

                if node_id == INTPTR_MAX:
                    rc = -1
                    break

                # Store value for all nodes, to facilitate tree/model
                # inspection and interpretation
                splitter.node_value(tree.value + node_id * tree.value_stride)
                if splitter.with_monotonic_cst:
                    splitter.clip_node_value(tree.value + node_id * tree.value_stride, parent_record.lower_bound, parent_record.upper_bound)

                if not is_leaf:
                    if (
                        not splitter.with_monotonic_cst or
                        splitter.monotonic_cst[split.feature] == 0
                    ):
                        # Split on a feature with no monotonicity constraint

                        # Current bounds must always be propagated to both children.
                        # If a monotonic constraint is active, bounds are used in
                        # node value clipping.
                        left_child_min = right_child_min = parent_record.lower_bound
                        left_child_max = right_child_max = parent_record.upper_bound
                    elif splitter.monotonic_cst[split.feature] == 1:
                        # Split on a feature with monotonic increase constraint
                        left_child_min = parent_record.lower_bound
                        right_child_max = parent_record.upper_bound

                        # Lower bound for right child and upper bound for left child
                        # are set to the same value.
                        middle_value = splitter.criterion.middle_value()
                        right_child_min = middle_value
                        left_child_max = middle_value
                    else:  # i.e. splitter.monotonic_cst[split.feature] == -1
                        # Split on a feature with monotonic decrease constraint
                        right_child_min = parent_record.lower_bound
                        left_child_max = parent_record.upper_bound

                        # Lower bound for left child and upper bound for right child
                        # are set to the same value.
                        middle_value = splitter.criterion.middle_value()
                        left_child_min = middle_value
                        right_child_max = middle_value

                    # Push right child on stack
                    builder_stack.push({
                        "start": split.pos,
                        "end": end,
                        "depth": depth + 1,
                        "parent": node_id,
                        "is_left": 0,
                        "impurity": split.impurity_right,
                        "n_constant_features": parent_record.n_constant_features,
                        "lower_bound": right_child_min,
                        "upper_bound": right_child_max,
                    })

                    # Push left child on stack
                    builder_stack.push({
                        "start": start,
                        "end": split.pos,
                        "depth": depth + 1,
                        "parent": node_id,
                        "is_left": 1,
                        "impurity": split.impurity_left,
                        "n_constant_features": parent_record.n_constant_features,
                        "lower_bound": left_child_min,
                        "upper_bound": left_child_max,
                    })

                if depth > max_depth_seen:
                    max_depth_seen = depth

            if rc >= 0:
                rc = tree._resize_c(tree.node_count)

            if rc >= 0:
                tree.max_depth = max_depth_seen
        if rc == -1:
            raise MemoryError()

posted @ 2026-09-04 04:09 绝不原创的飞龙阅读(4) 评论(0) 收藏举报

刷新页面返回顶部

龙哥盟

掠夺·扩张·投机·博弈