极限学习机（ELM）为什么难以与强化学习（RL）结合 —— ELM 结合 RL —— ELM + RL 性能效果极差（2）——elm+reinforce算法

前文：

极限学习机（ELM）为什么难以与强化学习（RL）结合 —— ELM 结合 RL —— ELM + RL 性能效果极差

给出代码：

import gym
import numpy as np
import random
from scipy.linalg import pinv


# 极限学习机 ELM（不变）
class ELM:
    def __init__(self, in_dim, hid_dim, out_dim):
        self.in_dim = in_dim
        self.hid_dim = hid_dim
        self.out_dim = out_dim
        self.W = np.random.randn(in_dim, hid_dim) * 0.5
        self.b = np.random.randn(1, hid_dim) * 0.5
        self.beta = np.random.randn(hid_dim, out_dim) * 0.01

    def hidden(self, x):
        x = x.reshape(1, -1)
        h = np.tanh(x @ self.W + self.b)
        return h

    def predict(self, x):
        h = self.hidden(x)
        out = h @ self.beta
        # 输出 logits，转概率
        # return np.exp(out) / np.sum(np.exp(out))
        return 1 / (1 + np.exp(-out))

    def train_batch(self, X, Y):
        H = np.vstack([self.hidden(x) for x in X])
        self.beta = pinv(H) @ Y

# REINFORCE with ELM (Policy Gradient)
def elm_reinforce():
    train_count = 0
    XXX = []
    YYY = []

    env = gym.make("CartPole-v1")
    s_dim = env.observation_space.shape[0]
    a_dim = env.action_space.n

    # ELM 直接输出策略 π(a|s)
    elm = ELM(s_dim, 1024, 1)
    gamma = 0.99
    eps = 1.0

    episode = 0
    solve_count = 0
    last_solved_ep = -1

    while True:
        s = env.reset()
        if isinstance(s, tuple):
            s = s[0]

        # 记录轨迹：s, a, r
        trajectory = []
        total_r = 0
        done = False

        # 采样一整条轨迹（蒙特卡洛）
        while not done:
            # 前向得到策略概率
            prob = elm.predict(s)
            # 按概率选动作
            # print(prob)
            if np.random.random()<prob[0]:
                # a = np.random.choice(a_dim, p=prob.ravel())
                a = 0
            else:
                a = 1

            ns, r, ter, tr, _ = env.step(a)
            done = ter or tr
            trajectory.append((s, a, r))
            s = ns
            total_r += r

        episode += 1

        # --------------------------
        # REINFORCE 核心：计算回报 G_t
        # --------------------------
        G = 0
        states  = []
        actions = []
        returns = []

        for s, a, r in reversed(trajectory):
            G = r + gamma * G
            states.append(s)
            actions.append(a)
            returns.append(G)

        # 反转回时间顺序
        states  = np.array(states)[::-1]
        actions = np.array(actions)[::-1]
        returns = np.array(returns)[::-1]

        # 回报标准化（稳定训练）
        if np.std(returns) > 1e-6:
            returns = (returns - np.mean(returns)) / np.std(returns)
        else:
            returns = returns - np.mean(returns)

        # --------------------------
        # 构建监督学习目标
        # 策略梯度目标：最大化 log π(a|s) * G_t
        # 这里转化为 ELM 拟合目标 Y
        # --------------------------
        X = states
        Y = np.zeros(len(X))

        for i in range(len(X)):
            s_i = X[i]
            a_i = actions[i]
            G_i = returns[i]
            prob = elm.predict(s_i).ravel()
            # 梯度上升 → 构造伪标签做最小二乘
            if a==0:
                Y[i] = np.clip(prob + G_i * 0.1, 0.0, 1.0)
            else:
                Y[i] = np.clip(prob - G_i * 0.1, 0.0, 1.0)

            # print(a, prob, G_i, Y[i])

        XXX.extend(X)
        YYY.extend(Y)
        if len(XXX)>1000:
            # 用轨迹训练 ELM
            elm.train_batch(XXX, YYY)
            XXX.clear()
            YYY.clear()
            train_count += 1

            print(f"Episode {episode:4d} | Reward: {total_r:4.0f} | 训练次数: {train_count:d}")

        # print(f"Episode {episode:4d} | Reward: {total_r:4.0f} | 训练次数: {train_count:d}")

        # 连续10次高分视为解决
        if total_r >= 490:
            if episode == last_solved_ep + 1:
                solve_count += 1
            else:
                solve_count = 1
            last_solved_ep = episode

            if solve_count >= 10:
                print("🎉 CartPole 已通过 ELM-REINFORCE 解决！")
                break

    env.close()

if __name__ == "__main__":
    elm_reinforce()

运行效果：

结论就是性能基本和随机差不多。

进一步说明RL 和 ELM 结合的可行性较差。

PS：

很多奇奇怪怪的研究方向之所以没有什么人研究，其实最大的可能性就是这根本走不通。