极限学习机（ELM）为什么难以与强化学习（RL）结合 —— ELM 结合 RL —— ELM + RL 性能效果极差

20年前读大学的时候，当时和大学同学谈恋爱，那个时候她被保送到东北大学的研究生，那个时候保研的课题就是ELM的NLP，小胡同学，那个时候就知道这个ELM了。

不过一直没有多接触这个ELM。这些年一直在搞RL，最近赋闲在家，突然想搞搞ELM和RL的结合，于是有了本文。

给出一个基于ELM的RL算法代码，思路就是用蒙特卡洛方式估计状态Q值，函数拟合使用ELM，代码如下：

import gym
import numpy as np
import random
from collections import deque
from scipy.linalg import pinv

# 极限学习机 ELM
class ELM:
    def __init__(self, in_dim, hid_dim, out_dim):
        self.in_dim = in_dim
        self.hid_dim = hid_dim
        self.out_dim = out_dim
        self.W = np.random.randn(in_dim, hid_dim) * 0.5
        self.b = np.random.randn(1, hid_dim) * 0.5
        self.beta = np.random.randn(hid_dim, out_dim) * 0.01

    def hidden(self, x):
        x = x.reshape(1, -1)
        h = np.tanh(x @ self.W + self.b)
        return h

    def predict(self, x):
        h = self.hidden(x)
        return h @ self.beta

    def train_batch(self, X, Y):
        # print(Y)
        H = np.vstack([self.hidden(x) for x in X])
        self.beta = pinv(H) @ Y

def elm_q_learning():
    env = gym.make("CartPole-v1")
    s_dim = env.observation_space.shape[0]
    a_dim = env.action_space.n

    elm = ELM(s_dim + 1, 1024, 1)
    memory = []
    gamma = 0.95
    eps = 1.0 # 0.1 # 1.0

    trials = 0
    ex_episode = 0
    result = []
    for epi in range(10000000000):
        s = env.reset()

        if isinstance(s, tuple):
            s = s[0]

        temp = []
        total_r = 0
        done = False
        step = 0

        while not done:
            if np.random.rand() < max(eps, 0.01):
                a = random.randint(0, a_dim-1)
            else:
                q_0 = elm.predict(np.concatenate((s, np.array([0]))))
                q_1 = elm.predict(np.concatenate((s, np.array([1]))))
                if q_0 > q_1:
                    a = 0
                else:
                    a = 1

            ns, r, ter, tr, _ = env.step(a)
            done = ter or tr
            total_r += r
            temp.append((s, a, r, ns, done))
            s = ns
            step += 1

        _r = 0
        for s, a, r, ns, done in temp[::-1]:
            _r = r+gamma*_r
            memory.append((np.concatenate((s, np.array([a])*1.0)), _r))

        result.append(total_r)
        # print(f"ELM-Q episode {epi:3d} | reward {total_r:4.0f} | eps {eps:.3f}")

        # 认为连续高分即解决
        if total_r >= 490:
            if epi == ex_episode+1:
                all_time += 1
            else:
                all_time = 1
            ex_episode = epi
            # print(epi, ex_episode, all_time)
            if all_time == 10:
                print("CartPole 已通过ELM强化学习解决！")
                break




        # 训练一次 ELM
        if len(memory) > 100000:
            trials += 1
            print("训练 ", trials, np.mean(result), epi, eps)
            X_batch = []
            Y_batch = []
            for s, q in memory:
                """
                q = elm.predict(s_b)[0]
                q_n = elm.predict(ns_b)[0]
                if d_b:
                    q[a_b] = r_b
                else:
                    q[a_b] = r_b + gamma * np.max(q_n)
                """
                X_batch.append(s)
                Y_batch.append(q)
            elm.train_batch(X_batch, np.array(Y_batch))

            eps *= 0.95
            memory.clear()
            result.clear()

    env.close()

if __name__ == "__main__":
    elm_q_learning()

运行效果：(第一次运行)

[11:59](base) devil@Monster:/tmp/example$ python main.py
/home/devil/anaconda3/lib/python3.11/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
  if not isinstance(terminated, (bool, np.bool8)):
训练  1 22.153965440850687 4513 1.0
训练  2 24.96455317024463 8519 0.95
训练  3 28.42609437180216 12037 0.9025
训练  4 32.649036891936014 15100 0.8573749999999999
训练  5 36.02557636887608 17876 0.8145062499999999
训练  6 42.23141891891892 20244 0.7737809374999999
训练  7 49.11738703339882 22280 0.7350918906249998
训练  8 59.5854675402025 23959 0.6983372960937497
训练  9 70.04481792717087 25387 0.6634204312890623
训练  10 79.02209944751381 26654 0.6302494097246091
训练  11 93.64700374531836 27722 0.5987369392383786
训练  12 112.24188129899215 28615 0.5688000922764596
训练  13 125.63362609786701 29412 0.5403600876626365
训练  14 152.0864946889226 30071 0.5133420832795047
训练  15 177.74600355239787 30634 0.48767497911552943
训练  16 169.5871404399323 31225 0.46329123015975293
训练  17 205.62217659137576 31712 0.44012666865176525
训练  18 182.19854280510017 32261 0.41812033521917696
训练  19 192.48846153846154 32781 0.3972143184582181
训练  20 157.65984251968504 33416 0.37735360253530714
训练  21 144.44877344877344 34109 0.35848592240854177
训练  22 114.58171428571428 34984 0.34056162628811465
训练  23 211.61310782241014 35457 0.3235335449737089
训练  24 212.97027600849256 35928 0.30735686772502346
训练  25 103.28482972136223 36897 0.2919890243387723
训练  26 69.58484005563282 38335 0.27738957312183365
训练  27 119.6531100478469 39171 0.263520094465742
训练  28 108.4512987012987 40095 0.25034408974245487
训练  29 101.86863543788188 41077 0.2378268852553321
训练  30 133.34533333333334 41827 0.2259355409925655
训练  31 132.65119363395226 42581 0.2146387639429372
训练  32 180.1456834532374 43137 0.20390682574579033
训练  33 109.98901098901099 44047 0.1937114844585008
训练  34 117.31264637002342 44901 0.18402591023557577
训练  35 109.02505446623094 45819 0.17482461472379698
训练  36 67.80067796610169 47294 0.16608338398760714
训练  37 85.69494430162811 48461 0.15777921478822676
训练  38 130.8732026143791 49226 0.14989025404881542
训练  39 65.35336381450033 50757 0.14239574134637464
训练  40 58.89517078916372 52455 0.1352759542790559
训练  41 45.091974752028854 54673 0.1285121565651031
训练  42 53.28023441662227 56550 0.12208654873684793
训练  43 63.90607028753993 58115 0.11598222130000553
训练  44 82.82533112582782 59323 0.11018311023500525
训练  45 68.11027910142954 60792 0.10467395472325498
训练  46 101.11931243680485 61781 0.09944025698709223
训练  47 45.72702331961591 63968 0.09446824413773762
训练  48 21.825223652629283 68551 0.08974483193085074
训练  49 18.070460704607047 74086 0.0852575903343082
训练  50 35.54051172707889 76900 0.08099471081759278
训练  51 52.48583420776495 78806 0.07694497527671314
训练  52 44.88779174147217 81034 0.07309772651287748
训练  53 76.25381097560975 82346 0.0694428401872336
训练  54 43.84049079754601 84628 0.0659706981778719
训练  55 26.46599629531622 88407 0.0626721632689783
训练  56 42.75299145299145 90747 0.059538555105529384
训练  57 83.08056478405315 91951 0.05656162735025291
训练  58 41.78446115288221 94345 0.053733545982740265
训练  59 52.734844491302056 96242 0.05104686868360325
训练  60 93.24487895716946 97316 0.04849452524942309
训练  61 20.63887742468015 102162 0.04606979898695193
训练  62 51.57503867973182 104101 0.04376630903760433
训练  63 41.05993431855501 106537 0.041577993585724116
训练  64 41.47678275290215 108949 0.03949909390643791
训练  65 59.63983303518187 110626 0.03752413921111601
训练  66 54.69983597594314 112455 0.03564793225056021
训练  67 55.56055555555555 114255 0.0338655356380322
训练  68 55.94686800894855 116043 0.032172258856130585
训练  69 30.25249470819474 119350 0.030563645913324056
训练  70 46.88331771321462 121484 0.029035463617657853
训练  71 39.18847962382445 124036 0.027583690436774957
训练  72 20.985312631137223 128802 0.02620450591493621
训练  73 28.960046323103647 132256 0.0248942806191894
训练  74 44.8163082437276 134488 0.023649566588229927
训练  75 71.07818052594172 135895 0.022467088258818428
训练  76 39.16170712607674 138449 0.021343733845877507
训练  77 59.548809523809524 140129 0.02027654715358363
训练  78 84.97451146983857 141306 0.019262719795904448
训练  79 71.00780695528744 142715 0.018299583806109226
训练  80 40.46804207119741 145187 0.017384604615803764
训练  81 28.20360970107163 148733 0.016515374385013576
训练  82 41.094905505341 151167 0.015689605665762895
训练  83 42.99140524280189 153494 0.01490512538247475
训练  84 73.59602649006622 154853 0.014159869113351011
训练  85 42.6682302771855 157198 0.01345187565768346
训练  86 75.47698113207547 158523 0.012779281874799287
训练  87 22.41326759300762 162985 0.012140317781059323
训练  88 59.22498519834222 164674 0.011533301892006355
训练  89 89.9568345323741 165786 0.010956636797406038
训练  90 40.239340305711984 168272 0.010408804957535735
训练  91 46.054788213627994 170444 0.009888364709658948
训练  92 25.233543505674653 174409 0.009393946474176
训练  93 34.51932367149758 177307 0.0089242491504672
训练  94 52.46303093864709 179214 0.008478036692943839
训练  95 66.84893048128342 180710 0.008054134858296647
训练  96 61.092241905925476 182347 0.0076514281153818135
训练  97 31.428346951602766 185529 0.0072688567096127225
训练  98 67.60067567567567 187009 0.006905413874132086
训练  99 66.53457446808511 188513 0.006560143180425482
训练  100 256.1074168797954 188904 0.0062321360214042075
训练  101 81.57328990228012 190132 0.005920529220333997
训练  102 32.315670436187396 193227 0.0056245027593172965
训练  103 42.82662671232877 195563 0.005343277621351432
训练  104 85.64554794520548 196731 0.0050761137402838595
训练  105 81.55338223308884 197958 0.004822308053269666
训练  106 29.83472553699284 201310 0.004581192650606183
训练  107 44.425144380275434 203561 0.0043521330180758735
训练  108 217.11062906724513 204022 0.0041345263671720795
训练  109 31.45125786163522 207202 0.003927800048813475
训练  110 66.50895819508958 208709 0.0037314100463728015
训练  111 146.76099706744867 209391 0.0035448395440541612
训练  112 40.0188075230092 211890 0.003367597566851453
训练  113 66.23973509933775 213400 0.00319921768850888
训练  114 203.380081300813 213892 0.003039256804083436
训练  115 78.2660406885759 215170 0.0028872939638792637
训练  116 30.187745246000603 218483 0.0027429292656853004
训练  117 41.75532359081419 220878 0.0026057828024010354
训练  118 54.9934029686641 222697 0.0024754936622809836
训练  119 104.18002081165453 223658 0.002351718979166934
训练  120 85.18553191489362 224833 0.0022341330302085875
训练  121 77.42337461300309 226125 0.002122426378698158
训练  122 94.19567262464723 227188 0.0020163050597632503
训练  123 78.80693459416864 228457 0.0019154898067750877
训练  124 121.16949152542372 229283 0.0018197153164363333
训练  125 69.90146750524109 230714 0.0017287295506145165
训练  126 46.58453656264555 232861 0.0016422930730837905
训练  127 27.734886300610093 236467 0.0015601784194296008
训练  128 93.55607476635514 237537 0.0014821694984581207
训练  129 109.21834061135371 238453 0.0014080610235352145
训练  130 38.07004187285877 241080 0.0013376579723584536
训练  131 95.77703349282297 242125 0.0012707750737405309
训练  132 73.40058694057227 243488 0.0012072363200535043
训练  133 111.39198218262806 244386 0.001146874504050829
训练  134 122.5483476132191 245203 0.0010895307788482875
训练  135 92.00183992640294 246290 0.001035054239905873
训练  136 134.84118438761777 247033 0.0009833015279105794
训练  137 42.77126977340744 249372 0.0009341364515150504
训练  138 74.98053892215569 250708 0.0008874296289392978
训练  139 41.6609745939192 253109 0.0008430581474923328
训练  140 39.21843137254902 255659 0.0008009052401177162
训练  141 40.21270607157218 258146 0.0007608599781118304
训练  142 48.900244498777504 260191 0.0007228169792062388
训练  143 48.404162633107454 262257 0.0006866761302459269
训练  144 106.25371549893843 263199 0.0006523423237336305
训练  145 90.06390639063906 264310 0.0006197252075469489
训练  146 83.55889724310777 265507 0.0005887389471696014
训练  147 98.7670286278381 266520 0.0005593019998111214
训练  148 218.8886462882096 266978 0.0005313368998205653
训练  149 56.0762331838565 268762 0.0005047700548295369
训练  150 45.08832807570978 270981 0.00047953155208806006
训练  151 66.18981481481481 272493 0.000455554974483657
训练  152 63.69490445859873 274063 0.0004327772257594741
训练  153 55.44376731301939 275868 0.0004111383644715004
训练  154 87.90773286467487 277006 0.00039058144624792536
训练  155 36.66092375366569 279734 0.0003710523739355291
训练  156 73.43465491923642 281096 0.00035249975523875265
训练  157 82.79056291390728 282304 0.000334874767476815
训练  158 16.97233536999321 288196 0.00031813102910297424
训练  159 28.424836601307188 291715 0.0003022244776478255
训练  160 80.0048 292965 0.0002871132537654342
训练  161 58.255096097845076 294682 0.00027275759107716247
训练  162 120.63088057901085 295511 0.00025911971152330434
训练  163 25.32387946315523 299460 0.00024616372594713913
训练  164 93.37628384687208 300531 0.00023385553964978215
训练  165 93.1927374301676 301605 0.00022216276266729304
训练  166 43.82479194042926 303888 0.00021105462453392839
训练  167 54.16242555495398 305735 0.00020050189330723196
训练  168 81.4486970684039 306963 0.00019047679864187035
训练  169 86.11521926053311 308126 0.00018095295870977683
训练  170 51.33282051282051 310076 0.00017190531077428798
训练  171 65.17263843648209 311611 0.00016331004523557357
训练  172 78.64150943396227 312883 0.00015514454297379488
训练  173 52.53991596638655 314787 0.00014738731582510513
训练  174 48.976003917727716 316829 0.00014001795003384986
训练  175 90.5361663652803 317935 0.00013301705253215737
训练  176 39.13575899843506 320491 0.0001263661999055495
训练  177 61.82396541074738 322110 0.000120047889910272
训练  178 84.71549534292971 323291 0.0001140454954147584
训练  179 93.40242763772176 324362 0.00010834322064402047
训练  180 49.82610861983059 326369 0.00010292605961181944
训练  181 130.70104438642298 327135 9.777975663122846e-05
训练  182 123.94059405940594 327943 9.289076879966704e-05
训练  183 36.53781512605042 330680 8.824623035968368e-05
训练  184 69.73031358885018 332115 8.383391884169949e-05
训练  185 23.253602975360298 336417 7.96422228996145e-05
训练  186 18.544409419618024 341810 7.566011175463378e-05
训练  187 24.491309669522643 345895 7.187710616690208e-05
训练  188 33.14683460391117 348912 6.828325085855697e-05
训练  189 49.31148348940365 350941 6.486908831562912e-05
训练  190 43.50260869565217 353241 6.162563389984766e-05
训练  191 118.65599051008304 354084 5.8544352204855274e-05
训练  192 54.13095238095238 355932 5.561713459461251e-05
训练  193 74.74962630792227 357270 5.283627786488188e-05
训练  194 34.499655172413796 360170 5.019446397163778e-05
训练  195 42.77929854576561 362508 4.768474077305589e-05
训练  196 58.83352941176471 364208 4.53005037344031e-05

运行效果：(第一次运行)

训练  1 22.3467381590706 4475 1.0
训练  2 24.759158415841583 8515 0.95
训练  3 28.22234762979684 12059 0.9025
训练  4 32.075368826170624 15177 0.8573749999999999
训练  5 37.41676019453797 17850 0.8145062499999999
训练  6 42.08245687841817 20227 0.7737809374999999
训练  7 50.10766149223836 22224 0.7350918906249998
训练  8 59.31672597864769 23910 0.6983372960937497
训练  9 66.97456492637215 25404 0.6634204312890623
训练  10 82.64710743801653 26614 0.6302494097246091
训练  11 96.19134615384615 27654 0.5987369392383786
训练  12 115.72601156069364 28519 0.5688000922764596
训练  13 143.78591954022988 29215 0.5403600876626365
训练  14 143.1874105865522 29914 0.5133420832795047
训练  15 157.5676100628931 30550 0.48767497911552943
训练  16 201.19277108433735 31048 0.46329123015975293
训练  17 203.24696356275302 31542 0.44012666865176525
训练  18 207.39337474120083 32025 0.41812033521917696
训练  19 119.44391408114558 32863 0.3972143184582181
训练  20 192.05374280230328 33384 0.37735360253530714
训练  21 136.67349726775956 34116 0.35848592240854177
训练  22 167.03338898163605 34715 0.34056162628811465
训练  23 122.72269938650307 35530 0.3235335449737089
训练  24 145.94460641399417 36216 0.30735686772502346
训练  25 73.0255288110868 37587 0.2919890243387723
训练  26 98.70710059171597 38601 0.27738957312183365
训练  27 87.13501742160278 39749 0.263520094465742
训练  28 138.9013888888889 40469 0.25034408974245487
训练  29 120.07563025210084 41302 0.2378268852553321
训练  30 87.69938650306749 42443 0.2259355409925655
训练  31 82.34156378600824 43658 0.2146387639429372
训练  32 97.01453488372093 44690 0.20390682574579033
训练  33 73.11769005847954 46058 0.1937114844585008
训练  34 92.27121771217712 47142 0.18402591023557577
训练  35 78.55852317360565 48415 0.17482461472379698
训练  36 105.78752642706131 49361 0.16608338398760714
训练  37 64.75533980582524 50906 0.15777921478822676
训练  38 61.46773202212661 52533 0.14989025404881542
训练  39 51.99064449064449 54457 0.14239574134637464
训练  40 75.46832579185521 55783 0.1352759542790559
训练  41 76.4140565317036 57092 0.1285121565651031
训练  42 60.428398791540786 58747 0.12208654873684793
训练  43 49.19813176007866 60781 0.11598222130000553
训练  44 72.63352685050798 62159 0.11018311023500525
训练  45 88.06602112676056 63295 0.10467395472325498
训练  46 127.13722998729352 64082 0.09944025698709223
训练  47 54.02159827213823 65934 0.09446824413773762
训练  48 74.45386904761905 67278 0.08974483193085074
训练  49 106.31137088204038 68219 0.0852575903343082
训练  50 110.8869179600887 69121 0.08099471081759278
训练  51 221.11920529801324 69574 0.07694497527671314
训练  52 87.00521739130434 70724 0.07309772651287748
训练  53 60.325693606755124 72382 0.0694428401872336
训练  54 32.73518821603928 75437 0.0659706981778719
训练  55 11.840042623727209 83883 0.0626721632689783
训练  56 36.5932699341624 86617 0.059538555105529384
训练  57 29.566361217853977 90000 0.05656162735025291
训练  58 52.359162303664924 91910 0.053733545982740265
训练  59 25.63268442622951 95814 0.05104686868360325

运行效果就是ELM与RL结合后，RL算法的性能极差，连最基本的RL算法性能都达不到，可以说RL算法是难以与ELM结合的，换句话说，由此推论，RL算法是难以与单隐藏层的非BP算法进行结合的，虽然没有做过其他的实验，但是从这里可以推论，RL结合罗杰斯回归，结合ELM，结合宽度学习（宽度神经网络，单隐藏层神经网络），都不会有什么太好的算法性能。

像ELM这样的单隐藏层算法，尤其是ELM这种快速训练的神经网络训练方法，这种训练方式会导致模型不对上一次训练时的训练参数进行基于。

ELM算法每次的训练都是根据单次训练所收集的数据集进行拟合，该种方式对于不依赖时间维度的监督学习和无监督学习的学习范式还是有着一定的适用范围的，但是强化学习算法/强化学习问题需要根据上一次训练的模型对环境进行探索和利用，在不断的迭代过程中推进数据采集的样本发布到累积奖励和更高的方向，而像ELM算法这样的训练方式通过单次的训练进行快速拟合所有当下分布下的样本，虽然可以有极快的训练速度，但是对于上次训练的参数是不进行记忆的，因此ELM做不到推进采样分布向累积奖励和更高的方向发展。

由上面的训练性能数据可以知道，ELM算法虽然理论上可以通过对单次训练数据的拟合与epsilon-greedy算法进行结合，在RL算法训练过程中通过逐渐的增加上次ELM训练出的策略网络进行样本采样，并且在episode reward在200以下的时候取得了一定的表现，但是当逐渐增加ELM算法训练出的策略模型的生成样本的占比，再训练出的ELM算法模型性能开始下降。

也就是说，ELM算法模型训练出的RL模型不具备时序记忆的能力，这种情况下的RL模型使用ELM进行策略函数的拟合，其模型性能完全取决于上次采样时的样本分布情况。而且，一个episode reward较高的样本集并一定能训练出下一个episode reward较高的ELM RL策略，elm rl的策略的性能表现依赖于较好的episode reward较高的样本集以及分布性较好的随机样本，也就是说上面的代码中，当epsilon-greedy的探索概率下降到一定程度时及时采样到的episode reward较高的样本集也会导致下一次elm训练出的策略性能较差。

从这里可以看到，elm这种不具备多次训练记忆的算法模型难以应用于强化学习这种需要不断推进采样样本分布趋向于更优样本采样分布的算法的。