RL——METRA: Scalable Unsupervised RL with Metric-Aware Abstraction

RL——METRA: Scalable Unsupervised RL with Metric-Aware Abstraction

作者：凯鲁嘎吉 - 博客园 http://www.cnblogs.com/kailugaji/

        近期，阅读了一篇发表在ICLR2024上的文章《METRA: Scalable Unsupervised RL with Metric-Aware Abstraction》，本博文从生成对抗网络的变种Wasserstein GAN的角度出发，来理解METRA的提出动机。首先介绍一些基础知识，包括：KL散度、JS散度、Wasserstein距离、Lipschitz条件、信息熵、联合熵、条件熵、前向与反向互信息、相对熵、Jensen不等式。然后通过Wasserstein GAN与METRA左右对比来理解METRA的由来。进一步，详细解读了METRA方法的公式推导过程、算法流程、直观理解以及与DIAYN、DADS和CIC方法之间的联系。

参考资料：

[1]      Park S, Rybkin O, Levine S. METRA: Scalable Unsupervised RL with Metric-Aware Abstraction. In International Conference on Learning Representations (ICLR), 2024.

[2]      平均场理论：凯鲁嘎吉 - https://www.cnblogs.com/kailugaji/p/10692797.html、https://www.cnblogs.com/kailugaji/p/12463966.html

[3]      生成对抗网络(GAN与W-GAN)：凯鲁嘎吉 - https://www.cnblogs.com/kailugaji/p/15352841.html

[4]      非对称度量即拟度量的定义：凯鲁嘎吉 - https://www.cnblogs.com/kailugaji/p/19210601

[5]      DIAYN：Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations (ICLR), 2019.

[6]      DADS：Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics aware unsupervised discovery of skills. In International Conference on Learning Representations (ICLR), 2020.

[7]      CIC：Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and P. Abbeel. Unsupervised reinforcement learning with contrastive intrinsic control. In Neural Information Processing Systems (NeurIPS), 2022.