macos（M4）上跑强化学习 - PyTorch LunarLander-v3 Stable-Baselines3

安装python环境

• 安装miniforge3
  正确安装后，就有了conda命令

配置python环境

# 创建环境，python3.11
conda create -n rl-env python=3.11
conda activate rl-env
# 关闭用这个
conda deactivate

报错‘CondaError: Run 'conda init' before 'conda activate'’ google 一下，init命令需要参数

使用PyTorch进行训练，所以需要安装PyTorch

pip3 install torch torchvision torchaudio

创建文件进行测试，是否安装成功，是否是M系列芯片

# test_env.py
import torch
print(torch.backends.mps.is_available()) # True = GPU可用
print(torch.backends.mps.is_built())

输出如图：

小模型完全没必要使用gpu，直接使用cpu可能速度更快。

You are trying to run PPO on the GPU, but it is primarily intended to run on the CPU when not using a CNN policy (you are using ActorCriticPolicy which should be a MlpPolicy). See https://github.com/DLR-RM/stable-baselines3/issues/1245 for more info. You can pass device='cpu' or export CUDA_VISIBLE_DEVICES= to force using the CPU.Note: The model will train, but the GPU utilization will be poor and the training might take longer than on CPU.

安装 gymnasium

pip install gymnasium
# 或
pip install "gymnasium[box2d]"
# 我用的这个带box2d的

创建文件进行测试，是否安装成功。参考自：https://zhuanlan.zhihu.com/p/694923486

# test_gymnasium.py

import gymnasium as gym
env = gym.make("LunarLander-v3", render_mode="human")
observation, info = env.reset()

for _ in range(100):
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)
    print('---')
    print("action: " + str(action))
    print("observation: " + str(observation))
    print("reward: " + str(reward))
    print("terminated: " + str(terminated))
    print("truncated: " + str(truncated))
    print("info: " + str(info))

    if terminated or truncated:
        observation, info = env.reset()

env.close()

生成如图：

之后将使用LunarLander-v3 这个环境进行模型训练。

使用Stable-Baselines3进行训练

使用Stable-Baselines3先让流程跑通

pip install 'stable-baselines3[extra]'

写个脚本开始训练进行测试

# test_sb3.py
import gymnasium as gym
import torch
from stable_baselines3 import PPO

# 检测设备
device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")
print("device: ", device)

# 创建环境
env = gym.make("LunarLander-v3")

# 简单PPO测试训练
model = PPO(
    "MlpPolicy", 
    env, 
    verbose=1, 
    device=device, 
    tensorboard_log="./ppo_logs/"  # 日志存放目录
    )
model.learn(
    total_timesteps=10000,
    tb_log_name="test_01",    # 本次实验名称
    )

# 测试运行
obs, _ = env.reset()
for _ in range(1000):
    action, _states = model.predict(obs)
    obs, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        obs, _ = env.reset()
env.close()

# 保存模型
model.save("ppo_LunarLander")
print("done")

运行结果部分截图：

查看训练效果的方法，使用tensorboard

tensorboard --logdir ./ppo_logs/

运行后，则可以针对日志文件进行查看，打开 http://localhost:6006/
如下图：

具体这个训练效果图要如何分析，先跳过，先尝试把流程跑完。

使用训练的模型

写个脚本来使用模型，看看效果：

# test_use_ppo.py
import gymnasium as gym
import torch
from stable_baselines3 import PPO

# 检测设备
device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")
print("device: ", device)

model = PPO.load("ppo_LunarLander", device=device)

env = gym.make("LunarLander-v3", render_mode="human")
obs, _ = env.reset()

for _ in range(1000):
    # 模型预测动作
    action, _states = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, info = env.step(action)
    
    if terminated or truncated:
        obs, _ = env.reset()

env.close()

效果如下图所示：

至此，训练就完成了。
剩下就可以一步一步针对全流程中的内容进行拆解学习。