Action Chunking Transformer要点分析

1、机器人干活，肯定要从模仿人类开始。现实情况是人上一百，形形色色，林子大了什么样的鸟都有。同样一件事，不同的人干活的方法、风格肯定不一样，就比如开车：刚拿到驾照的新手上路肯定战战兢兢，速度压到最低，生怕遇到突发情况来不及刹车出事！但有多年驾龄的老鸟就不一样了：在交规和安全都允许的情况下肯定是速度飙到最快，肯定是不会磨磨唧唧前进的！所以机器人干活的核心要点之一：学会人类action的风格！具体该怎么做了？

　　想想之前生图的思路：最早的方案是AE，也就是auto encoder，核心是通过神经网络把原始图片encoder到latent Z向量，这个latent Z向量相当于“浓缩”了整张image的精华，所以人类action的风格是不是也能用类似的方法“浓缩”到一个latent Z向量了？ACT就是这么干的！整个思路图示如下：

Training Reparametrization Input Ground-truth label z ~ Ν (μ,σ2), E~N(0,1) Step 1: sample data (Observation) (Action) z = μ +σ . ε Demo sample Dataset 4 RGB images joints 4× (480×640×3) (14) action sequence (k x 14) batch size Step 2: infer z z_mean (32) Z z_std (32) sample with linear layer 4 reparametrization 512 -> 64 joints embedded joints transformer Weight matrix [CLS] (14) linear layer 2 (512) (512) encoder (512) 14 -> 512 4x self-attention blocks ... + action sequence embedded action sequence (k x 14) linear layer 1 Sinusoidal Position Embedding (k × 512) [CLS] embedded embedded action sequence 14 -> 512 (512) joints (512) (k x 512)
  •  step 1 sample data：
    • input(observation)：观察数据，包括摄像头采集的图像，还有物理机器臂的状态joints
    • ground truth label(action)：正确的action trajectory
  •  step 2 infer Z：
    • action trajectory本质是时序数据，所以用transformer encoder来编码生成latent Z是再合适不过的了！
      • token内部做cross attention，让token之间做交互，把全局信息融合到某些token里面；
      • [CLS]本身没任何意义，目的是接受全局信息；最终取cls token对应的输出即可。已经包含了全局其他token的信息
    • Z要求服从正太分布，避免Z过于离散，保持连续性

  • z主要不是为了让机器人“凭空产生随机动作”，而是帮助模型吸收人类示范中的动作变化和多模态性。

2、第一步学习了人类的aciton风格后，最核心的就是预测action了！action本质是个时序数据，每次可以只预测1个action，也可以预测多个action，ACT的名字既然是action chunking，肯定是预测多个啦！这样做的好处是啥了？

  • Single step：可能产生累计误差；无法建模有时间需求的场景，比如热油，需要等一段时间，这类是non markov的，但预测的时候只能预测一步 

  • Action chunking：减少compounding error（误差累积），同时预测未来K步，可以应对non markov的场景

Execute 0 1 2 3 4 5 6 7 t=0 x [0.5, 0.3, 0.2, 0.1] = t=1 Weight formula Wi = exp(-m*i) t=2 (Older action has bigger weight) t=3 Predict
 3、往事具备，开始训练啦！大致的思路和流程如下：
ACT Model 4 RGB images Training 4× (480×640×3) Step 3: predict action sequence predicted action sequence 1 15×20×728 ... ... flatten 2 300×728 transformer transformer Ziploc linear layer 5 728 -> 512 encoder decoder 3 4x self-attention blocks 7x cross-attention blocks 300×512 + Sinusoidal PosEmb 4 ... ... .. ResNet18 cam 1 cam 4 position embeddings (fixed) +Sinusoidal PosEmb linear layer 6 linear layer 7 14 -> 512 32 -> 512 (k x 512) joints
  • 4个cam的原始图像通过各种linear、flattern等变换，叠加posEmb进入transformer
  • 物理机械臂的状态joints也进入transformer
  • 上面第一步CVAE得到的action trajectory风格的latent Z向量！
　　上述3路数据一并输入transformer encoder，内部经过层层的embedding和FFN后，互相充分融合对方的信息，然后进入transformer decoder生成action chunking！train的架构确定后，就要确定loss函数了。
  • 因为最终的目的是输出正确的action chunking，所以最重要的loss肯定是预测action chunking和ground truhe的action chunking做比对，看看预测的准不准！
  • latent Z也不能忽视：Z的分布最好是小范围内的正太分布，避免过于分散，导致采样的时候结果离散！
Algorithm 1 ACT Training Lreconstruct 1: Given: Demo dataset D, chunk size k, weight B. Align predicted actions with 2: Let at, Ot represent action and observation at timestep t, Öt ground-truth actions. represent ot without image observations. 3: Initialize encoder q (z|at:+k, 0+) Principle mine - List, at ttkED log Te (at:+k|St) 4: Initialize decoder To (ât:+k|Ot, z) 5: for iteration n = 1, 2, ... do Implement MSE, L1 Loss, etc. 6: Sample Ot, at:++k from D 7: Sample z from qø (z|at:t+k, Öt) Lregularization 8: Predict at:t.+k from Te (ât:+k Ot, 2) 9: Creconst = MSE (at:+k, at:t+k) 10: Lreg = DKL(qo(z|at:t+k, Ot) || N(0, I)) Align the z distribution to Gaussian. 11: Update 0, o with ADAM and L = Lreconst + BLreg q(z|x) ~ N(u, diag(o2)) d 112 DKL(q(z|x) || p(z)) = (1+log(o) - 12 - 03) i=1 N(0,I)
 4、模型训练好后就是实际测试了！和train比，是不知道action chunking的，所以Z就只能用0替代了！
  • Z服从N(0,1)的正态分布，0的概率是最大的，所以干脆就直接设置为0了！
Testing incoming observations predicted action sequence 1 15×20×728 ... .. flatten 2 300×728 transformer transformer Ziploc linear layer 5 728 -> 512 encoder decoder 3 4x self-attention blocks 7x cross-attention blocks 300×512 Sinusoidal + + PosEmb 4 ... ... ResNet18 cam 1 cam 4 position embeddings (fixed) +Sinusoidal PosEmb linear layer 6 Linear layer 7 14 -> 512 32 -> 512 (k × 512) 10 joints joints
 整体模块的展示如下：
  • CVAE的时候condition中是没用image的，可能是为了提升速度，并且只用于train！
  • train阶段condition加上了image和机械臂状态的joints
ACT Model 9 V (u, O) 9 Only used in training action sequence Z style variable 1 . . . 2 transformer transformer transformer encoder encoder decoder 3 1 + 4 -A- . [CLS] joints ,action sequence + PosEmb 480X640X3 CNN +PosEmb cam 1 cam 4 joints 2 position embeddings (fixed) Condition Condition w/o images 1 1. Model could focus on the increment, don't need to predict Set to zero during inference Just for faster training absolute joint pose directly from image. 2. Make the model aware of minor systematic differences between the leader and follower arms.
 最终评测结果：
  • success成功率：chunking size=100左右最高！后续轻微下降，原因可能是输出的chunk size过长，越来越接近 open-loop control，机器人失去 reactive behavior，缺少真实的反馈！
  • Temporal Ensemble：对 overlapping action chunks 做加权平均，提升约3.3%，说明有用，但好像不明显
  • CVAE：script data差异不大， 但human data相差33%，非常明显！从 scripted data 切换到 human data 后，所有方法性能下降，因为 human demonstrations 有 stochasticity 和 multi-modality，说人话就是human的action多种多样，比如拿杯子，有人喜欢慢慢靠近后调整手腕抓杯子，有人喜欢快速直接抓杯子，目的都是一样的，但是有不同的风格！latent Z就是为了捕捉这些风格！
  • high control frequency：因为这些任务都是“毫米级精度 + 闭环视觉反馈”，频率过低会导致采样少，action的时候出现偏差会来不及修正的，进而导致累计误差越来越大！降低频率会造成约62%的 completion-time slowdown
Ablation CVAE objective (DKL loss) Temporal Ensemble Important for learn from High control frequency Chunking is key somehow improves human demonstrations is necessary 60 -1% 70 35 - 50 50 +3.3% 60 30 45 37.5 37.5 50 25 +4% 40 20 25 25 30 -20% 30 15 duration (sec) success (%) success (%) success (%) 12.5 12.5 15 20 10 7 -33.3% 0 10 5 0 0 fully-closed-loop 10 100 200 400 Ours BC-ConvMLP VINN Scripted Data Human Data K fully-open-loop 5Hz 50Hz 5Hz 50Hz - Ours BC-ConvMLP - VINN no TE with TE With CVAE No CVAE thread zip tie unstack cups (a) (b) (c) (d) Fig. 8: (a) We augment two baselines with action chunking, with different values of chunk size k on the x-axis, and success rate on the y-axis. Both methods significantly benefit from action chunking, suggesting that it is a generally useful technique. (b) Temporal Ensemble (TE) improves our method and BC-ConvMLP, while hurting VINN. (c) We compare with and without the CVAE training, showing that it is crucial when learning from human data. (d) We plot the distribution of task completion time in our user study, where we task participants to perform two tasks, at 5Hz or 50Hz teleoperation frequency. Lowering the frequency results in a 62% slowdown in completion time.
  
                    ACT
                     │
        ┌────────────┼─────────────┐
        │            │             │
        ↓            ↓             ↓
   Action Chunk    CVAE      Temporal Ensemble
        │            │             │
        │            │             │
   减少长期误差   处理人类       平滑重叠
                  示范多样性       action
        │            │             │
        └────────────┼─────────────┘
                     ↓
                高频闭环执行
                     ↓
                  Robot

 

 
posted @ 2026-08-19 22:32  第七子007  阅读(3)  评论(0)    收藏  举报