【机翻】李飞飞：世界模型的功能分类
A Functional Taxonomy of World Models 世界模型的功能分类
原文链接：https://drfeifei.substack.com/p/a-functional-taxonomy-of-world-models
Renderers, Simulators, Planners, and the Loop That Connects Them 渲染器、模拟器、规划器以及连接它们的循环
Jun 04, 2026 2026年6月4日
“The world is everything that is the case.”
“世界就是一切事实。”
— Ludwig Wittgenstein, Tractatus Logico-Philosophicus, 1921
——路德维希·维特根斯坦， 《逻辑哲学论》 ，1921
The world is not made of words.
世界并非由言语构成。
In an earlier essay, we argued that spatial intelligence is AI’s next frontier and that world models are the path to it. Here, the World Labs team and I want to go one level deeper: of the many things now being built and called ‘world models,’ which functional pieces actually compose that capacity — and what is each one for?
在之前的一篇文章中 ，我们论证了空间智能是人工智能的下一个前沿领域，而世界模型是通往这一领域的途径。在这里，我和世界实验室团队希望更深入地探讨这个问题：在目前正在构建并被称为“世界模型”的众多事物中，究竟哪些功能组件构成了这种能力——而每个组件的用途又是什么？
Language models have given machines an extraordinary command of concepts, vocabulary, and reasoning, but the physical world, virtual or real, runs on a different substrate. Where language models learn the statistical structure of text, world models learn the statistical structure of space and time: how light falls on a surface, how a garden looks from an angle no camera has captured, how objects respond to force and follow the laws of physics.
语言模型赋予了机器非凡的概念、词汇和推理能力，但物理世界，无论是虚拟的还是真实的，都运行在不同的底层架构之上。语言模型学习文本的统计结构，而世界模型学习时空的统计结构：光线如何照射在物体表面，从相机未曾捕捉到的角度看花园是什么样子，物体如何响应力并遵循物理定律。
That makes “world model” one of the most important and most overloaded terms in AI today. Computer vision, robotics, reinforcement learning, and generative AI each claim to be building world models, and each means something quite different. A video model that produces gorgeous but physically impossible flames, a language model improvising a playable game, and a physics engine that faithfully simulates combustion all go by the same name.
这使得“世界模型”成为当今人工智能领域最重要也最被滥用的术语之一。计算机视觉、机器人技术、强化学习和生成式人工智能都声称在构建世界模型，但它们各自的含义却截然不同。一个能生成华丽但物理上不可能的火焰的视频模型 、一个能即兴创作可玩游戏的语言模型 ，以及一个能忠实模拟燃烧过程的物理引擎， 都使用着同一个名称。
The ancient Greeks could never agree on what the world was made of, whether fire, water, or indivisible atoms, because “world” was never a single thing. It was always a stand-in for whatever totality a given thinker needed to reason about. AI has inherited the same problem, at exactly the moment when the field needs precision.
古希腊人始终无法就世界的构成达成共识，究竟是火、水，还是不可分割的原子，因为“世界”从来都不是一个单一的实体。它始终是特定思想家需要思考的任何整体的代名词。人工智能也继承了同样的问题，而此时，这个领域恰恰最需要的是精准性。
The loop beneath the taxonomy 分类学之下的循环
Cutting through that confusion starts with a diagram older than any of the technology in question. Reinforcement learning textbooks, including the canonical Sutton and Barto, have used a version of the same picture for decades to describe how an agent interacts with a world. The formal name for this picture is the partially observable Markov decision process, or POMDP, and the original definition of the term “world model” belongs to that tradition.
要理清这些困惑，首先要从一张比相关技术本身还要古老的图表入手。强化学习教材，包括经典的萨顿和巴托的著作，几十年来一直使用类似的图表来描述智能体如何与世界互动。这张图表的正式名称是部分可观测马尔可夫决策过程（POMDP），而“世界模型”一词的最初定义也源于此。
An agent, which can be a person, a robot, or a software system, takes actions. Those actions affect the state of the world. The agent never sees the state directly. What reaches the agent are observations: the photons that fall on a retina, the readings from a sensor, and the pixels in a video frame. New observations inform new actions, and the loop continues.
智能体（可以是人、机器人或软件系统）会采取行动。这些行动会影响世界的状态。智能体本身并不直接感知世界的状态。智能体接收到的是观测数据：落在视网膜上的光子、传感器的读数以及视频帧中的像素。新的观测数据会指导新的行动，如此循环往复。
The word “state” needs unpacking, because the meaning shifts from field to field. This is not the chemist’s state, the difference between solid, liquid, and gas. This is the physicist’s and roboticist’s state: a complete description of what is happening in the world at a given moment, including every object, every position, every velocity, every property. State is the underlying reality of the world; complete in principle, but never directly visible to any agent inside it. Observations are an agent’s partial view of that reality. Actions are what the agent does in response.
“状态”一词需要进一步阐释，因为它的含义会因领域而异。这里所说的状态并非化学家所说的状态，即固态、液态和气态之间的区别。这里所说的状态是物理学家和机器人学家所说的状态：是对某一特定时刻世界所发生的一切的完整描述，包括每一个物体、每一个位置、每一个速度和每一个属性。状态是世界的基本实在；原则上是完整的，但身处其中的任何主体都无法直接感知到它。观察是主体对这种实在的局部感知。行动则是主体对这种实在的回应。
This loop — agent to action to state to observation and back — is the structure that gave the modern term “world model” its technical meaning. The phrase itself is older, traced to Kenneth Craik’s 1943 proposal that minds reason by running “small-scale models” of reality, and carried into neural networks by the late 1980s and early 1990s. And the loop also explains what people mean by the term today. The different things now being called world models are in fact different projections of this same loop. Each one outputs a different piece of it.
这个循环——主体→行动→状态→观察→返回——赋予了现代术语“世界模型”以技术意义。这个术语本身出现得更早，可以追溯到肯尼斯·克雷克1943年提出的理论：思维通过运行现实的“小规模模型”来进行推理，并在20世纪80年代末90年代初被应用于神经网络。这个循环也解释了人们今天对这个术语的理解。现在被称为世界模型的各种事物，实际上是同一个循环的不同投射。每个模型都输出了循环中的不同部分。
Three functions of a world model 世界模型的三个功能
The first kind of world model is a renderer. A renderer outputs observations in the form of pixels meant for human eyes, and the quality that matters most is visual fidelity. A video model that turns a text prompt into a cinematic drone shot is a renderer. So is an interactive system like Google’s Genie 3, or World Labs’ own RTFM, where the model generates frames in real time conditioned on user input. The model carries no explicit understanding of three-dimensional structure. It produces what a viewer would see, not what is. The buildings in the drone shot may look flawless from above, but try to drive through the city below and they fall apart.
第一种世界模型是渲染器。 渲染器以像素的形式输出观测结果，供人眼观看，其最重要的质量是视觉保真度。例如， 将文本提示转化为电影级无人机航拍镜头的视频模型就是一个渲染器。像谷歌的 Genie 3 或 World Labs 的 RTFM 这样的交互式系统也是如此 ，这些模型会根据用户输入实时生成帧。模型本身并不具备对三维结构的明确理解。它生成的是观看者所看到的，而不是实际存在的。无人机航拍镜头中的建筑物从空中俯瞰可能完美无瑕，但如果你开车穿过下面的城市，就会发现它们摇摇欲坠。
The second kind is a simulator. A simulator outputs state: a geometrically, physically or dynamically faithful representation of the world that humans and computer programs can both compute on and interact with. Where the renderer’s contract is purely visual, the simulator’s contract is structural, demanding geometry that holds up under inspection, physics that respects Newton’s laws, and dynamics that behave the way the world needs to behave given the laws of physics. A simulator serves two consumers at once. Human professionals such as architects, designers, filmmakers, and game developers need accuracy beyond visual plausibility. Computer programs such as reinforcement learning agents, robot controllers, and autonomous vehicles use simulators as training grounds where they can interact with the world at scale, testing scenarios that would be dangerous, expensive, or impossible to run in reality.
第二种是模拟器。模拟器输出状态：一种在几何、物理或动力学上忠实还原世界的表示，人类和计算机程序都可以对其进行计算和交互。 渲染器的契约纯粹是视觉上的，而模拟器的契约则是结构性的，它要求几何结构经得起检验，物理定律符合牛顿定律，动力学行为符合物理定律所要求的世界运行方式。模拟器同时服务于两个用户群体。建筑师、设计师、电影制作人和游戏开发者等专业人士需要超越视觉逼真性的精确度。强化学习智能体、机器人控制器和自动驾驶汽车等计算机程序则将模拟器作为训练场，在大规模环境下与世界交互，测试那些在现实中危险、昂贵或不可能运行的场景。
The third kind is a planner. A planner outputs actions. Given an observation and a goal, a planner answers the question of what the agent should do next. This is, in many ways, the inverse of the renderer. Where a renderer takes actions as input and produces observations, a planner takes observations as input and produces actions, closing the perception-action loop. Vision-Language-Action models, model-based systems, and the new wave of World Action Models are all attempts at planners: systems that can decide what a robot should do in an unstructured world.
第三种是规划器。规划器输出动作。 给定观测结果和目标，规划器回答智能体下一步应该做什么。这在很多方面与渲染器正好相反。渲染器以动作作为输入并生成观测结果，而规划器以观测结果作为输入并生成动作，从而形成感知-动作的闭环。视觉-语言-动作模型、基于模型的系统以及新一代的世界动作模型都是规划器的尝试：这些系统能够决定机器人在非结构化世界中应该做什么。
These three categories describe most of what is actually shipping today, and the distinction between them is useful in practice. The categories are not, however, fundamentally separate. The same underlying knowledge of how the world works—geometry, physics, dynamics—sits beneath all of them. A model that can render a cup from any angle ought, in principle, to be able to simulate what happens when the cup is pushed and plan a hand to pick the cup up. Increasingly, the most interesting research deliberately blurs the boundaries between the three.
这三大类涵盖了当今市场上绝大多数的产品，它们之间的区别在实践中也很有用。然而，这三类产品并非本质上截然分开。它们都基于相同的底层知识——几何学、物理学和动力学——来解释世界的运行规律。一个能够从任意角度渲染杯子的模型，原则上应该能够模拟杯子被推动时发生的情况，并规划出一只手拿起杯子的动作。越来越多的研究开始有意模糊这三者之间的界限，而这些研究也越来越引人注目。
Why simulation is the linchpin 为什么仿真至关重要
Of the three categories, the simulator gets the least public attention, and is the most consequential of the three. This essay addresses this asymmetry.
在这三类技术中，模拟器受到的公众关注最少，但其影响却最为深远。本文将探讨这种不对称现象。
The renderer is by far the most commercially mature. A number of image- or text-to-video products are expanding in the consumer or enterprise markets rapidly. Google’s Nano Banana model has put renderer-quality image generation in the hands of potentially hundreds of millions of users. The technology is real, and the markets are real. Yet renderers optimize for visual plausibility rather than physical accuracy, and that ceiling matters. Their outputs are beautiful, but they cannot be trusted to design a building or train a robot.
渲染器是目前商业上最成熟的技术 。许多图像或文本转视频产品正在消费者和企业市场迅速扩张。谷歌的 Nano Banana 模型已将渲染器级别的图像生成功能带给了数亿潜在用户。这项技术是真实存在的，市场也是真实存在的。然而，渲染器优化的是视觉上的逼真度，而非物理上的精确度，而这种局限性至关重要。它们的输出效果固然精美，但却无法用于建筑设计或机器人训练。
The planner is the most intriguing and the most nascent, closely connected to the rapidly evolving field of robotic learning. The field has produced robotic demos in the last two years that look impressive in videos, but candor is required about what those demos actually show. Almost all have been confined to heavily constrained laboratory setups, with narrow object sets and short task horizons. None have been validated at the complexity, variability, or duration that real-world deployment demands. The gap between a compelling demo reel and a robot that reliably works in a kitchen, a warehouse, or an operating room remains vast. The commercial bets are nonetheless substantial. A wave of well-funded entrants is racing to ship general-purpose planning systems, while the largest infrastructure players are positioning planning atop broader simulation stacks. A robot that can plan is a robot that can work, and the entire industry is racing to be the one that gets there first.
规划器是最引人入胜、也最具发展潜力的领域，它与快速发展的机器人学习领域紧密相连 。 过去两年，该领域涌现出许多令人印象深刻的机器人演示视频，但我们必须坦诚地指出这些演示的实际效果。几乎所有演示都局限于高度受限的实验室环境，目标对象范围狭窄，任务周期短。没有一个演示能在真实世界部署所需的复杂性、可变性和持续时间下进行验证。引人入胜的演示视频与能够在厨房、仓库或手术室等场所可靠运行的机器人之间仍然存在巨大差距。尽管如此，商业投资的规模依然巨大。大量资金雄厚的企业正竞相推出通用规划系统，而大型基础设施提供商则将规划功能置于更广泛的仿真堆栈之上。能够进行规划的机器人才能真正发挥作用，整个行业都在竞相成为第一个实现这一目标的企业。
Simulation is the bridge between the two. If language is an abstraction of the world and pixels are a projection of it, then geometry, physics, and dynamics are the world itself. A simulator must work at that level: the structural backbone from which both visual appearance (for renderers) and action consequences (for planners) can be derived.
模拟是连接两者的桥梁。 如果说语言是对世界的抽象，像素是对世界的投影，那么几何、物理和动力学就是世界本身。模拟器必须在这个层面上工作：它是视觉外观（对于渲染器而言）和动作后果（对于规划器而言）得以生成的结构骨架。
A model that masters simulation can project its understanding into pixels for human consumption, and into action predictions for embodied agents. A model that masters only rendering, or only planning, cannot do either. The commercial surface area is enormous. NVIDIA’s Omniverse alone targets what the company estimates as more than a trillion dollars of addressable market in factories, warehouses, supply chains, and digital twins. Robotics training, autonomous vehicle testing, architectural visualization, engineering, and drug discovery all depend on something simulation-shaped.
能够精通仿真技术的模型可以将自身的理解转化为像素图像供人类使用，并预测实体智能体的行为。而仅精通渲染或仅精通规划的模型则无法做到这两点。其商业应用前景极其广阔。仅英伟达的 Omniverse 平台就瞄准了该公司估计超过万亿美元的潜在市场，涵盖工厂、仓库、供应链和数字孪生等领域。机器人训练、自动驾驶汽车测试、建筑可视化、工程设计和药物研发等都依赖于某种仿真技术。
The hardest open problems in the field live there too. Three-dimensional data with explicit geometry, material properties, and physical annotations is orders of magnitude scarcer than the internet video that renderers train on. The sim-to-real gap, which is the difference between how things behave in simulation and how they behave in reality, persists. Generative simulators introduce a new risk on top of that: AI-generated geometry can look correct while containing self-intersections or wrong scale that produce nonsensical physics. Multi-physics simulation at scale, where rigid bodies, deformable objects, fluids, and cloth all interact, remains orders of magnitude more expensive than single-domain simulation.
该领域最棘手的未解难题也存在于此。具有明确几何形状、材料属性和物理标注的三维数据比渲染器训练所用的互联网视频稀少几个数量级。模拟与现实之间的差距，即事物在模拟中的行为与它们在现实中的行为之间的差异，依然存在。生成式模拟器在此基础上引入了新的风险：人工智能生成的几何体可能看起来正确，但其中包含自相交或错误的比例，从而产生不合理的物理现象。大规模多物理场模拟（其中刚体、可变形物体、流体和布料相互作用）的成本仍然比单域模拟高出几个数量级。
At World Labs, Marble is our first move into this territory. It takes multimodal prompts (text, image, video, or spatial sketch) and generates explorable 3D environments, outputting Gaussian splats for visual exploration alongside collision meshes a physics engine can operate on. But Marble is only the first chapter of a much longer arc being written across the field as the lines between rendering, simulation, and planning begin to collapse.
在 World Labs， Marble 是我们进军这一领域的第一步。 它接收多模态提示（文本、图像、视频或空间草图），并生成可探索的 3D 环境，输出高斯散射体以供视觉探索，同时还输出可供物理引擎操作的碰撞网格。但 Marble 仅仅是这一领域漫长发展历程的开端，渲染、模拟和规划之间的界限正逐渐模糊。
Where the boundaries are collapsing and what comes next 边界在哪里正在消融，接下来会发生什么？
But more is to come. The most important pattern in the field right now is that the three categories are starting to blend into one another. The shared insight is that the knowledge required to render a world, simulate it, and act in it is largely the same. Continuing the earlier example, a model that truly understands how a cup sits on a table (its geometry, material properties, response to force, etc.) should be able to render that cup from any angle, simulate what happens when the cup is pushed, and plan for a hand to pick the cup up. The three categories are three projections of a single underlying understanding.
但未来还有更多发展。目前该领域最重要的趋势是，这三个类别正开始相互融合。其共同之处在于，渲染世界、模拟世界以及在世界中行动所需的知识大体相同。继续之前的例子，一个真正理解杯子在桌面上如何放置（其几何形状、材料属性、受力响应等）的模型，应该能够从任何角度渲染杯子，模拟杯子被推动时的情况，并规划出一只手拿起杯子的动作。这三个类别是对同一底层理解的三种不同呈现方式。
For example: a small but growing number of recent work from various robotics labs have demonstrated that—at least conceptually—a pretrained video renderer can be used as the backbone for joint world-and-action prediction, suggesting a bridge between the renderer and the planner by letting one model imagine what will happen and what to do. World Labs’ Marble already outputs Gaussian splats and collision meshes from a single model, dissolving the boundary between the renderer and the simulator. Every level is moving from passive output to interactive system, with renderers becoming action-conditioned, simulators generating worlds that are more controllable and editable, and planners deliberating rather than just reacting.
例如：近期来自各个机器人实验室的少量但不断增长的研究表明，至少在概念上，预训练的视频渲染器可以作为联合世界和动作预测的骨干网络，这为渲染器和规划器之间搭建了一座桥梁，让一个模型能够预想将会发生什么以及应该做什么。World Labs 的 Marble 已经能够从单个模型输出高斯碰撞和碰撞网格，从而打破了渲染器和模拟器之间的界限。 每个层级都在从被动输出转向交互式系统： 渲染器变得与动作相关，模拟器生成更可控、更可编辑的世界，规划器则进行思考而非仅仅做出反应。
The logical endpoint is a unified world model: one foundation model that can render photorealistic views, produce physically accurate structure, and plan action sequences, switching between output modalities depending on what the downstream consumer needs. We will still face a number of daunting challenges. The data picture is uneven, with renderers awash in internet video while simulators and planners face acute shortages of 3D assets and robot demonstrations. Optimizing for visual beauty can sacrifice the precision a robot or a high-fidelity simulation needs. Reconciling these tensions inside a single architecture is the defining open problem in world model research today, and this is what World Labs sets out to do as we continue to evolve Marble.
最终目标是构建一个统一的世界模型：一个基础模型，能够渲染逼真的视图、生成物理上精确的结构，并规划动作序列，还能根据下游用户的需求切换输出模式。然而， 我们仍然面临着诸多严峻的挑战。数据分布极不均衡，渲染器拥有海量的网络视频，而模拟器和规划器却面临着 3D 资源和机器人演示的严重短缺。追求视觉美感可能会牺牲机器人或高保真模拟所需的精度。如何在单一架构中协调这些矛盾，是当今世界模型研究领域尚未解决的关键问题，也是 World Labs 在持续改进 Marble 的过程中努力的方向。
The direction, however, is clear. The same bet the field has been making since the late 1980s — that a sufficiently rich model of the world is all that any agent needs to see worlds, build them, and act in them — is the bet now driving an entire generation of research. What gives that “big bet” weight is the convergence already underway: three threads, each already driving and shaping multi-billion-dollar industries on its own, that began as separate research programs are starting to behave like one. Taken together, as the boundaries between them collapse, they will reshape something larger: the relationship between machine intelligence and the physical world it inhabits - the long arc of spatial intelligence.
然而，方向是明确的。自20世纪80年代末以来，该领域一直秉持着同样的信念——只要拥有足够丰富的世界模型，任何智能体就能感知世界、构建世界并在其中行动——而这一信念如今正驱动着整整一代的研究。赋予这一“重大赌注”以分量的是，融合已然发生：三个方向各自独立发展，各自驱动并塑造着价值数十亿美元的产业，它们最初是独立的研究项目，如今却开始像一个整体一样运作。随着它们之间界限的消融，它们将共同重塑一个更大的格局：机器智能与其所处的物理世界之间的关系——空间智能的漫长发展历程。
Language gave machines a way to talk about that world. World models are how machines will finally come to understand, imagine, reason and interact with it.
语言赋予机器一种描述世界的方式。世界模型则是机器最终理解、想象、推理和与世界互动的方式。
浙公网安备 33010602011771号