VLM-3D空间理解

VLM-3D空间理解

CoT

Thinking in space

Notably, prevailing linguistic reasoning techniques (e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve performance, whereas explicitly generating cognitive maps during question-answering enhances MLLMs’ spatial distance ability.

《Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces》

语言学prompt方法：

1. zero-shot chain-of-thought: 在问题后面增加：Let's think step by step.​诱发大模型逐步思考的能力。再运行模型要求模型从上个回答中提取显示的答案（比fuzzy匹配精度更高）。
2. Self-Consistency w/ CoT：生成多个答案投票；
3. Tree-of-Thought: 先要求模型生成回答该问题的3个计划，再运行多次模型要求投票选出最优计划，再要求模型根据计划执行，回答问题。

‍

We prompt Gemini-1.5 Pro to first generate a cognitive map based on the given video and question, and then to use the predicted map to answer the question.

Visual CoT

《Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning》

they often lack interpretability and struggle with complex visual inputs, especially when the resolution of the input image is high or when the interested region that could provide key information for answering the question is small. To address these challenges, we collect and introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions.

制作了一个438K大小的数据集，包含：

1. question
2. answer
3. Cot-BBox

其中98K数据包含下图所示的步骤指引：

对于包含CoT BBox的训练数据，训练时在问题后面附加"Please provide the bounding box coordinate of the region that can help you answer the question better."，同时根据该问题的真值抠出对应图像区域，将其编码后的特征和完整图像编码后的特征统一送入模型。

训练分为两个阶段：

1. 在图片字幕数据上仅训练projector；
2. 在visual-cot数据集上微调全部模型；

Visual Encoder

Video-3D LLM

《Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding》

By treating 3D scenes as dynamic videos and incorporating 3D position encoding into these representations, our Video-3D LLM aligns video representations with real-world spatial contexts more accurately.

将深度图中每个像素深度坐标转换为全局坐标（根据相机内参和外参），再作为position embedding和图像feature融合。

‍

Chat-Scene

《Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers》

we decompose the input 3D scene into a set of object proposals, each assigned a unique identifier token.

1. 对图像施加2d 图像检测器dino，得到每个目标的2d embedding再经过2d projector
2. 对点云施加3d点云检测器，得到每个目标的3d embedding，再经过3d projector
3. 在词汇表中添加目标索引例如<OBJ032>,<OBJ034>​,通过tokenizer转换为embedding
4. 结合上诉3个目标级别的embedding得到一个目标的完整embedding，用于替换