当大型语言模型-LLMs-尝试推理-基于文本和视觉抽象的实验
当大型语言模型（LLMs）尝试推理：基于文本和视觉抽象的实验
原文：
towardsdatascience.com/when-llms-try-to-reason-experiments-in-text-and-vision-based-abstraction/
引言
元学习，即系统学习如何学习的能力，传统上通过基于梯度的优化、记忆增强网络或显式任务嵌入来探索。但随着大型语言模型（LLMs）的兴起，尤其是具有高级推理能力的 o3 系列，一个新的问题出现了：我们能否将 LLMs 本身作为基于任务的领域（如ARC）中的元学习器？由弗朗索瓦·肖莱特引入的抽象和推理语料库（ARC）是一个专门设计来测试广泛泛化的基准。它提供了最小监督的输入-输出转换谜题，每个任务只有少量示例，并且通常没有任务之间的共享表面结构。换句话说：一个元学习的游乐场。为了了解典型的抽象和推理任务，读者可以访问 ARC 游戏页面。
来自 ARC 网站的示例游戏。从演示网格中可以看出，测试网格的任务是将黑色区域转换为黄色，只要它们完全被绿色边界包围。
数据和设置
为了探索 LLMs 如o3-mini是否能够在抽象推理任务上执行元学习，我使用了ARC Prize 2025 Kaggle 比赛的数据。数据集存储库可以在这里找到（Apache 2.0 许可证）。该数据集由输入-输出网格转换组成，挑战模型从少量示例中推断抽象规则。
每个任务提供：
-
一些训练示例（
输入和输出二维网格） -
模型必须预测相应输出的单个测试输入网格
第二个数据集提供了每个测试输入网格的解决方案网格。以下是一个简化的数据格式示例：
# training examples - dictionary of dictionaries.
# Here is an extracted task
{'train': [{'input': [[6, 6, 0], [6, 0, 0], [0, 6, 6]],
'output': [[6, 6, 0, 6, 6, 0, 0, 0, 0],
[6, 0, 0, 6, 0, 0, 0, 0, 0],
[0, 6, 6, 0, 6, 6, 0, 0, 0],
[6, 6, 0, 0, 0, 0, 0, 0, 0],
[6, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 6, 6, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 6, 6, 0, 6, 6, 0],
[0, 0, 0, 6, 0, 0, 6, 0, 0],
[0, 0, 0, 0, 6, 6, 0, 6, 6]]},
{'input': [[4, 0, 4], [0, 0, 0], [0, 4, 0]],
'output': [[4, 0, 4, 0, 0, 0, 4, 0, 4],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 4, 0, 0, 0, 0, 0, 4, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 4, 0, 4, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 4, 0, 0, 0, 0]]},...,
'test': [{'input': [[7, 0, 7], [7, 0, 7], [7, 7, 0]]}]
}
# example of solution to test input grid - dictionary of lists
# Here is the extracted solution for the only test input grid above
[[[3, 2, 3, 2, 3, 2],
[7, 8, 7, 8, 7, 8],
[2, 3, 2, 3, 2, 3],
[8, 7, 8, 7, 8, 7],
[3, 2, 3, 2, 3, 2],
[7, 8, 7, 8, 7, 8]]]
每个网格是一个 0 到 9 的整数 2D 数组，代表彩色像素。网格大小各异，网格变换也可能从输入网格到输出网格的大小变化。为了可视化数组，我使用了matplotlib的自定义颜色映射：
from matplotlib import colors
cmap = colors.ListedColormap([
'#8B00FF', # Violet
'#4B0082', # Indigo
'#0000FF', # Blue
'#FFFF00', # Yellow
'#00FF00', # Green
'#FF7F00', # Orange
'#FF0000', # Red
'#964B00', # Golden
'#000000', # Black
'#FFFFFF', # White
])
norm = colors.Normalize(vmin=0, vmax=9)
# Function to visualize an array
def visualize_matrix(matrix, title='', cmap=cmap, norm=norm):
plt.imshow(matrix, cmap=cmap, norm=norm)
plt.title(title)
plt.axis('off') # Remove axes
plt.show()
对于模型交互，我通过 LangChain 使用了 OpenAI 的o3-mini模型。稍后，我们还将使用gpt-4.1：
from langchain_openai import ChatOpenAI
import getpass
import os
# Prompt for a secret input
openai_key = getpass.getpass("Enter your OpenAI API key: ")
os.environ["OPENAI_API_KEY"] = openai_key
AGENT_MODEL = "o3-mini" # reasoning model, https://platform.openai.com/docs/models
AGENT_LLM = ChatOpenAI(model=AGENT_MODEL)
# AGENT_LLM = ChatOpenAI(model=AGENT_MODEL, reasoning_effort='low')
为了处理 LLM 的响应，特别是当模型返回一个作为 Python 代码的三重反引号内的预测输出网格时，我编写了一个实用工具：
import re, ast
def extract_python_code(response_string):
match = re.search(r"```python\s*(.*?)```py", response_string, re.DOTALL)
if match:
return ast.literal_eval(match.group(1).strip())
return None
这种设置使我能够构建一个完整的推理循环：用少量示例提示模型，提取并应用生成的算法，评估其在新的测试输入上的性能，最后使用评估来改进算法。
使用 o3-mini 进行推理测试
为了评估 LLM 是否可以在抽象推理任务上进行“元学习”，我使用一个受人类可能如何处理少量任务启发的闭环推理设置测试了<a href="https://platform.openai.com/docs/models" rel="noreferrer noopener" target="_blank">o3-mini</a>模型。对于每个 ARC 挑战，我向模型提供了一小批演示输入-输出网格对，并要求它推导出一个可重复使用的算法。
我使用 LangChain 的ChatPromptTemplate定义了一系列提示，以模拟推理、应用、评估和改进。这个过程模仿了一个带有有限监督的内部训练循环：
-
PROMPT_REASON：模型被提供训练示例，并要求用伪代码推断一个通用算法。
-
PROMPT_SOLVE：生成的算法应用于新的输入（包括训练和测试）。
-
PROMPT_ASSESS：当算法失败时，模型会收到与其预测输出与预期输出相比的反馈。
-
PROMPT_SUMMARIZE_FEEDBACK：模型总结从失败的尝试中累积的反馈，以迭代改进其方法。
from langchain_core.prompts import ChatPromptTemplate
PROMPT_REASON = ChatPromptTemplate.from_messages(
[
(
"system",
"You are an expert in solving abstract reasoning tasks. "
"You will be given several demonstration input-output pairs of 2D arrays. "
"Your goal is to develop a single algorithm that maps each input array to its corresponding output array.\n\n"
"Each input and output is a 2-dimensional array of integers between 0 and 9\. "
"Solving the task involves:\n"
"- Analyzing the demonstration pairs\n"
"- Identifying abstract patterns or transformations\n"
"- Formulating a general rule or algorithm that works across all examples\n"
"- Producing pseudocode that implements the rule\n\n"
"If prior attempts were made, you will also receive feedback summarizing what went wrong. "
"Carefully use this feedback to improve your solution.\n\n"
"Return only the updated algorithm as pseudocode. Do not describe or explain it.\n\n"
"### Feedback (summary of previous attempts):\n{attempt_history}\n\n"
"### Demonstration Pairs:\n{train_pairs}\n"
),
(
"ai",
"Answer:"
)
]
)
PROMPT_SOLVE = ChatPromptTemplate.from_messages(
[
(
"system",
"You are an expert in abstract reasoning. "
"Previously, you analyzed demonstration input-output pairs and developed an algorithm "
"to transform input arrays into output arrays.\n\n"
"Now, use that algorithm to generate an output array for a new, unseen input array.\n\n"
"Only return the output array, formatted as valid Python code within a code block. "
"For example:\n```python\n[[2, 3], [5, 6]]\n```py\n"
"### Developed algorithm:\n{reasoning_template}\n\n"
"### New input array:\n{test_input}\n"
),
(
"ai",
"Answer:"
)
]
)
PROMPT_ASSESS = ChatPromptTemplate.from_messages(
[
(
"system",
"You are an expert in abstract reasoning. "
"A solution array was generated by applying the algorithm to the input array. "
"Compare the generated solution to the actual target output. "
"Analyze why the two arrays differ, and provide **clear and concise feedback** on how to improve the algorithm.\n\n"
"Only return your feedback-do not repeat the arrays or algorithm.\n\n"
"### Algorithm:\n{reasoning_template}\n\n"
"### Input array:\n{test_input}\n\n"
"### Solution array (generated by algorithm):\n{solved_test_output}\n\n"
"### Target output array:\n{test_output}\n"
),
(
"ai",
"Answer:"
)
]
)
PROMPT_SUMMARIZE_FEEDBACK = ChatPromptTemplate.from_messages(
[
(
"system",
"You are an expert in summarizing feedback on algorithm development. "
"You will be given a history of past attempts, each containing an algorithm and feedback about its performance.\n\n"
"Your goal is to produce a **concise summary** of the most important lessons learned-"
"focusing on how the algorithm should be improved and what mistakes should be avoided in future versions.\n\n"
"Return only the feedback summary. Do not repeat the original attempts or feedback.\n\n"
"### Attempt History:\n{attempt_history}\n"
),
(
"ai",
"Answer:"
)
]
)
这些提示被链接到一个简单的 LangChain 管道：
reasoning_chain = PROMPT_REASON | AGENT_LLM
solve_chain = PROMPT_SOLVE | AGENT_LLM
assess_chain = PROMPT_ASSESS | AGENT_LLM
summarize_feedback_chain = PROMPT_SUMMARIZE_FEEDBACK | AGENT_LLM
对于每个 ARC 挑战：
-
模型接收演示对和先前反馈；
-
模型以伪代码的形式生成一个新的算法（
reasoning_template）； -
算法在所有演示上进行了测试；
-
如果失败，模型：对不匹配的预测获得详细反馈；总结尝试中的错误；细化算法的下一个版本；
-
一旦模型正确完成所有演示，我将在未见过的测试输入上对其进行测试。
这个过程在每个挑战中重复，直到达到最大尝试次数。一个成功的算法可以概括提供的示例，并且正确应用于保留的测试案例。这个设置测试了模型是否能够提取抽象模式，随着时间的推移改进其推理，并从非常少的例子中进行泛化。
reasoning_templates = {}
for i, id in enumerate(id_train_challenges):
print(f"Training on challenge {i} ID: {id}")
train_pairs = train_challenges[id]['train']
test_input = train_challenges[id]['test'][0]['input'] # only pick the first test input
test_output = train_sols[id][0] # only pick the first test output
train_pairs_str = ''
for i, train_pair in enumerate(train_pairs):
train_pairs_str += f"Demonstration pair {i+1}:\n input grid: {train_pair['input']} \n output grid: {train_pair['output']}\n"
train_pairs_str = train_pairs_str.strip()
# keep trying until you figure out how to solve the challenge
right_wrong = "incorrect"
# Start with an empty reasoning template, which will be refined over time
reasoning_template = ''
k = 1
max_attempts = 5
attempt_history = []
attempt_history_summary = ''
while right_wrong == "incorrect":
print(f"Attempt {k} to solve the challenge...")
# Build the reasoning message with the current reasoning template and attempt history
# This message will be used to generate a new reasoning template
reason_message = {
"train_pairs": train_pairs_str,
"attempt_history": attempt_history_summary,
}
res = reasoning_chain.invoke(reason_message)
reasoning_template = res.content
# Assess reasoning template
wrong_pairs = []
for train_pair in train_pairs:
demo_input = train_pair['input']
demo_output = train_pair['output']
# Test the reasoning template on the demonstration pair
test_message = {
"test_input": demo_input,
"reasoning_template": reasoning_template,
}
res = solve_chain.invoke(test_message)
solved_demo_output = extract_python_code(res.content)
# Compare the output with the demonstration output
if solved_demo_output != demo_output:
wrong_pairs.append((demo_input, demo_output, solved_demo_output))
if len(wrong_pairs) > 0:
right_wrong = 'incorrect'
print(f"Reasoning template failed on {len(wrong_pairs)} demonstration pairs.")
if k >= max_attempts:
print(f"Max attempts reached ({max_attempts}). Stopping for challenge {id}.")
reasoning_templates[id] = ''
break
print("Assessing the reasoning template...")
assessment_res = f'Algorithm failed on {len(wrong_pairs)} demonstration pairs. Here is the feedback:\n'
for demo_input, demo_output, solved_demo_output in wrong_pairs:
assess_chain_message = {
"reasoning_template": reasoning_template,
"test_input": demo_input,
"solved_test_output": solved_demo_output,
"test_output": demo_output,
}
res = assess_chain.invoke(assess_chain_message)
assessment_res += f" - From input {demo_input} to output {demo_output}, your solution was {solved_demo_output}: {res.content.strip()}\n"
attempt_history.append({
"attempt": k,
"reasoning_template": reasoning_template,
"feedback": assessment_res
})
summary_message = {
"attempt_history": attempt_history,
}
summary_res = summarize_feedback_chain.invoke(summary_message)
attempt_history_summary = summary_res.content.strip()
else:
print("Solution is correct!")
right_wrong = "correct"
reasoning_templates[id] = reasoning_template
# test it against the test input/ output .... but do not give feedback (this is supposed to be unknown)
test_message = {
"test_input": test_input,
"reasoning_template": reasoning_template,
}
res = solve_chain.invoke(test_message)
solved_test_output = extract_python_code(res.content)
if test_output != solved_test_output:
print(f"Test output does not match the true output for challenge {id}.")
else:
print(f"Test output matches the true output for challenge {id}.")
#visualize_matrix(test_input, "Input grid")
#visualize_matrix(test_output, "True output")
#visualize_matrix(solved_test_output, "Test Output")
print("-" * 40) # Separator between entries
k += 1
结果：当推理起作用时
在某些情况下，o3-mini能够仅从几个输入输出演示中正确推断出可泛化的算法。一个这样的例子是生成基于小 2×2 输入网格的图案平铺。
只需一次尝试，模型就收敛到了以下伪代码：
BEGIN
Let input be a 2x2 grid, where:
input[0] = [a, b]
input[1] = [c, d]
Initialize output as an empty list.
FOR each row index r from 0 to 5 DO:
Let original_row ← input[r mod 2]
IF (FLOOR(r / 2)) mod 2 = 1 THEN
Let base_row ← REVERSE(original_row)
ELSE
Let base_row ← original_row
ENDIF
Initialize new_row as an empty list.
FOR repeat from 1 to 3 DO:
Append all elements of base_row to new_row.
ENDFOR
Append new_row to output.
ENDFOR
RETURN output
END
这里是预期的解决方案（真实输出）和模型伪代码（测试输出）。
真实输出网格（图片由作者提供）。
伪代码的测试输出网格（图片由作者提供）。
该算法展示了几个显著的推理能力：
-
模式抽象：模型从有限的数据中推断出重复平铺模式；
-
模块逻辑：它引入了基于模的索引（
r mod 2和(r // 2) mod 2）来交替行行为，模仿视觉镜像； -
网格构建：解决方案通过重复复制和反转将 2×2 输入放大到更大的 6×6 网格；
模型在没有硬编码规则的情况下发现这种结构，这表明它正在进行一种算法综合，由少量示例抽象指导。
这里是另一个成功的例子。
示例网格和测试输入网格（图片由作者提供）。
再次，只需一次尝试，模型就收敛到了以下伪代码：
BEGIN
Let N = 3
Create output as a 2D array of size (N×N) × (N×N), filled with 0
FOR each row r from 0 to N–1:
FOR each column c from 0 to N–1:
IF input[r][c] ≠ 0 THEN
FOR each i from 0 to N–1:
FOR each j from 0 to N–1:
Set output[(r * N) + i][(c * N) + j] = input[i][j]
RETURN output
END
这里是预期的解决方案和模型伪代码中的解决方案。
真实输出网格（图片由作者提供）。
伪代码的测试输出网格（图片由作者提供）。
该算法有效地将整个输入网格在每个输入单元非零的位置上平铺到输出网格中。平铺是对齐的，使得每个原始网格的副本都放置在由(r * N, c * N)确定的偏移量处——这是非零输入单元的放大坐标。
这里令人印象深刻的是，模型：
-
学习条件放置。它只在输入值非零的地方粘贴输入；
-
使用坐标算术来缩放放置位置，显示空间理解；
-
将输入视为控制逻辑和内容，将布局检测与重复结合。
当基于文本的推理不够用时
在下面的任务中，模型被给出了一小部分演示输入输出网格对，并要求推断转换规则。
示例网格和测试输入网格（图片由作者提供）。
文本模型（o3-mini）生成了一个详细的伪代码解决方案，结构化、合理且内部一致：
BEGIN
Let original ← input grid
Let output ← deep copy of original
Let R ← number of rows in original
Let C ← number of columns in original
// Compute ring index for every cell that is part of a non-zero region.
// A cell's ring index is defined as:
// - 0 if the cell is on the boundary of the grid OR if at least one of its 4-neighbors is 0
// - Otherwise, 1 + min(ring index of its 4-neighbors)
Create grid ring of size R × C, filled with −1
For each cell (r, c) in original:
If original[r][c] ≠ 0 then
If r = 0 OR c = 0 OR r = R−1 OR c = C−1 then
ring[r][c] ← 0
Else if any 4-neighbor (up, down, left, right) of (r, c) has value 0 then
ring[r][c] ← 0
changed ← true
While changed is true:
Set changed ← false
For each cell (r, c) in original such that original[r][c] ≠ 0 and (ring[r][c] < 0 or not finalized):
Let candidates ← list of ring indices from any 4-neighbor (r′, c′) with a non-negative ring value
If candidates is non-empty then
newRing ← min(candidates) + 1
If ring[r][c] < 0 OR newRing < ring[r][c] then
ring[r][c] ← newRing
changed ← true
// Process propagation from outer ring inward.
// For each cell with ring index ≥ 1, update its value based on a neighbor with ring index one less.
// Selection is deterministic: "up, left, down, right"
Let maxRing ← maximum value in ring
For layer from 1 to maxRing:
For each cell (r, c) such that ring[r][c] = layer:
For direction in [(−1,0), (0,−1), (1,0), (0,1)]:
nr ← r + direction.row
nc ← c + direction.col
If (nr, nc) is in bounds AND ring[nr][nc] = layer − 1 then
Let base ← output[nr][nc]
If base = 2 then
newValue ← 8
Else if base mod 2 = 0 then
newValue ← base + 2
Else // base is odd
newValue ← base + 1
output[r][c] ← newValue
Break out of neighbor loop
Return output
END
乍一看，这似乎是一种复杂的策略：为每个非零区域计算环索引，然后使用确定性规则向内扩散值。它甚至包括更新逻辑：
-
如果“基数”单元格是 2 → 分配 8
-
如果基数是偶数 → 加 2
-
如果基数是奇数 → 加 1
但是这个整个算法，尽管连贯，却是错误的。它完全无法匹配演示中显示的真实底层转换。
预期测试输出网格（图片由作者提供）。
从伪代码测试输出网格（图片由作者提供）。
而不是推理蓝色边界的区域及其嵌套结构，模型基于边缘距离和零的邻近性生成了一种通用的填充算法。即使反馈也在不断改进之前确定的程序方法：
['Key lessons are to: • Precisely compute the ring index so that only',
'true boundary (or external zero) cells get index 0, ensuring that',
'inner cells receive higher indices for proper propagation. • Use a',
'reliable, consistent method for selecting the "base" value for',
'updates-ideally by considering all adjacent lower-ring cells or using',
'a deterministic order-and use an immutable copy of the original grid',
'for these lookups. • Apply the parity‐based update rules correctly so',
'that cells with ring index ≥ 1 get the specified value increments',
'(especially the special case when the base is 2) rather than remaining',
'unchanged. • Ensure that the update logic cascades inward, allowing',
'inner cells to correctly inherit and build upon values from outer',
'rings.']
那么出了什么问题？
-
拓扑而非视觉。模型专注于连通性和边缘邻近性，忽略了视觉定义的区域。
-
程序性而非推理性。逻辑是刚性的、手工制作的，而非从示例中的模式中推导出来。
-
与演示无关。没有迹象表明模型有意义地结合了少样本示例。它可能默认使用熟悉的模式——使用层进行空间增长。
这并不令人惊讶。仅文本的 LLM 没有视觉基础。它们将网格作为符号输入——一排数字，而不是封闭的图形或嵌套模式。因此，它们的归纳偏差倾向于符号或图状算法，而不是感知抽象。
在这种情况下，模型陷入了常见的陷阱：生成听起来合理但实际上错误的东西。它产生了一种可能适用于扩散任务但不适于当前任务的时空传播方案。这突显了基于文本的少样本提示在抽象视觉推理中的关键弱点：模型的“推理”与感知理解脱节。它是基于内部先验而非外部线索发明算法。
当推理失败时：图像模型也不是魔法
为了提高泛化能力，我从纯文本推理过渡到基于图像的提示，利用 LangChain 通过 GPT-4.1 的多模态能力。这个设置将输入-输出网格示例编码为 base64 图像，这些图像与描述任务的天然语言提示一起呈现。
from langchain_core.messages import HumanMessage
import io
import base64
AGENT_MODEL = "gpt-4.1"
# Prompt for image based reasoning
PROMPT_REASON_IMG = """You are an expert at solving abstract reasoning tasks.
These are unique reasoning tasks with limited examples. You are given demonstration input-output 2D grids.
The colormap used is as follows:
{{
'Violet': 0,
'Indigo': 1,
'Blue': 2,
'Yellow': 3,
'Green': 4,
'Orange': 5,
'Red': 6,
'Golden': 7,
'Black': 8,
'White': 9
}}
Your goal is to develop a single algorithm that maps each input grid to its corresponding output grid.
A successful solution involves:
- Analyzing the demonstration examples carefully
- Identifying underlying visual or spatial patterns
- Formulating a general transformation rule
- Translating this rule into clear pseudocode
If this is not your first attempt, a summary of previous feedback is also provided. Review it carefully and incorporate it to improve your solution.
Test your algorithm against the demonstrations to ensure it works.
Return **only the algorithm pseudocode**, formatted as plain text. Do not explain it or add extra commentary.
"""
# If your array is 10x10 and you want each cell to be 20x20 pixels (cell_px), the image will be 200x200 pixels.
# Convert matrix into image
def visualize_grid_fig(matrix, cmap=cmap, norm=norm, cell_px=20, show=False):
if type(matrix) is not np.ndarray:
matrix = np.array(matrix)
h, w = matrix.shape[:2]
figsize = (w * cell_px / 100, h * cell_px / 100) # inches
fig, ax = plt.subplots(figsize=figsize)
ax.imshow(matrix, cmap=cmap, norm=norm)
ax.axis('off')
if show:
plt.show()
else:
plt.close(fig)
return fig
# encode image for model
def fig_to_base64(fig, dpi=100):
buf = io.BytesIO()
fig.savefig(buf, format='png', dpi=dpi, bbox_inches='tight')
buf.seek(0)
img_base64 = base64.b64encode(buf.read()).decode('utf-8')
buf.close()
return img_base64
# In the loop replace reasoning code with this
# reasoning with images
reason_message = [{"type": "text", "text": PROMPT_REASON_IMG}]
for i, example in enumerate(train_pairs):
#fig_in = visualize_grid_fig(example['input'], cmap, norm)
#fig_out = visualize_grid_fig(example['output'], cmap, norm)
fig_in = visualize_grid_fig(example['input'], )
fig_out = visualize_grid_fig(example['output'], )
fig_in = fig_to_base64(fig_in)
fig_out = fig_to_base64(fig_out)
reason_message.append({"type": "text", "text": f"### Input grid {i+1}:"})
reason_message.append({"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{fig_in}"}})
reason_message.append({"type": "text", "text": f"### Output grid {i+1}:"})
reason_message.append({"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{fig_out}"}})
reason_message.append({"type": "text", "text": f"### Feedback (summary of previous attempts): {attempt_history_summary}"})
reason_message = HumanMessage(content=reason_message)
res = AGENT_LLM.invoke([reason_message])
reasoning_template = res.content
结果伪代码标志着在表达性方面的明显进步。模型能够：
-
使用视觉特征而非纯粹符号结构检测蓝色边界的方块；
-
根据方块大小和嵌套深度应用规则以推断内部填充颜色；
-
在填充之前按大小排序已识别的方块，有效地避免覆盖冲突。
这是生成的伪代码：
1\. Let grid be the input 2D array.
2\. Create output_grid as a copy of grid.
3\. Identify all blue-bordered squares in the grid:
a. For each possible top-left corner (i, j):
i. For each possible square size s (s ≥ 3, up to min(grid height, grid width)):
- Check if the square of size s starting at (i, j) is fully within bounds.
- Check if all *border* cells of this square are Blue (value = 2).
- Check that the *interior* cells (not on the border) do not contain any Blue (2).
- If all conditions are met, record the square as (i, j, s).
4\. Sort the list of detected blue-bordered squares by size in ascending order (smallest first).
5\. For each detected square (i, j, s), in sorted order:
a. Determine the fill color:
- If the square is the smallest (no other blue-bordered square is fully inside it), set fill color = Black (8).
- If the square is the largest (no other blue-bordered square fully contains it), fill color =
- If there are exactly 2 blue-bordered squares, set fill color = Green (4).
- If there are three blue-bordered squares in the grid, fill color = Yellow (3).
- If the square is nested (not smallest or largest), fill color = Black (8).
- (More complex rules may generalize beyond these based on demonstrations.)
b. Fill the interior of the square:
For each cell (x, y) strictly inside the square (i+1 ≤ x < i+s−1) and (j+1 ≤ y < j+s−1):
- If output_grid[x][y] is not Blue (2), set it to the chosen fill color.
6\. Return output_grid.
Special notes:
- Never overwrite Blue (2) border pixels.
- When filling, later (larger) squares overwrite earlier (smaller) fills in overlapping regions.
- Only process valid blue-bordered squares (minimum size 3x3, complete border).
- If there are multiple disjoint blue-bordered squares, treat each independently for fill color assignment as above matching the demonstration logic.
模型明显表现出结构化推理。它发展出嵌套几何形式的内部表示，并试图应用从示例中推导出的基于规则的转换。
从基于图像推理模型伪代码测试输出网格（图片由作者提供）。
然而，尽管取得了这些进展，该模型仍然无法可靠地泛化。在新配置中，它错误地分配了填充颜色，依赖于基于大小优先级或刚性嵌套假设等脆弱的启发式方法。例如，它可能会假设最大的正方形总是填充黄色，即使在新环境中这种逻辑不再成立。这种失败揭示了更深层次的局限性：即使有图像输入，该模型也不以人类的方式“看到”。它不构建空间关系的整体感知表示。相反，它将图像转换为符号模式，并应用确定性程序，如洪水填充、排序或位置索引。
在实践中，这意味着模型从内部抽象进行推理，而不是从感知基础进行推理。它推断出“较小的正方形变黑”，或“根据大小排名填充”，而没有完全理解为什么那些分配在演示中发生。因此，任何与预期布局的偏差都可能导致它出错。
这表明，虽然多模态提示扩展了模型的表达范围，但它还没有提供人类所显示的那种灵活、通用的视觉推理。这些任务最终可能需要更强的程序归纳、元学习或混合系统，这些系统将感知分组与学习规则相结合。
结论
在这项研究中，我探讨了大型语言模型（无论是基于文本的还是多模态的）是否可以从抽象推理任务的例子中进行元学习。具体来说，我关注了来自 ARC 数据集的一类问题，其中解决方案需要识别视觉模式、学习转换并将它们推广到新的测试输入。
通过直接提示实验，我发现：
-
基于文本的模型（例如，
o3-mini）经常产生看似合理的算法，这些算法在拓扑或程序上合理，但与任务的视觉逻辑完全脱节。这些模型依赖于对标记网格的符号推理，并默认使用熟悉的启发式方法，如洪水填充、环传播或基于规则的更新，而不管提供的例子是什么。 -
多模态模型（例如，具有视觉功能的 GPT-4）在模式检测和关系推理方面显示出明显的改进。它们成功地识别了蓝色边界的区域，并根据相对大小或嵌套调整行为。然而，它们的泛化仍然脆弱：它们仍然应用了脆弱的规则，如基于固定大小的分配，并在与演示不同的新布局中失败。
这些发现表明，即使有视觉输入，当前的 LLM（大型语言模型）也不像人类那样“看到”。它们以符号方式处理图像，而不是以感知方式。它们的推理是由内部构建的规则驱动的，而不是对形状、层次或可用性的灵活、通用的视觉理解。
我观察到的局限性强化了一个核心紧张关系：仅凭少量样本提示，即使有图像，也不足以进行稳健的抽象。真正的泛化可能需要：
-
程序归纳：从示例中推断可重用、结构化的转换；
-
感知基础：开发能够以组合方式解析和处理视觉场景的架构；
-
元学习架构：构建能够动态调整推理策略而不是应用预先学习的启发式算法的模型；
今天的 LLMs（大型语言模型）在广度上令人惊叹，但它们仍然是基于先验知识进行猜测，而不是在人类意义上学习如何学习。它们缺乏对抽象和转换的强大归纳偏见。ARC 风格的任务清楚地揭示了这一差距：成功需要的不只是模式识别，还需要以结构化、组合的方式从示例中进行推理。这些结果并不令人沮丧，反而更加明确。我们现在知道了天花板在哪里。下一代模型，那些具有混合架构、持久记忆和显式元学习能力的模型，可能最终会突破这一限制。
浙公网安备 33010602011771号