Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ability of VLM agents, existing work has mainly followed two lines: post-training methods such as supervised fine-tuning and reinforcement learning, and agentic systems that call external spatial tools to gather intermediate evidence. We study a complementary and underexplored route: can a frozen VLM agent improve its spatial reasoning through parameter-update-free self-evolution, without depending on external expert spatial tools at inference time? We present Spatial Memory Agent (SMA), an experience-grounded runtime framework that converts verified spatial experience into reusable transferable lessons. In a verifiable spatial environment, SMA queries the frozen VLM, obtains a predicted answer and reward, and uses verifier-guided reflection to distill compact transferable lessons. SMA assigns each lesson a Transfer Reliability Score (TRS), initialized uniformly and calibrated from later retrieval outcomes as visit evidence of future transfer reliability. During read-only deployment, SMA retrieves lessons by semantic filtering and similarity-TRS combined ranking to guide frozen-model inference. Across five representative spatial benchmarks and four base VLMs, SMA achieves the highest macro average in every base-model block and the best accuracy in most of the 20 evaluations, establishing a practical parameter-update-free path for spatial self-evolution across the evaluated frozen model scales and environments.
Figure 1. Main benchmark comparison across four frozen base VLMs. Each radial panel reports accuracy for the five main benchmark slices together with SITE-image and ViewSpatial, while each colored bar identifies one memory method and the center labels the corresponding base model.
Figure 2. SMA workflow from verified experience to read-only deployment. A frozen VLM solves a verifiable spatial problem, a verifier-guided reflection step compresses the rollout into a transferable memory card, and semantic filtering plus TRS-based ranking selects memories for a new task without parameter updates or deployment-time writeback.
SMA transforms a verifier-scored rollout into a compact memory card containing a task summary and transferable lesson. The verified target guides reflection, while anti-leakage rules forbid copying answers into memory.
A semantic filter first proposes relevant cards. Combined ranking then balances semantic relevance with calibrated TRS, prioritizing procedures that have demonstrated transfer value in later visits.
After experience acquisition, the memory bank, visit counts, rewards, and TRS values remain fixed. Retrieved lessons guide new tasks while the VLM base model remains frozen.
We evaluate No memory, RAG, MemP, MemRL-R, MemRL-GT, and SMA on five representative spatial benchmark slices and four frozen VLMs. The table reports deployment accuracy, with Avg. computed across the five benchmark columns.
| Model | Method | RoboS. | ERQA | Omni3D | SAT | EmbS. | Avg. |
|---|---|---|---|---|---|---|---|
| Qwen3.5-122B-A10B | No memory | 61.2 | 54.5 | 40.0 | 83.7 | 87.3 | 65.3 |
| RAG | 56.8 | 55.5 | 40.4 | 82.0 | 86.4 | 64.2 | |
| MemP | 62.4 | 55.0 | 39.2 | 82.3 | 87.5 | 65.3 | |
| MemRL-R | 63.0 | 56.0 | 40.4 | 85.3 | 86.5 | 66.2 | |
| MemRL-GT | 64.0 | 53.5 | 40.0 | 83.7 | 86.6 | 65.6 | |
| SMA (ours) | 65.5 | 60.5 | 43.2 | 87.0 | 87.6 | 68.8 | |
| Qwen3.6-35B-A3B | No memory | 57.1 | 49.5 | 37.2 | 78.0 | 86.3 | 61.6 |
| RAG | 55.0 | 50.5 | 42.4 | 84.3 | 84.1 | 63.3 | |
| MemP | 55.2 | 51.5 | 42.0 | 82.3 | 87.6 | 63.7 | |
| MemRL-R | 53.6 | 54.0 | 40.8 | 84.0 | 86.8 | 63.8 | |
| MemRL-GT | 52.7 | 51.0 | 43.6 | 81.3 | 87.0 | 63.1 | |
| SMA (ours) | 57.9 | 57.5 | 45.2 | 85.3 | 87.7 | 66.7 | |
| Qwen3.6-27B | No memory | 54.1 | 53.0 | 41.6 | 82.3 | 85.7 | 63.3 |
| RAG | 59.3 | 54.5 | 44.0 | 85.0 | 86.5 | 65.9 | |
| MemP | 65.4 | 51.5 | 44.0 | 86.0 | 87.1 | 66.8 | |
| MemRL-R | 62.2 | 51.5 | 44.8 | 83.7 | 86.8 | 65.8 | |
| MemRL-GT | 67.2 | 55.5 | 43.6 | 87.0 | 87.2 | 68.1 | |
| SMA (ours) | 68.5 | 58.0 | 47.6 | 87.0 | 87.9 | 69.8 | |
| Qwen3.5-9B | No memory | 58.1 | 46.5 | 37.2 | 77.3 | 84.1 | 60.6 |
| RAG | 55.5 | 49.5 | 31.6 | 77.7 | 81.8 | 59.2 | |
| MemP | 53.7 | 53.0 | 34.4 | 78.0 | 84.2 | 60.7 | |
| MemRL-R | 54.2 | 43.5 | 36.4 | 76.7 | 82.5 | 58.7 | |
| MemRL-GT | 52.7 | 49.0 | 34.4 | 80.3 | 83.8 | 60.0 | |
| SMA (ours) | 58.5 | 52.0 | 40.8 | 81.3 | 84.9 | 63.5 |
Table 1. Main results in accuracy (%, Acc.) on five spatial benchmark slices. Dark-gray bold cells mark the best values per column within each base-model block; light-gray cells mark the second-best values.
SMA's gains extend across the evaluated frozen base-model scales rather than being tied to a single model setting.
| Setting | Accuracy | Δ vs. SMA |
|---|---|---|
| SMA (ours) | 68.5 | - |
| without summary | 65.3 | -3.2 |
| without transferable lesson | 65.0 | -3.5 |
| without semantic filter | 62.7 | -5.8 |
| with raw model output | 64.1 | -4.4 |
| Reward-only reflection | 63.0 | -5.5 |
Table 2. Component and reflection ablations on RoboSpatial with Qwen3.6-27B.
Figure 3. Sensitivity of the TRS weight η and retrieval depth k on RoboSpatial with Qwen3.6-27B.
Reliable transfer requires both structured memory writing and calibrated retrieval; either unfiltered memories or reward-only reflection weakens the memory bank.
We test model and benchmark transfer without changing the frozen VLM. For model transfer, memories written with Qwen3.5-122B-A10B are deployed with Qwen3.6-27B on the same benchmark. For benchmark transfer, Qwen3.6-27B writes memories on one benchmark and deploys them on another.
| Transfer setting | No memory | Transferred bank | Δ |
|---|---|---|---|
| Model transfer: Qwen3.5-122B-A10B → Qwen3.6-27B | |||
| RoboSpatial | 54.1 | 63.5 | +9.4 |
| ERQA | 53.0 | 56.5 | +3.5 |
| Omni3D | 41.6 | 44.8 | +3.2 |
| SAT | 82.3 | 88.0 | +5.7 |
| EmbSpatial | 85.7 | 87.3 | +1.6 |
| Benchmark transfer: Qwen3.6-27B | |||
| ERQA → RoboSpatial | 54.1 | 61.7 | +7.6 |
| EmbSpatial → RoboSpatial | 54.1 | 61.4 | +7.3 |
| EmbSpatial → Omni3D | 41.6 | 44.4 | +2.8 |
| Omni3D → EmbSpatial | 85.7 | 87.2 | +1.5 |
Table 3. Representative model and benchmark transfer results. Dark-gray bold cells mark transferred-bank accuracy; light-gray cells show the corresponding no-memory target baseline. Δ is the gain over target inference without memory.
The memory bank transfers across both models and benchmarks, improving performance beyond the setting in which its memories were written.
We bin retrieved memories by their mean Transfer Reliability Score (TRS) and measure deployment accuracy across ERQA, EmbSpatial, Omni3D, RoboSpatial, and SAT for Qwen3.6-27B. We also group memories by whether their source questions were answered successfully or failed during memory writing.
| Source outcome | TRS | Acc. |
|---|---|---|
| Success | 0.522 | 85.7% |
| Failure | 0.452 | 61.4% |
Table 4. Source-quality diagnostic for Qwen3.6-27B: mean TRS and downstream accuracy grouped by source-question outcome.
Figure 4. TRS-reliability diagnostic for Qwen3.6-27B.
We group evaluation questions by the outcomes of their three retrieved memories: all success, mixed, or all failure. For each group, we report the number of questions, deployment accuracy, and mean TRS.
| Composition | N | Acc. | TRS |
|---|---|---|---|
| All success | 13,226 | 93.0% | 0.909 |
| Mixed | 10,264 | 65.1% | 0.653 |
| All failure | 603 | 39.0% | 0.480 |
Table 5. Retrieved-memory composition for Qwen3.6-27B; N is the number of evaluation questions, Acc. is deployment accuracy, and TRS is mean transfer reliability.
We compare SMA with the similarity-oriented MemP reference on Qwen3.6-27B. The analysis reports retrieved-memory similarity and deployment accuracy for each of the five shared benchmarks, together with their macro averages.
Figure 5. Similarity reduction and accuracy gain over MemP for Qwen3.6-27B.
The best memory is not always the nearest memory; transfer reliability turns retrieval from semantic matching into evidence-weighted procedure selection.
We compare One-Pass Memory Writing with continual rewriting over ten passes on RoboSpatial, ERQA, Omni3D, EmbSpatial, and SAT using Qwen3.6-27B. The protocol analysis tracks memory-bank size, content redundancy, and TRS-update coverage across passes.
Figure 6. Memory-writing protocol scaling across ten passes.
One-Pass Memory Writing is far more efficient than continual rewriting: it maintains a smaller, less redundant memory bank while increasing TRS-update coverage.
We quantify accuracy gain over no-memory inference for ten post-hoc atomic spatial abilities, averaged across four base models on the common RoboSpatial, ERQA, SAT, and EmbSpatial scope. The chart compares SMA and MemP for each ability.
Figure 7. Mean atomic-ability gains over No memory.
Atomic gains are broad rather than isolated: SMA improves every atomic ability on average, with the largest lift on correspondence-style spatial checks.
Figure 8. Qualitative examples of retrieved high-TRS procedure memories across five spatial reasoning cases.
@misc{zhang2026spatialmemoryagent,
title = {Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence},
author = {Zhang, Haokai and Ding, Yuhang and Zhou, Yunshu and Du, Xinze and Zhang, Shengtao and Zhao, Zhiyue and Xi, Yuling and Chen, Hao},
year = {2026},
eprint = {2608.12743},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2608.12743}
}