Abstract
Complex robotic manipulation tasks frequently require a long-term memory of past events and actions. As conditioning on full histories renders policies prone to spurious correlations and degrades performance, many approaches to policy memory involve compressing historical information through expensive VLM queries in-the-loop to process only task-salient information. In this paper, we propose an alternative approach in which computationally intensive VLM queries are made during train-time to learn a lightweight latent memory that can be efficiently queried at deployment time. Our representation, which we call the workspace token, is trained by (1) using a VLM to identify current and historical information necessary for completing a task, then (2) distilling these into the workspace token using a set-reconstruction decoder loss. In both simulation and hardware, we show that the workspace token can be used as a drop-in replacement for observations during deployment, enabling policies to solve memory-intensive tasks without the need for VLM reasoning in-the-loop, in effect serving as a latent harness for distilling a stronger reasoning models ability to solve long-horizon tasks to a reactive robotic policy. We further demonstrate that the workspace tokens are not only more lightweight, but also lead to better policy performance compared to conditioning policies on explicit modalities like curated past image frames, motivating a latent approach to history curation and reasoning model harnesses more broadly.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Dashora, N., Chen, D., Shenfeld, I., Marangola, J., Agrawal, P., & Simchowitz, M. (2026). Workspace Models: Lightweight Robotic Memory via Saliency-Driven Supervision. https://omanscience.com/en/articles/workspace-models-lightweight-robotic-memory-via-saliency-driven-supervision
MLA 9
Dashora, Nitish, et al. "Workspace Models: Lightweight Robotic Memory via Saliency-Driven Supervision." https://omanscience.com/en/articles/workspace-models-lightweight-robotic-memory-via-saliency-driven-supervision.
Chicago (author–date)
Dashora, Nitish, Douglas Chen, Idan Shenfeld, John Marangola, Pulkit Agrawal, and Max Simchowitz. 2026. "Workspace Models: Lightweight Robotic Memory via Saliency-Driven Supervision." https://omanscience.com/en/articles/workspace-models-lightweight-robotic-memory-via-saliency-driven-supervision.
Harvard
Dashora, N., Chen, D., Shenfeld, I., Marangola, J., Agrawal, P. and Simchowitz, M. (2026) 'Workspace Models: Lightweight Robotic Memory via Saliency-Driven Supervision', Available at: https://omanscience.com/en/articles/workspace-models-lightweight-robotic-memory-via-saliency-driven-supervision.
Vancouver
Dashora N, Chen D, Shenfeld I, Marangola J, Agrawal P, Simchowitz M. Workspace Models: Lightweight Robotic Memory via Saliency-Driven Supervision. https://omanscience.com/en/articles/workspace-models-lightweight-robotic-memory-via-saliency-driven-supervision
IEEE
N. Dashora, D. Chen, I. Shenfeld, J. Marangola, P. Agrawal, and M. Simchowitz, "Workspace Models: Lightweight Robotic Memory via Saliency-Driven Supervision," https://omanscience.com/en/articles/workspace-models-lightweight-robotic-memory-via-saliency-driven-supervision.