Abstract
Vision-Language-Action (VLA) models rely strongly on language for describing task information, despite having multimodal inputs. We hypothesize that other modalities in the state space may present opportunities for supplemental task conditioning, which may be particularly relevant in cluttered or otherwise ambiguous scenes. We introduce two tuned models to test this hypothesis: (1) an electrophysiology-conditioned VLA (EC-VLA) that incorporates 8-channel electromyography envelopes as continuous conditioning input concatenated to the proprioceptive vector, and (2) a visually-annotated VLA (VA-VLA) that incorporates visual segmentation annotations to the image inputs. On a cube-selection task evaluated across three participants, EC-VLA matches a language-prompted baseline in uncluttered, in-distribution conditions and substantially outperforms it in cluttered, out-of-distribution scenes. Similarly, VA-VLA shows modest improvements over a language-prompted baseline in in-distribution scenes with substantial improvement in cluttered, out-of-distribution trials. Together, these results provide strong evidence for the potential benefit of task-conditioning beyond language.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Staley, E. W., Pyles, C. O., Hingorani, R., Camargo, F., Milsap, G., Markowitz, J., Fifer, M. S., & Wolmetz, M. (2026). Continuous Conditioning of VLAs with Augmenting EMG and Visual Task Descriptors. https://omanscience.com/en/articles/continuous-conditioning-of-vlas-with-augmenting-emg-and-visual-task-descriptors
MLA 9
Staley, Edward W., et al. "Continuous Conditioning of VLAs with Augmenting EMG and Visual Task Descriptors." https://omanscience.com/en/articles/continuous-conditioning-of-vlas-with-augmenting-emg-and-visual-task-descriptors.
Chicago (author–date)
Staley, Edward W., Connor O. Pyles, Rahul Hingorani, Frank Camargo, Griffin Milsap, Jared Markowitz, Matthew S. Fifer, and Michael Wolmetz. 2026. "Continuous Conditioning of VLAs with Augmenting EMG and Visual Task Descriptors." https://omanscience.com/en/articles/continuous-conditioning-of-vlas-with-augmenting-emg-and-visual-task-descriptors.
Harvard
Staley, E. W., Pyles, C. O., Hingorani, R., Camargo, F., Milsap, G., Markowitz, J., Fifer, M. S. and Wolmetz, M. (2026) 'Continuous Conditioning of VLAs with Augmenting EMG and Visual Task Descriptors', Available at: https://omanscience.com/en/articles/continuous-conditioning-of-vlas-with-augmenting-emg-and-visual-task-descriptors.
Vancouver
Staley EW, Pyles CO, Hingorani R, Camargo F, Milsap G, Markowitz J, et al. Continuous Conditioning of VLAs with Augmenting EMG and Visual Task Descriptors. https://omanscience.com/en/articles/continuous-conditioning-of-vlas-with-augmenting-emg-and-visual-task-descriptors
IEEE
E. W. Staley, C. O. Pyles, R. Hingorani, F. Camargo, G. Milsap, J. Markowitz, M. S. Fifer, and M. Wolmetz, "Continuous Conditioning of VLAs with Augmenting EMG and Visual Task Descriptors," https://omanscience.com/en/articles/continuous-conditioning-of-vlas-with-augmenting-emg-and-visual-task-descriptors.