Vision-Language Models for Navigation and Manipulation (VLMNM)

Introduction

---

With the rising capabilities of LLMs and VLMs, the past two years have seen a surge in research work using VLMs for navigation and manipulation. Fusing the capabilities of visual interpretation with natural language processing, these models are poised to redefine how robotic systems interact with both their environment and human counterparts. The relevance of this topic cannot be overstated; as the frontier of human-robot interaction expands, so does the necessity for robots to comprehend and operate within complex environments using naturalistic instructions. Our workshop will not only reflect the state-of-the-art advancements in this domain, by featuring a diverse set of speakers, from senior academics to researchers in early careers, from industry researchers to companies producing mobile manipulation platforms, from researchers who are enthusiastic about using VLMs for robotics to those who have reservations about it. We aim for this event to be a catalyst for originality and diversity at ICRA 2024. We believe that, amidst a sea of workshops, ours will provide unique perspectives that will push the boundaries of what's achievable in robot navigation and manipulation.

In this workshop, we plan to discuss:

• How can VLMs/LLMs enhance robotics navigation and manipulation?
• How to extract world knowledge from pre-trained VLMs/LLMs and apply them to navigation and manipulation?
• How to integrate VLMs/LLMs with robot components, such as perception, control, and planning? How to account for partial observability and uncertainty?
• Benchmarks and datasets to assess the generalization capabilities of VLMs/LLMs for navigation and manipulation.
• Capabilities and limitations of VLMs/LLMs for navigation and manipulation (e.g. in task planning, spatial understanding)
• New interaction modes between robots and humans enabled by VLMs/LLMs.

All accepted workshop papers: OpenReview. Please bring a poster. We will have double-sided A0 portrait poster boards.

FAQ

---

Final Schedule

---

Call for Papers

---

We invite submissions including but not limited to the following topics:

• Applications:
• Integration of VLM/LLMs for manipulation and navigation
• VLM/LLMs for perception/scene understanding/state estimation
• VLM/LLMs for control/skill learning/motion generation
• VLM/LLMs for decision-making/reasoning/planning
• VLM/LLMs as world models
• VLMs/LLMs for multimodal task specifications
• VLMs/LLMs for human-robot/robot-robot interactions
• VLMs/LLMs for scene and task generation
• New Capabilities:
• Open-vocabulary perception/navigation/manipulation
• Commonsense reasoning with VLM/LLMs
• Generalization to unseen object categories, environments, and tasks
• Bootstrapping learning from scarce data
• Natural language interaction with everyday users
• Datasets/Benchmarks:
• Internet-scale data for training robotics foundation models
• Mobile manipulation benchmarks for VLM/LLM-based systems
• Limitations:
• Failure modes of VLM/LLMs
• Robustness of VLM/LLMs
• Certifiabilities of VLM/LLMs

Submissions should have up to 4 pages of technical content, with no limit on references/appendices. Submissions are suggested to follow the ICRA double-column format with the template available here. We encourage authors to upload videos, code, or data as supplementary material (due on the same day as the paper). Following the main conference, our workshop will use a single-blind review process. We welcome both unpublished, original contributions and recently published relevant works. Accepted papers will be presented as posters or orals and made public via the workshop’s OpenReview page with the authors’ consent. We strongly encourage at least one of the authors to present on-site during the workshop. Our workshop will feature a Best Paper Award.
Important Dates:

• Submission portal opens: January 29, 2024
• Paper submission deadline: March 11, Monday, 2024 (AoE)
• Notification of acceptance: March 29, 2024 (Results viewable on OpenReview) April 1, 2024 (Announcing Spotlights)
• Camera-ready deadline: April 26, 2024
• Workshop @ ICRA 2024: May 17, 2024

Organizers

---

Chris Paxton FAIR, Meta	Fei Xia Google Deepmind	Karmesh Yadav Georgia Tech	Nur Muhammad Mahi Shafiullah New York University
Naoki Wake Microsoft Research	Weiyu Liu Stanford University	Yujin Tang Sakana AI	Zhutian Yang MIT, NVIDIA Research

Contact

---

For further information or questions, please contact vlm-navigation-manipulation-workshop [AT] googlegroups [DOT] com

Access Map

---