Capstone Project: Autonomous Conversational Humanoid
The capstone project integrates everything you've learned across all four modules to build an autonomous humanoid robot capable of natural human interaction.
Project Overview
Build a humanoid robot system that can:
- Understand spoken commands using speech recognition (Module 4)
- See and identify objects using perception (Module 3)
- Plan and execute tasks using LLM-based planning (Module 4)
- Navigate safely using Nav2 (Module 3)
- Maintain conversation context across multiple turns (Module 4)
- Operate in simulation using Gazebo or Isaac Sim (Modules 2-3)
Project Requirements
Minimum Viable Product (MVP)
Your capstone must demonstrate:
| Requirement | Description | Module |
|---|---|---|
| Voice Control | Accept and transcribe voice commands | 4 |
| Object Detection | Identify at least 5 object types | 3 |
| Task Planning | Decompose commands into actions | 4 |
| Navigation | Navigate to specified locations | 3 |
| Manipulation | Pick and place objects | 1-3 |
| Dialogue | 3+ turn conversation with context | 4 |
Extended Goals
For exceptional projects:
- Multi-modal interaction (speech + gesture)
- Error recovery with user clarification
- Learning from user preferences
- Real robot deployment (if hardware available)
System Architecture
Project Phases
Phase 1: Foundation (Week 1)
- Set up development environment
- Import humanoid robot into simulation
- Verify ROS 2 communication
- Test basic motion control
Deliverable: Robot moving in simulation with working ROS 2 topics
Phase 2: Perception (Week 2)
- Configure camera sensors
- Set up object detection pipeline
- Implement localization (SLAM or known map)
- Create scene understanding node
Deliverable: Robot detecting objects and localizing in environment
Phase 3: VLA Integration (Week 3)
- Integrate speech recognition
- Set up LLM task planner
- Implement dialogue manager
- Connect perception to planning
Deliverable: End-to-end voice command to action
Phase 4: Polish & Demo (Week 4)
- Add error handling and recovery
- Implement multi-turn dialogue
- Test edge cases
- Prepare demonstration
Deliverable: Complete working system demonstration
Evaluation Criteria
| Criteria | Weight | Description |
|---|---|---|
| Functionality | 40% | Does the system work as specified? |
| Integration | 25% | How well do components work together? |
| Code Quality | 15% | Clean, documented, maintainable code |
| Presentation | 10% | Clear demonstration of capabilities |
| Innovation | 10% | Creative extensions beyond requirements |
Demo Scenarios
Your demonstration should include:
Scenario 1: Object Retrieval
"Robot, please bring me the red cup from the kitchen"
Expected behavior:
- Robot acknowledges command
- Navigates to kitchen
- Identifies red cup
- Picks up cup
- Returns to user
- Hands over cup
- Confirms completion
Scenario 2: Multi-Turn Interaction
User: "Find the blue ball" Robot: "I found it on the table" User: "Put it in the box" Robot: "Done. Anything else?" User: "No, thank you"
Scenario 3: Error Recovery
"Pick up the green cube" Robot: "I don't see a green cube. I see a red cube and a blue cube. Which one?"
Getting Started
Repository Structure
capstone/
├── launch/
│ └── full_system.launch.py
├── config/
│ ├── robot_config.yaml
│ ├── perception_config.yaml
│ └── dialogue_config.yaml
├── src/
│ ├── perception/
│ ├── planning/
│ ├── dialogue/
│ └── control/
├── worlds/
│ └── capstone_world.sdf
└── README.md
Starter Code
Begin with the code examples from each module:
- Speech:
docs/code-examples/module-4/speech/ - Planning:
docs/code-examples/module-4/llm-planning/ - Dialogue:
docs/code-examples/module-4/dialogue/ - Simulation:
docs/code-examples/module-2/
Resources
Submission
Submit your capstone project with:
- Code Repository: All source code with README
- Video Demo: 5-10 minute demonstration video
- Technical Report: Architecture decisions and lessons learned
- Presentation: 15-minute presentation of your system
Good luck! This project represents the culmination of your Physical AI & Humanoid Robotics journey.