Draw - Talk - Move
Multimodal AI - Robotic Input / Output System, 2024
Interaction Design, HCI Research
Class project for : Drawing++ taught by Zach Lieberman
Let’s say I draw a circle and say, “This is me,” then draw a rectangle next to it and say, “This is my apartment.” If you saw me drawing those shapes and heard what I said, you’d remember the circle as a person and the rectangle as a house.
But if someone else walks in later and sees the paper without having seen or heard the context, they’ll have no idea what the shapes mean.
But if someone else walks in later and sees the paper without having seen or heard the context, they’ll have no idea what the shapes mean.
don’t just draw,
don’t just talk,
don’t just move
Draw-Talk-Move.
don’t just talk,
don’t just move
Draw-Talk-Move.
When we were a kid,
we already knew how to draw
we already knew how to draw
We also knew how to move objects and create stories out of them.
Can we draw, talk, and move all at the same time? Can our drawings understand what we drew — and move like the subject they represent?
Final Diagram
Transforming to this
How it works
2. Talk to the Drawing - like, “This is a pingpong ball”
3. Attach the drawing to the robot arm
4. Talk to or move the robot arm - like, “Drop the ball please!”
5. Then, it should bounce as if there is an invisible floor
6. OR, you can add another drawing or object - like, bring a pingpong paddle and touch the drawing attached to the robot arm
7. Then, it should bounce off from the paddle
1. Draw
2. Talk to the Drawing - like, “This is a pingpong ball”
3. Attach the drawing to the robot arm
4. Talk to or move the robot arm - like, “Drop the ball please!”
5. Then, it should bounce as if there is an invisible floor
6. OR, you can add another drawing or object - like, bring a pingpong paddle and touch the drawing attached to the robot arm
7. Then, it should bounce off from the paddle
Demo - MIT Media Lab, with UR Robot Arm at the Future Sketches Group
Now here, I’m drawing a circle and saying to the interface that it’s a steel ball.
Then it will fall as if it is a heavy steel ball
Then when I say it’s light as feather, it will fall as if it’s feather
We can make the interface take on a character — like a cat, for example.
It’s saying ‘Yes’
Now it’s a No
Then, we can draw on our own face and talk to the LLM — in this case, Google Gemini. Based on the drawings and the image of the person, it will generate the robot’s movements.
(Disclaimer: The robot demo shown below was created using hard-coded animations, not generated by the LLM.)
Process and References
Original Idea sktech
Bret Victor – Stop Drawing Dead Fish, 2015
Karl Toby Rosenberg et al. – Draw Talking, 2023
Bill Verplank
Ernst Caramelle 'Video-Ping Pong' 1974
Jinha Lee – ZeroN, 2012