Draw - Talk - Move



Multimodal AI - Robotic Input / Output System, 2024
Interaction Design, HCI Research
Class project for : Drawing++ taught by Zach Lieberman


 
Let’s say I draw a circle and say, “This is me,” then draw a rectangle next to it and say, “This is my apartment.” If you saw me drawing those shapes and heard what I said, you’d remember the circle as a person and the rectangle as a house.

But if someone else walks in later and sees the paper without having seen or heard the context, they’ll have no idea what the shapes mean.









don’t just draw,
don’t just talk,
don’t just move
Draw-Talk-Move.








When we were a kid,
we already knew how to draw








We also knew how to move objects and create stories out of them.









Can we draw, talk, and move all at the same time? Can our drawings understand what we drew — and move like the subject they represent?





Final Diagram








Transforming to this







How it works


1.  Draw

2.  Talk to the Drawing - like, “This is a pingpong ball”

3.  Attach the drawing to the robot arm

4.  Talk to or move the robot arm - like, “Drop the ball please!”

5.  Then, it should bounce as if there is an invisible floor

6.  OR, you can add another drawing or object - like, bring a pingpong paddle and touch the drawing attached to the robot arm

7.  Then, it should bounce off from the paddle






Demo - MIT Media Lab, with UR Robot Arm at the Future Sketches Group






Now here, I’m drawing a circle and saying to the interface that it’s a steel ball.




Then it will fall as if it is a heavy steel ball




Then when I say it’s light as feather, it will fall as if it’s feather




We can make the interface take on a character — like a cat, for example.
It’s saying ‘Yes’



Now it’s a No










Then, we can draw on our own face and talk to the LLM — in this case, Google Gemini. Based on the drawings and the image of the person, it will generate the robot’s movements.

(Disclaimer: The robot demo shown below was created using hard-coded animations, not generated by the LLM.)






Process and References



Original Idea sktech


Bret Victor – Stop Drawing Dead Fish, 2015


Karl Toby Rosenberg et al. – Draw Talking, 2023


Bill Verplank


Ernst Caramelle 'Video-Ping Pong' 1974


Jinha Lee – ZeroN, 2012