concept

Designing embodied interfaces for real-time GenAI

Using a Research through Design approach, this project investigates how people make sense of real-time generative AI across different embodied modalities. Key questions include:

- How do users interpret the relationship between their actions and AI responses?
- Is the AI experienced as a tool, collaborator, or unpredictable partner?
- How do participants develop strategies through play, exploration, or control?
- How do different modalities shape expressiveness, predictability, and engagement?

Tangible: clay-based terrain building

Participants sculpt with colored clay on a tabletop, where the system analyzes color, shape, height, and spatial arrangement to generate evolving AI landscapes such as mountains, vegetation, and bodies of water.

Auditory: sound-based generation

Participants use musical instruments and sound to influence the AI-generated environment, where variations in pitch, rhythm, and tempo transform elemental qualities such as earth, water, and fire within the scene.

Gestural: hand-tracked creatures

A webcam tracks hand gestures to generate mythical creatures in different ways based on participants’ movements. Different finger represents a different creature type or characteristic.

Prototyping process

Evolving tangible interfaces for AI co-creation

I explored different tangible approaches for co-creating with AI, starting with simple building blocks that the system identified by shape. To make the interaction more intentional and expressive, I later introduced color-coded materials representing different terrain elements and environmental features.

Embedding prompting into embodied interaction

Beyond generating visuals from what appeared on screen, I wanted participants to shape the AI prompt directly through interaction itself. For example, in the gesture prototype, different finger combinations act as prompt modifiers, where each bent finger represents a different mythical creature or characteristic.