TEI Prototype

Overview

TEI Prototype is a research game prototype built at the CMU School of Design with Haeyoung Kim for the ACM TEI (Tangible, Embedded and Embodied Interaction) conference.

Two players navigate a procedurally generated maze as physics fish. The twist: you control your fish not with a joystick, but with how loud you talk, how you move your body (tracked by an Azure Kinect), and how close you get to your partner. It’s the kind of game that’s impossible to play quietly or alone.

My Role

I was the Lead Unity Developer — took the research specs Haeyoung had designed and built the whole thing from scratch.

The interesting part was getting three completely different inputs — microphone audio, Kinect skeleton data, and keyboard — to all talk to each other cleanly. The mic pipeline was fun to build: it samples RMS volume, runs auto-calibration so it works in any room, applies attack/decay smoothing so fish speed feels fluid rather than jittery, and maps the result directly to locomotion. Speak up and your fish swims faster. Go quiet and it slows down.

The Kinect side maps hand position and pelvis height to the same movement axes as the keyboard, so players can switch between physical and keyboard control without breaking anything. I also added a proximity chest-touch ability swap — when the two players physically step close enough to each other, they can exchange abilities, which is a nice bit of embodied design that you can’t fake with a button press.

Rounding it out: a Prim’s-algorithm maze generator for fresh layouts each run, a shared FOV zoom budget that ties the two players together mechanically, and a CSV session logger for capturing playtesting data for the research paper. Also wrote the full developer documentation site so anyone running studies could calibrate the mic and understand the Kinect gestures without digging through code.

Credits

• John Choi — Lead Unity Developer, CMU School of Design
• Haeyoung Kim — Research & Design Lead, CMU School of Design

Built in Unity 2022.3+. Licensed under MIT.