Mount Lebanon Virtual Pet
A holographic AI virtual pet dragon that lives in a school classroom, responding to touch via Leap Motion, speech recognition, and facial tracking — built for Mellon Middle School to support social-emotional learning.
Downloads
Overview
Mt. Lebanon Virtual Pet was a collaborative project between CMU’s Entertainment Technology Center, Mt. Lebanon School District, and Automadable, aimed at enhancing the student experience at Mellon Middle School using AI and emerging display technology.
The project created a holographic “virtual team pet” — Rufus, a chibi dragon — that “lives” within the school’s 8th grade Iron Brigade classrooms. Rufus embodies three core pillars:
- CASEL’s five components of Social-Emotional Learning (SEL)
- AMLE’s 16 Characteristics of Successful Schools
- Scientifically proven benefits of pet therapy
Middle school students make multiple transitions between classrooms each day, and the virtual pet serves as a consistent, engaging social anchor — helping students develop emotional regulation, focus, and a sense of connection to their school community.
Hardware Setup
The installation features a cutting-edge hardware stack:
- Looking Glass Hologram Display — A revolutionary 3D light field display rendering Rufus as a true volumetric hologram
- Leap Motion Controller — Tracks students’ hand gestures for real-time “petting” interactions
- Logitech Webcam — Enables live facial tracking so Rufus responds to who’s looking at him
- Intel NUC Mini PC — Compact, silent compute unit powering the whole experience
My Role
I served as Lead Software Developer of the Mt. Lebanon Virtual Pet project. I designed and built:
- Real-time Leap Motion hand tracking integration for touch/pet interactions
- Facial recognition pipeline for detecting and responding to student presence
- Speech recognition for voice-activated interactions with Rufus
- The full Looking Glass hologram rendering pipeline in Unity
- Reactive social-emotional behavior system — Rufus reacts differently to petting, tickling, and speech
Key Features
- Holographic 3D dragon rendered on a Looking Glass light field display
- Real-time hand gesture recognition via Leap Motion (petting, tickling)
- Live facial tracking — Rufus notices and responds to who’s in the room
- Speech recognition for interactive voice commands
- Reactive social-emotional behavior animations
- Designed to support SEL programming in a middle school environment
Partner Organizations
Carnegie Mellon University Entertainment Technology Center — The ETC is a premier professional graduate program for interactive entertainment, exploring transformational games, innovation by design, and interactive storytelling through interdisciplinary, design-based research projects.
Mt. Lebanon School District — A highly acclaimed public school system in Allegheny County, Pennsylvania, serving Mt. Lebanon — a suburb of Pittsburgh. The district has received multiple National Blue Ribbon School awards and was ranked 1st in Western Pennsylvania by the Pittsburgh Business Times.
Credits
- Jeffrey Halliday — Interdisciplinary Teacher, Mellon Middle School / Mt. Lebanon School District
- John Balash — Director of Educational Engagement at CMU Entertainment Technology Center
- John Choi — Lead Software Developer
Gallery
