Articulab SCIPR — Sensing Curiosity in Play and Responding
A mixed-reality educational system using virtual human agents, augmented reality toys, and multimodal sensing to foster curiosity and exploration in elementary school students. John Choi built the virtual peer's gesture/expression system and an automatic fiducial-marker card tracking system that recognizes 27 marker positions at 5 FPS.
Overview
SCIPR (Sensing Curiosity in Play and Responding) is a project of the Articulab at Carnegie Mellon University, led by Professor Justine Cassell and Zhen Bai. It develops technologies to foster curiosity, exploration, and self-efficacy in elementary school students, combining a role-playing game, a virtual peer, multimodal sensing, and augmented reality toys.
Curiosity is essential to science learning — it motivates students to explore and produce knowledge as scientists rather than passive consumers. SCIPR addresses the “teach to the test” problem by creating a playful mixed-reality environment where curiosity is actively triggered and sustained.
The system has four integrated components:
- Role-Playing Game — Outbreak (designed by the OH! Lab): a card game where children switch roles between question-asker and question-answerer to explore science concepts
- Virtual Child Peer — an AI agent named Jayden that models exploratory and collaborative learning, scaffolding curiosity in real children
- Multimodal Sensing (MultiSense) — automatically recognizes children’s engagement, curiosity, and learning states from facial expression, eye gaze, head gesture, and voice quality
- Mixed-Reality AR Toys — curiosity-evoking augmented reality objects that let real and virtual children play together in shared physical/digital space
My Work
My work for the SCIPR project was divided into two parts:
1. Virtual Peer Character System Built the virtual peer’s full range of gestures and expressions using BML (Behavior Markup Language) scripts — mapping verbal behaviors (suggest card, positive evaluation, hypothesize, etc.) to corresponding nonverbal behaviors (gaze at players, shrug, hand on chin, raise hand, head nod, etc.) for natural synchronized animation.
2. Automatic SCIPR Game Tracker Replaced the human observer who previously tracked all card game elements manually with a fully automated computer vision tracking system:
- Tracks Gear Cards and Battery Tokens in real time using fiducial marker CV
- Recognizes 27 fiducial marker positions and orientations simultaneously
- Runs at 5 FPS with full positional and rotational tracking
- Feeds into a real-time AI Game Reasoner that makes gameplay decisions
A custom 3D-printed player robot game piece was also designed for the physical game.
Team
- Justine Cassell — Principal Investigator
- Zhen Bai, Bhargavi Paranjape, Tanmay Sinha, David Slebodnick, Luo Yi Tan — ArticuLab
- Oberlin Wintern 2016: Patrice DiChristina
- Spring 2016 Interns: Tristan Marino, Aliya Blackwood
- Spring 2016 Capstone: Su Baykal, Benjamin Boesel, Ryan Donegan, Sam Gao, Ian Go
- Summer 2016 Interns: Jiamin Ping, Lu Sun, Shrey Shah, Bo Kim, Hyunji Do, Julia Luo, Sam Li, Stephanie Bao, Sina Siddiqi
- Fall 2016 Interns: Zoey Feng, Angela Liang, Michelle Deng, Hea Jin Cha, Jarret Lin, Changning Shou, Gaurav Lahiry, Olamitundun Oladipo
- Spring 2017 Interns: Lu Sun, Danny Choo, Zoey Feng, Angela Liang, Michelle Deng, Lauren Yan, Kaily Bruch, Prachi Mahableshwarkar, Kanisha Vaughn, Vivian He
Funding
Generously supported by the Heinz Family Foundation and the National Science Foundation (NSF).
Gallery
