Articulab RAPT — Rapport-Aware Peer Tutor

Overview

RAPT (Rapport-Aware Peer Tutor) is a virtual partner developed in the ArticuLab at Carnegie Mellon University that students can collaborate with, and which responds to and helps foster socio-emotional awareness — building students’ ability to communicate and collaborate effectively while improving their learning outcomes.

Individual instruction from a private tutor is the most effective form of learning, yet not every student has access to one. RAPT addresses this gap using cutting-edge socially-aware AI: the system models students’ knowledge and the rapport that develops between the student and the virtual partner over time, then reasons about the most pedagogically and socially beneficial response to maximize both learning and rapport.

Currently running studies with 6th–8th grade students evaluating the system’s impact on self-efficacy, motivation, engagement, and Algebra knowledge.

My Work

My work for the RAPT project was primarily focused on overhauling the virtual agent’s graphical fidelity using modern Unity real-time rendering techniques — transforming the virtual classroom environment from PS2-era graphics to PS4 quality.

Improvements included:

Populating the scene with more books, computers, lamps, plants, globes, and classroom objects
More natural and uniform lighting
Glass shaders on beakers
Skybox with dynamic clouds
Bloom on the sky
Motion blur
Soft shadows
Anti-aliasing
Ambient occlusion
Subtle color grading
Textures on all scene objects
Screen space reflections on cabinets
Subsurface scattering on skin materials

The new and improved RAPT environment is shown side-by-side with the original in the gallery above, clearly highlighting the graphical improvements.

System Architecture

The RAPT system contains five AI modules:

Computational model of rapport — explains how dyadic interactions build, maintain, and destroy rapport through conversational strategies (Zhao et al., 2014)
Conversational strategy classifier — recognizes self-disclosure, praise, shared experience, back-channel, and other social strategies with >80% accuracy
Rapport level estimator — uses temporal association rule learning on visual (eye gaze, smiles) and verbal behaviors to estimate rapport every 30 seconds (IVA 2016 Best Student Paper)
Social reasoner — spreading activation network that selects the next conversational strategy given rapport level, tutoring goals, and context
NLG + nonverbal generation — BEAT (Behavior Expression Animation Toolkit) generates BML behavior plans executed by SmartBody on the virtual agent

Team

Justine Cassell, Amy Ogan, Louis-Philippe Morency — Principal Investigators
Michael Madaio, Yoichi Matsuyama, Robert Huerbin, David Slebodnick, Tanmay Sinha, Ran Zhao — ArticuLab
Research Assistants: Michelina Astle, Jake Beley, Rishabh Chatterjee, Naomi Eigbe, Alvaro Granados, Rae Lasko, Tiffany Lee, Sarah Lehman, Jeffrey Li, William Liu, Kun Peng, Lynnette Ramsay, Anne Widom, Caroline Wu, Zian Zhao

Madaio, Cassell & Ogan (2017). The Impact of Peer Tutors’ Use of Indirect Feedback and Instructions. CSCL 2017 — Best Student Paper
Madaio, Ogan & Cassell (2017). Using Temporal Association Rule Mining to Predict Dyadic Rapport in Peer Tutoring. EDM 2017
Zhao, Sinha, Black & Cassell (2016). Socially-Aware Virtual Agents: Automatically Assessing Dyadic Rapport from Temporal Patterns of Behavior. IVA 2016 — Best Student Paper
Matsuyama, Bhardwaj, Zhao, Romero, Akoju & Cassell (2016). Socially-Aware Animated Intelligent Personal Assistant Agent. SIGDIAL 2016

Funding

Supported by the National Science Foundation Cyberlearning Award No. 1523162 and the Institute of Education Sciences, U.S. Department of Education (Grant R305B150008) to Carnegie Mellon University. Computing power by DroneData.