My research interest lies at the intersection of machine learning, computer vision, computational linguistics and signal processing — building the computational foundations to enable computers with the abilities to analyze, recognize and predict subtle human communicative behaviors during social interactions. Human face-to-face communication is a little like a dance: participants continuously adjust their behaviors based on their interlocutor’s speech, gestures and facial expressions during social interaction. The actual sophistication of human communication comes to the fore when we try to create computers that can understand and participate, however crudely, in this type of social interaction.

Human Communication Dynamics
I formalize this new research endeavor with a human communication dynamics framework, addressing four key computational challenges: behavioral dynamics to model the appearance and temporal variations of individual communicative behaviors and their effects on perceived meanings; multimodal dynamics to model the interdependence between different communicative channels including visual gestures and expressions, language, and acoustic signals; interpersonal dynamics to model the social and conversational influence between participants during dyadic or small-group interactions (i.e., micro-level); and societal dynamics to model the cultural and behavioral changes in a larger groups (i.e., meso-level) or in different societies (i.e, macro-level).

Multimodal Machine Learning
Central to this research effort is the introduction of new probabilistic models that can learn temporal and fine-grained dependencies across behaviors, modalities and interlocutors. These energy-based probabilistic models must address core computational issues with human communication dynamics, including hierarchical and nonlinear feature representation, multiview learning and potential over fitting on smaller training sets. For example, I created a family of latent probabilistic models (HCRF, LDCRF, CCNF, etc.) designed to automatically learn the hidden dynamics present in human verbal and nonverbal communication.

Health Behavior Informatics
This research has many applications in education (learning analytics), business (negotiation, interpersonal skills training) and social multimedia (opinion mining, social influence). One area I am particularly excited about is the development of new decision support tools for healthcare applications. For example, how can we quantify, analyze and summarize patient verbal and nonverbal behaviors during psychotherapy? This information could not only help clinicians between therapy sessions but also facilitate collaboration with other medical team members by providing objective behavior measures for the patient’s medical record.