Beyond Realism: Making Digital Humans Socially Intelligent

Beyond Realism: Making Digital Humans Socially Intelligent

With recent advancements in computer graphics, 3D digital humans have achieved an impressive level of visual realism. They are increasingly being integrated into diverse applications, including video games, customer service and finance chatbots, educational and healthcare simulations, remote communication, and social extended reality (XR).

However, their ability to interact and move naturally within social contexts remains limited. As humans, we are highly attuned to non-verbal behaviors in emotional and social interactions. For digital humans to engage with us more naturally, they must be equipped with non-verbal communication skills such as facial expressions, gestures, and gaze. As they are deployed in more interactive environments, the demand to generate their behavior automatically and in real time is growing. Yet, capturing and synthesizing the nuanced, individual nature of non-verbal behaviors remains a significant challenge, hindered by limitations in data availability, the complexity of algorithms, and evaluation methodologies.

In this talk, I will explore how AI and deep learning techniques—particularly those leveraging motion capture technology—can be used to model and generate the non-verbal behaviors of digital humans. I will present the state-of-the-art, highlight our recent research, and offer a critical examination of current evaluation practices in non-verbal behavior synthesis.