Researching Emotion and Gesture

Communication is the exchange of a message between a sender and a receiver, with every message carrying both content and feeling. This research began by examining human emotions through the lens of energy and valence. Due to our experimental approach, the work focused on the four main quadrants of emotion: Joy, Sadness, Anger, and Calm. The next step was to look into psychology and body language to map the common gestures associated with these emotions.

Developing the Recognition System

The first step in using the Leap Motion was determining the optimal placement and support for accurate hand tracking. Initial tests involved attaching the device to the user’s chest, aiming for a more comfortable and easily interchangeable setup. However, it soon became clear that the best way to achieve the intended results was to mount it on the user's head, ensuring a wider and more reliable field of view.

With the Leap Motion placement finalized, the focus shifted to developing the Deep Neural Network. After researching available resources, we adapted a GitHub library and created an interface, which analyzed four key movement parameters: symmetry, trajectory spread, spatial extent, and energy. The network was trained using sequences of repeated gestures to ensure accuracy while minimizing detection noise. Once trained, the system would only validate an emotion if its confidence level exceeded 50%, triggering the corresponding visual and audio response.

Designing Expressive Avatars

The avatars design focused on creating a universally adaptable form that could effectively convey emotions. A flame-like shape with a simple face was chosen to maintain an approachable feel. Research on color psychology, shapes, and cultural perceptions guided the design choices. The avatars were color-coded based on the arousal circle of color. To enhance emotional impact, additional elements like waves were used to differentiate each emotion, balancing smooth, friendly movements with sharper, more intense features where needed. Facial expressions were carefully crafted based on established studies, ensuring clear, universally recognizable emotions through subtle variations in eyes and mouth orientation.

The avatar’s audio was composed based on cross-cultural studies on music and emotion. Key audio descriptors—such as melody, pitch variation, tempo, rhythm complexity, and amplitude—were used to align with universal emotional responses. These principles ensured that each avatar's sound design reinforced its visual representation.

Bringing the System to Life

With all system components developed, we integrated them into Unity. Since the final output was a holographic display, we used Unity's canvas to render views from four cameras, creating the necessary 3D visualization. Finally, an algorithm was implemented to trigger videos and sounds based on the Deep Neural Network’s detected emotions, with each emotion consisting of three animations: fade-in, loop, and fade-out.

Although the experience revolves around a Natural User Interface (NUI), human emotional expression is largely unconscious. To help users become more aware of their own behaviors before interacting with the system, we gamified the introduction phase. Users first picked a card depicting an emotion alongside insights into its typical expressions. This step primed them for the main interaction.

Once equipped with the headband-integrated Leap Motion sensor, users performed various gestures, seeing and hearing their movements translated in real time into the avatar. The holographic display was chosen not only for its wider visibility but also to enhance the “wow” factor and playful nature of the experience.