Comprehensive Summary

This research presents a multimodal framework for integrating full-body tracking (FBT) and eye tracking (ET) into a standalone VR application on the Meta Quest Pro, designed to support objective and scalable Autism Spectrum Disorder (ASD) screening. The core technical challenge addressed was the fundamental conflict between Meta Movement SDK’s estimation-based skeletal model and OSC-based external body tracking from SlimeVR — which caused proportion distortion and misalignment when used together. To resolve this, the research replaced Meta’s FBT skeleton with the Virtual Motion Capture (VMC) protocol as the primary framework, integrating SlimeVR for full-body kinematics, OpenXR for head tracking, and Meta’s SDK exclusively for eye and hand tracking — enabling seamless multimodal data capture in a fully wireless, PCVR-independent standalone build.

The resulting system captures three interconnected streams of behaviorally relevant data: full-body kinematic data via SlimeVR IMU sensors at 100Hz for motor planning and postural analysis, eye tracking data via Meta Quest Pro at 90Hz for gaze pattern and area-of-interest (AOI) assessment, and task interaction logs for behavioral sequence and cognitive flexibility evaluation — all mapped directly to established DSM-5 ASD diagnostic criteria. The framework was validated through preliminary testing demonstrating real-time avatar synchronisation, accurate AOI detection, and gaze heatmap visualisation, with minor challenges in height calibration and bone retargeting documented and resolved. Developed in collaboration with Malaysia’s WAFA Therapy Center, the system is positioned as a portable, clinically grounded diagnostic tool, with future phases planned to incorporate ECG heartbeat tracking, facial expression analysis, and formal comparative studies against established tools such as ADOS-2.

Project Justification

Autism Spectrum Disorder (ASD) affects millions of children worldwide, yet current diagnostic methods — including the Autism Diagnostic Observation Schedule (ADOS) and the Modified Checklist for Autism in Toddlers (M-CHAT) — remain time-intensive, heavily reliant on subjective parental input, and frequently inadequate in identifying milder symptoms, often resulting in delayed diagnoses and missed early intervention windows that are critical to improving long-term developmental outcomes. Existing clinical tools lack the ability to capture objective, quantifiable behavioral data in a controlled yet naturalistic environment, creating a significant gap between the richness of behavioral information that ASD children exhibit and what traditional assessment methods can reliably measure. This project was developed to address that gap directly — by leveraging the immersive and controllable nature of Virtual Reality on the Meta Quest Pro platform, combined with high-precision full-body tracking via SlimeVR and eye tracking through Meta’s Movement SDK, the framework enables the simultaneous, real-time capture of motor patterns, gaze behaviour, and task interaction sequences that directly correspond to the core diagnostic criteria defined in DSM-5. The result is a portable, wireless, and clinically grounded screening tool capable of delivering the kind of objective, multimodal behavioral data that current diagnostic practices cannot, bringing earlier, more accurate, and more accessible ASD identification within reach for clinical and research settings alike.

Objectives

1. To design and develop a standalone multimodal VR framework on the Meta Quest Pro that seamlessly integrates full-body tracking via SlimeVR, eye tracking via Meta Movement SDK, and head tracking via OpenXR — resolving the skeletal conflict between estimation-based and OSC-based tracking systems using the Virtual Motion Capture (VMC) protocol.

2. To capture and log objective, quantifiable behavioral data across three assessment modules — motor planning and postural analysis, behavioral sequence and repetitive pattern detection, and visual attention and gaze measurement — aligned directly to DSM-5 ASD diagnostic criteria for clinically meaningful screening.

3. To evaluate the technical feasibility and real-world performance of the multimodal framework through preliminary implementation testing, and establish a scalable, clinically grounded foundation for future formal validation against established ASD diagnostic tools such as ADOS-2.

Phases Of Work

Estimated Budget

Project Timeline