We developed a system that controls devices through eye tracking. The model was trained on synthetic data from a VR environment, which cut labeling costs severalfold.
Transferring a model trained on VR data to real devices without losing accuracy (domain adaptation).
We split development into two consecutive stages. First, we ran a short study of existing HCI (human-computer interaction) techniques for tracking pupil movement in real time and classifying its offset from center. Based on it, we chose an imitation learning model, where training data is generated from a different modality. In the second stage, we generated data in a VR simulator to train and validate the model. This stage also involved work with specialized hardware and deploying the solution to the client’s edge devices.
- 0101
Interaction research
Analyzing scenarios for controlling devices with pupil movement.
- 0202
VR simulation
Building a virtual environment to generate training scenarios and data.
- 0303
Eye-tracking data collection
Building a dataset of gaze movements and user behavior.
- 0404
AI model development
Training an imitation learning model on synthetic data.
- 0505
Device integration
Connecting the control system to interfaces and embedded hardware.
- 0606
Scenario testing
Verifying the system’s real-time stability and accuracy.
- 01Python
- 02PyTorch
- 03OpenCV
- 04MediaPipe
- 05AR Haptics
- 06VR SDK
- 07Imitation Learning
- 08React
Have a similar challenge?
We’ll review it and estimate within 5 business days.