Скачать или смотреть Hand Tracking for Mobile Virtual Reality

This is a presentation from the defence of my Bachelor thesis - Hand Tracking For Mobile Virtual Reality.

Demo app APK: https://drive.google.com/file/d/1xpGG...


Full source code and text of the thesis are available at https://github.com/AlexFSmirnov/andro...

Abstract:
Mobile Virtual Reality inspired by Google Cardboard (where a smartphone is placed in a cheap headset with no electronics) has lost its popularity after standalone VR headsets appeared on the market. The devices like Oculus Quest support full tracking of the real world position and rotation of the headset and a pair of controllers (or even just the user’s hands), all without any external sensors or computers, while Cardboard-like applications only track the rotation of the user's head. We tried to create a framework that could provide a similar experience to the standalone headsets - a mobile VR application capable of tracking itself and the user's hands in 6 degrees of freedom, all while only requiring a smartphone with a few cheap additions. We used Unity AR Foundation to achieve headset tracking and a variety of OpenCV algorithms to solve hand tracking - ArUco markers, Color Thresholding, Camshift, and deep learning approaches like OpenPose and YOLOv3. Our main focus was on testing the hand tracking algorithms, and we discovered that while they are far from perfect, the concept is feasible, and, with some improvements, the framework could become a real competitor in the space of standalone VR headsets.