Though virtual reality has become pretty ubiquitous over the last year, the question of making individuals aware of these experience without the need for a headset still remains. And that’s where mixed reality comes in the picture. But, not only does it require bulky green screen setups and other technical equipment but you’re also not able to fully interact or immerse in the experience.
Compared to physical reality, sharing these experiences with others can be difficult, as VR headsets make it challenging to create a complete picture of the people participating in the experience.
The said technique is the solution for alleviating the quirks in user experience but the headset seems to constantly get in the way. The viewers can only see the huge packed mass of silicon and plastic on your face rather than your gaze and face expressions. Here, that is the primary pain point and Google is working on a solution for the same. It has today shared some of its findings through a couple of blog posts, which add,
Even though MR facilitates sharing, the headset continues to block facial expressions and eye gaze, presenting a significant hurdle to a fully engaging experience and complete view of the person in VR.
According to a blog post, Google Machine Perception team and Daydream Labs have teamed up to make this a reality. These two have devised a way to remove VR headsets and finally show the true identity of an individual behind them. But, the process has still not been simplified but instead requires you to have even more high-tech and bulky equipment, as well as developer knowledge. This procedure requires the company to use machine learning, 3D computer vision, and advanced rendering techniques.
To remove the VR headset from a person’s bust in the real world, which is being recorded for viewing at a later stage, the researchers first need to conduct a camera scan of the user’s face. This scan will then be used to reconstruct a dynamic 3D-rendered model of the user’s face, which blinks and moves eyes in different directions. This is the basis of being able to subtly remove the VR headset from the user’s face.
Next, you will still be requiring a VR headset, possibly an HTC Vive retrofitted with eye-tracking technology, to be able to implement this technique. The eye-tracking technology will enable the headset to detect the positioning of the user’s gaze. Then, the researchers use each of their individual entities to create a fulfilling 3D model of the user’s face. It doesn’t completely remove the VR headset from the picture but instead replaces it with a transparent version of the same. Describing the same, the blog post says,
The core idea behind our technique is to use a 3D model of the user’s face as a proxy for the hidden face. This proxy is used to synthesize the face in the MR video, thereby creating an impression of the headset being removed.
The said techniques haven’t yet been perfected by the researchers at Google, but the tech giant is currently exploring ways to make this accessible to YouTube content creators. It has also partnered with YouTube Spaces to conduct all of its experiments at their locations.
This technique makes the technology feel more immersive and allows others to comprehend your emotions through face expressions rather than just your voice. It could become an important limb of the VR experience going forward but we still need to work more on making the experience less alien and dizzy.
