A high-end HMD perfect for safety-critical training simulations.
I love covering this industry, the pace of innovation is awesome, and watching companies large and small vie for market share with powerful and revolutionary tech is inspiring. We have come a long way. Better, lighter, untethered HMDs: Check. Cheaper, easier to use VR solutions: Check. Better access to good VR content: Check (mostly).
And now we have high-resolution VR with hand and eye-tracking which is an important step towards breaking down a barrier that has been holding VR back from broader adoption in some industries. Low-resolution VR might be fine for games and low interaction immersion, but when you are training for one of the most complicated and dangerous missions mankind has ever faced, every detail matters.
The latest HMD launched by Finnish company Varjo steps into that gap by offering industrial-grade “human-eye resolution” plus hand and eye-tracking. The company has just announced a new partnership with Boeing on a VR training program using its new VR-2 Pro headset for astronauts on the Boeing Starliner program preparing for missions to the International Space Station.
The VR-2 Pro is the latest professional iteration of Varjo’s initial VR-1 model and after extensive development with partner pilot companies including Audi, Volkswagen, and Saab, it features improved Ultraleap hand tracking, 20/20 Eye Tracker tech, and integration with affordable SteamVR content and peripherals as well as the OpenVR development platform from Valve.
“We are proud to be delivering the technology that is pushing industrial training applications to their furthest reaches – even to space,” said Niko Eiden, CEO and co-founder of Varjo. “With our devices, astronauts can see and virtually interact with the switches and control panels inside their Starliner capsule and read the real-time data on their crew displays. Advancements like this have the potential to transform the way any pilot is trained.”
The Boeing Starliner has been developed in partnership with NASA; when the first crewed mission aboard the CST-100 Starliner takes place in 2021, the crew will have already banked hundreds of training hours for each phase of the mission – including launching, docking, re-entering the atmosphere, and landing phases – using Varjo’s human-eye resolution VR devices. Chris Ferguson, the company’s first corporate astronaut, and Starliner commander will soon start using the Varjo VR training system regularly in preparation for the program’s Crew Flight Test, which will be the first Starliner mission with humans on board.
Imagine approaching the ISS in the Starliner spacecraft. Your speed is relatively low, but the ISS weighs about 420,000 kilograms – the equivalent of more than 330 cars. And even though the ISS is weightless in space, it still has mass. And the spacecraft needs to be steered to a fine point at the docking port by following a cone-shaped path. Projecting the display panels and trajectory data precisely is crucial if VR is to be an effective training tool for such a vital operation.
I‘ve recently managed to get some hands-on time with a VR-2 Pro test unit and seen for myself what a huge difference Varjo’s high visual fidelity makes when interacting with control panels and virtual objects. It’s easy to see how this seemingly simple advancement – namely the ability to read text, displays, and buttons as you would in real-life – is crucial in making it possible to train for dangerous and precise procedures such as docking a space ship from inside a complex cockpit.
The spacecraft’s crew console consists of two displays, each about the size of an iPad, which shows mission-critical flight data such as the velocity and trajectory of the aircraft as it moves in space. For the VR training to be effective, astronauts need to be able to read all the displays simultaneously while operating the simulated aircraft with their hands or controllers. With earlier VR headsets, reading the displays in the immersive training environment was only possible when leaning in very close to the displays – but then the astronauts couldn’t see their hands, making it unsuitable for training.
The most important psychological effect that this human-eye resolution, however, is ‘full immersion’ by which I mean how real-life levels of detail better convince the brain it is operating in a real-world situation so that your brain is not wasting any effort suspending disbelief.
It stands to reason that full immersion like this should vastly increase the effectiveness of group practice for unplanned events and working together in simulated dangerous situations, while building the team’s responses and decision-making abilities, without ever putting the astronauts in danger. The Starliner team can thus perform all the routine interactions, voice, and switch commands, while being immersed in the same surroundings they’d see when sitting in the actual spacecraft.
The Varjo VR-2 is so far the only device to deliver immersive content with this level of pixel-perfect clarity. This allows training for even the most challenging and safety-critical procedures of spaceflight, such as docking to the International Space Station, to be conducted in VR. Further, the system extends the astronaut training program by enabling training all the way through the two week-quarantine period in crew quarters before launch.
Before I dive into my experience of using the headset, let’s look at the device specs for the VR-2 Pro. The HMD features a next-generation Bionic Display™ with resolution at over 20/20 vision (over 60 PPD / 3000 PPI), flicker-free screen refresh at 60/90 Hz. Combining two 1920 x 1080 low persistence micro-OLEDs and two 1440 x 1600 low persistence AMOLEDs.
Basically this means that each eyepiece uses two screens, one higher resolution in the center and one that covers the peripheral vision. The two are cunningly blended with foveated rendering and optically transmitted through the carefully engineered glass lens in the headset. These combined optics feature a contrast ratio of (over 10,000:1) enabling the deepest blacks and individually calibrated colors for outstanding accuracy and image quality across an 87-degree field of view. The lens design combines different refractive index lenses for minimal color aberrations, zero ghost rays, and minimal reflections and each surface coating has been designed to maximize the brightness and clarity of the light needed to achieve human-eye resolution.
The real power of the Varjo solution, however, is released when this amazing image quality is combined with their advanced stereo 20/20 Eye Tracker™ and the integrated Ultraleap hand tracking which is faster and more accurate due to automatic user calibration. The hand tracking technology in the VR-2 Pro is especially impactful for interacting with physical controllers and switches as part of their VR training but also makes VR navigation and freehand tasks easy, delivering a natural and intuitive experience.
I tried out several demos which were impressive in very different ways and served to highlight various aspects and potential use cases for the tech. Surprisingly, the sheer impact of the VR-2 Pro’s resolution came across best in some of the most mundane settings. I was able to walk up and down a supermarket aisle checking out the products on the shelves, their prices, and label information while the eye-tracking monitored where my gaze was resting, highlighting which products I was actually examining at any given time.
I was blown away by how immersive that experience felt, perhaps because the setting was so familiar and relatable, as well as so beautifully reconstructed in VR. I could read every brand label and price tag from center aisle and as I approached the shelves it really felt like I could “reach out and touch” the various items. Tins, packets, and bottles of the familiar products were so crisp and well-lit I was staggered. Somehow even more impressive than this was the rendering of the shelves themselves and the packing crates the sugar was stacked on. The lighting, shadows, and textures were so well made I spent quite a while pacing up and down just exclaiming “wow.”
The next demo I tried was set in an airport control tower, where eye and hand tracking played a crucial role. Standing in a control tower I could look around the control center and out through the 360 windows while planes taxied around, took off, and landed. Once my eyes settled on a plane, terminal, or control surface relevant info popped up and control interactions became possible by pinching my fingers. I could look at the computer screen and bring up the weather report, look at a taxiing plane, and bring up its flight number and info. I could reach out to switch between feeds on the security monitor and identify parts of the terminal building and related information. The intuitiveness of the interface made the experience feel natural and much more immersive, as if I wasn’t in VR at all, allowing me to concentrate on the tasks at hand.
“Being able to see in high fidelity and interact without controllers revolutionizes professional VR. Varjo’s human-eye resolution visual fidelity is unprecedented. So is the accuracy and low latency of Ultraleap’s hand tracking” explained Steve Cliffe, CEO of Ultraleap. “The Varjo VR-2 Pro is the best integration of the two technologies and sets a new standard for natural user experience in VR. We’re very excited about the value it will unlock for demanding use-cases such as training, simulation and industrial design.”
I also got a taste of the astronaut training through a few airplane cockpit experiences, although they didn’t contain any interactive elements. The reason Boeing chose to work with Varjo on the Starliner training program became clear as the level of detail on the control panels was stunning. From a pilot’s seated position I could read every button’s tag and easily perceive the angle and position of flick switches. I could also clearly read the computer screens and the detailed flight info they displayed.
“We have seen first-hand what the power of human-eye resolution in VR can offer in terms of expanding the realm of applications for the enterprise,” said Urho Konttori, Co-Founder and Chief Product Officer of Varjo. “Today we bring the Resolution Revolution overnight to nearly all industrial applications, unlocking the next level of professional VR.”
Overall the quality of the tech was outstanding, also bearing in mind it was running of one Nvidia 2080 Ti GPU in a small but powerful micro PC. However, a few things stood out to me that are worth pointing out. The complex foveated rendering with the multiple screens and lenses was mostly unnoticeable while interacting in VR, but when looking at static scenes I became much more aware of the separation between the crystal-clear center of my vision and the peripheral areas as well as the different resolutions displayed.
The effect was also noticeable when purposely pushing the boundaries of the eye-tracking, but still relatively small when you consider the amount of technology being employed. On the plus side, I experienced virtually no eye strain, even after a few hours of use; the HMD itself remained cool and comfortable throughout each of my sessions. Again considering the amount of tech packed into this headset it felt surprisingly light.
“The highest visual fidelity of Varjo’s HMDs enables a trainee to perform optimally within these safety-critical training simulations in VR. For instance, pilots are able to easily read all the gauges, dials, and switches in a virtual cockpit. With Varjo’s 20/20 Eye Tracker, we are able to accurately track and capture where trainees are looking, allowing us to measure and compare performance over time and across a student population,” enthused Scott Schneider, CEO & Co-Founder of HTX Labs, “And with the VR-2’s built-in hand tracking technology, our users can train more naturally and effectively, removing the friction of controllers.”
The sheer improvement in resolution is amazing and enough to put the VR-2 Pro at the top of the list for any mission critical use case. Yet the main significance of this technical accomplishment is how users will be more deeply immersed in the experience because their eyes and brains won’t be struggling to interpret a pixelated digital environment. The smooth and intuitive implementation of hand and eye track will also deepen the level of immersion therein improving the outcome for the user. Using both these technologies Varjo has delivered the most natural and intuitive VR UX I have yet experienced. If it looks real and feels real, our minds will quickly accept that what we’re experiencing is real.
The VR-2 and the VR-2 Pro are available for $4,995 and $5,995 respectively; $795 when sold together with Varjo’s software and support services. Varjo truly is at the cutting edge of immersive technology and a company to keep an eye on as we all step into an augmented future.
Image Credit: Varjo
