It won’t be long before 3D holograms are part of everyday life.
In virtual reality, the more realistic the digital world, the greater the demand for fidelity. The greater demand for fidelity, the greater the demand for bandwidth. Nothing is more demanding than cinematic realism. We expect photo real worlds to have absolutely perfect fidelity. This is why realism in VR always involves a sterile space station or a desert planet. It is going to be some time before we cavort with artificially intelligent characters in Westworld or The Matrix.
Lucasfilm Ltd.This can be done easily today with—or without—an AR capable cell phone.
Augmented Reality will soon be more than cartoony Pokemon and Snapchat Filters (and Facebook’s coming Camera Effects Platform). It will soon feature realistic holograms of people, too.
“VR has given birth to a process at the intersection of games and movies,” said James George, co-founder of DepthKit.
TwitchScreen capture of prototype Star Wars AR game for Disney’s Lucasfilm/Lenovo AR headset.
Courtesy of The McKittrick HotelThe epic success of “Sleep No More” in New York represents a tipping point for Immersive Entertainment.
A multi-character drama in VR or AR would be like “Sleep No More”, the smash hit environmental theater production in New York now going into the sixth year. It might also be so bandwidth intensive it would take many hours, even days, to download over a typical home cable modem. As the video below shows, a cinematically real, 3D George Takei is not something that can be made casually.
Last June, I met Pete Forde of Itsme at VR Toronto, where he did a realistic 3D capture of me, and then animated it. 70 cameras captured me as I stood, arms akimbo, in a heavily lit booth in Itsme’s office. The image was processed automatically in just a few minutes.
Charlie FinkPete Forde’s mobile capture room used 70 cameras to create my photo-real 3D avatar.
Once captured, my 3D avatar can be animated. It can do anything. I flipped over it. Or, rather, Forde flipped me.
My animated avatar can be placed in any VR or AR application. Itsme is unique because they are capturing my avatar, not George Takei’s. Forde says he’s working on a 3D capture method that will do away with the booth.
To promote the film Marjorie Prime volumetric capture start-up 8icaptured star Jon Hamm (an AI character in the movie) and has just now made him available to everyone via their free Holo smartphone app. 8i previously unveiled the Jon Hamm hologram at the 2017 Sundance International Film Festival. Using 8i’s Holo app, users can place the Holo-Hamm in their living room, snap a selfie, or record a video. The Holo app features a variety of photorealistic 3D holograms that can be added to your camera view, moved, rotated and resized. Holo unveiled a similar collaboration for Spider-Man earlier this year.
Like Microsoft’s Takei demo, Holo-Hamm and Spiderman are grounded in place. In the theater that’s called “park and bark”. Capture stages are small. A tracking shot with 41 cameras is hardly feasible.
8i CEO Steve Raymond told me the company’s overarching goal is to “capture human performance so authentic they bridge the uncanny valley”. “Uncanny valley” describes the hypothesis that human replicas which appear almost, but not exactly, like real human beings elicit feelings of eeriness and revulsion (or uncanniness) among some observers.
“The process of capturing a person volumetrically in 3D from an array of cameras pointing inward is called volumetric capture. Volumetric capture is video based, which makes the approach photoreal and scalable. It allows us to reconstruct the person being recorded with full volume and depth, color and light, so that when viewed in a VR or AR headset, or in mobile AR, the person looks real as if they are actually there and you can walk around them and see them from any angle” 8i spokesperson Amy Sezak told me in an email.
While capturing multiple photo real 3D characters in dramatic situations is nearly impossible today, the photogrammetric capture of the real world is relatively easy.
Linden LabYou can explore this photo real, full-scale Egyptian tomb on Sansar.
RealityVirtual of New Zealand is developing a method of VR cinematography they call “volumetric videogrammertry”. They use photogrammetry, drones, and high-speed cameras to capture real environments in startling detail. Because the file sizes are enormous, the company has developed a way to reduce the files automatically, without sacrificing fidelity. Founder Simon Che de Boer told me his company is creating “a one button cloud based processing service for people who want to make cinematic VR”.
Like Itsme and RealityVirtual, DepthKit is a seed stage start up that provides the tools independent content creators need to capture volumetric video of actors to tell their stories in VR. DepthKit is a suite of tools that allows creators to capture, edit, and publish volumetric experiences. To use DepthKit all that’s needed is a camera, a Kinect, and a suitable computer.
DepthKit’s method is the most inexpensive, and artists are able to stylize the results to go beyond just photorealistic looking holograms. People can be made more impressionistic, and therefore less computationally demanding. “You don’t need to capture reality in perfect detail to make to make a movie believable. Movies exist in a kind of magical reality already,” Porter said.
Scatter StudiosMoe Black from Waffle Crew dances in the Litefeet style made famous through unsanctioned performances on NYC trains. Captured by Scatter with DepthKit.
Founders Alexander Porter and James George developed DepthKit for projects they were producing themselves through their other company, Scatter. They decided to make the unique methods of low-cost volumetric capture they pioneered available to everyone.
Depth KitUsing DepthKit anyone can now capture impressionistic 3D with a basic SLR, an Xbox Kinnect, and a high end PC.
DepthKit supports the export of an image sequence for later use in standard editing applications like Adobe Premiere, Final Cut Pro etc. Applications to participate in DepthKit’s closed beta are on their web site.
On the other end of the spectrum are the high-end Framestore and Quantum Capture. Quantum, based in Toronto with an LA office, has a method that captures and then animates avatars. They are marketing this to both entertainment and game producers. “Our characters are completely naturalistic,” CTO and co-founder Craig Alguire told me. “Facial animation and lip sync are perfect. These are essentially programmable humans.”
In 2013, LA-based Framestore brought Audrey Hepburn back from the dead for a chocolate commercial. Hepburn’s re-appearance generated some nice royalties (aka “found money”) for her estate. Of course, reanimation requires skilled animators, but animators are the most versatile actors in the world. A lion cub one day, Maleficent the next.
A bootstrapped Isreali start up, Octosense, also specializes in capturing performers in 3D and realistically animating them. According to CTO Omer Breiner, the company “makes high level realistic 3D content with files sizes no larger than video”. Octosense is positioning itself to sell B-to-B services to media companies. Breiner thinks their technology could greatly enhance a digital assistant. Here’s a demo they did with the avatar of singer Rhianna.
“Major advances in mixed and augmented reality always seemed unachievable,” said Octosense CEO Deve Ouazanan. “We’ve already made amazing progress and are now working to release the first realistic personal digital assistant. Our R&D department is already setting its sights on what can be achieved in terms of realism and natural facial movement, as we create the next generation of mixed reality.”
The tools to make to capture and create realistic performances for VR and AR are just now reaching the hands of creators. This will greatly enhance the artistic and storytelling potential of this new medium. Even with the crudest of tools, it feels like a door to a new world is opening.
This article originally appeared in Forbes. The author would like to thank Gabriella Weiss who helped to research this column.
