An older iPhone app makes a surprise return with exciting AR features.
In case you haven’t seen it, a new tweet from a medical doctor, software entrepreneur, and 3D artist, Fayez Salka shows him using AR technology to bring a virtual piano player to life on a real-world piano.
The video, which has already garnered almost 200K views on Twitter and continues to climb, shows Salka walking through a piano store aiming his phone’s camera at various pianos. Instantly, an AR pianist appears and begins playing different songs from a handful of award-winning pianists while sheet music of the corresponding notes scrolls across the bottom in real-time.
The app uses machine learning to detect the piano, and then figures out the exact 3D position and orientation of your piano in a 3D space down to 1 cm accuracy. It does this just by looking at a single frame and without the use of 3D depth cameras.
During an interview with VRScout, Salka explained the importance of this approach saying, “This is crucial so we can overlay and align the digital pianist exactly on the correct place on your piano,” adding, “What’s cool about our Core ML model is that it works with any piano model or brand under any lighting condition, without the need to use special image targets or markers or the need to scan the piano model ahead of time.”
Though the app isn’t available in the iOS store just yet, Salka and his Massive Technologies Inc partner, Anas Wattar, see the potential of their AR experience as an excellent tool for providing a new way to experience piano music on your smartphone or AR headset, as well as a helpful resource for music education.
“You’ll be able to slow down the song up to 100 times without losing information regarding the hand motion,” said Salka. “This is not possible with traditional recorded video. Obviously all of that carries on to the AR headset too.”
The full version of the app will also use your smartphone’s microphone to listen to you while you practice, and through machine learning, give you instant feedback on your playing; it even uses the AR pianist to offer you tips on how to exactly move your hands during your performance.
For Salka and Watter, this started out as a passion project and ended up turning into an actual business and a full-time job. As a matter of fact, Massive Technologies had some success with an older AR app called Piano 3D, which was downloaded more than 7 million times and was even featured by Apple multiple times on the front page of the App Store and on demo devices in Apple retail stores.
For now, they are both just really excited that their latest AR experience is building up a lot of buzz from other AR enthusiasts and creators and are looking forward to the next phases.
“We have other exciting things cooking in the oven that we would really love to share later on. But for now, the plan is to soft release the app in hopefully the next two weeks followed by a full release later on.”
Check out Massive Technologies website for updates on the launch of the app and to check out their first app, Piano 3D.
Feature Image Credit: Massive Technologies
