Zurich mixed reality researchers designing post-smartphone future

Attendees at the International Frankfurt Motor Show try out the HoloLens. Copyright 2017 The Associated Press. All Rights Reserved.

Zurich has become a hub for technology giants researching computer vision and artificial intelligence. They’re working on the next generation of personal devices – called Mixed Reality – that could some day replace smartphones.

Imagine wearing glasses from the future. Much like your smartphone today, these allow you to access any information you want. Instead of looking down at your screen, you are looking right in front of you. Holograms are floating in your environment. Arrows displayed on the street could help you find a location, or a hologram of a colleague could appear sitting on an empty chair next to you during a meeting.

Computer vision revolves around the ability of a camera and a computer to do what humans can do with their eyes. Specifically, that means “looking at the world, interpreting the world, and figuring out how to move through the world,” says Marc Pollefeys, a professor of computer science at Zurich’s Federal Institute of Technology (ETHZ), who is also the director of Microsoft’s Mixed Reality and AI Zurich labExternal link. Existing applications of computer vision include self-driving cars and the ability to unlock your phone by looking at it.

Whereas virtual reality (VR) disconnects the user from the real world, augmented reality (AR) and mixed reality (MR) enhance it. Pollefeys and his team in Zurich, which includes Microsoft researchers and doctoral students from ETHZ, are focusing on MR.

AR usually refers to “overlaying the world with some information.” MR takes it a step further, Pollefeys explains, by combining the real world with virtual elements, like holograms. Those elements go beyond “just floating text,” he says.  

“There is an interaction between the real and virtual – it is one mixed-world representation.”

HoloLens

How is that made possible? Recent advances in deep learning (another area of artificial intelligence), better graphics processing units (GPUs), and big data sets have revolutionised computers’ ability to extract semantic information from images. For example, when a machine sees a chair, it can “know” that it’s a chair. Pollefeys explains that this, together with a good understanding of how to extract geometric information from images enabling devices to track themselves in space, means that computer vision is now able to solve much more complex problems than ever before.

The team at the Microsoft Mixed Reality and AI lab in Zurich is busy researching the future of MR. These days, that future is the HoloLens. It’s a wearable device, shaped like a bulky pair of glasses, with a computer inside. At the moment, the device serves specialised markets, allowing industrial workers to service machines via remote assistance or surgeons to see patient scans during operations. It targets professionals that perform complicated tasks with their hands but need to access information at the same time.

But long-term, Pollefeys thinks that the device “is something that could replace mobile phones”. The desktop computer and laptop were game changers for access to information, but they have little awareness of their environment and struggle to add context. The smartphone, a pocket computer, provides some level of context by, for example, tracking your location.  

But mixed-reality devices can go deeper. Equipped with an array of cameras and sensors, they can “really see the context you are in,” says the Mixed Reality Lab director. They understand your environment and what you are doing. They can place information right in front of you, “directly over the world, mixed with the world,” Pollefeys explains.

“It is in that sense much more powerful at helping you do a task.”

Hurdles to consumer market

The idea of replacing smartphones with head-mounted, computer-like devices isn’t new. A major experiment in the field was the launch in 2013 of Google Glass. The AR device, which looked like a pair of glasses, allowed users to read a map, a text message, or a weather forecast – floating in the near distance.

The concept wasn’t a success, in no small part, because of its outward-facing cameraExternal link which made others uneasy. Two years after its release, Google Glass was taken off the market. It then made a comeback in 2017External link, as Glass Enterprise Edition, an updated version targeting the industrial market.

Microsoft has taken the opposite route, starting with applications for professionals. Asked about what elements he thinks will be key for success on the consumer market, Pollefeys responds: “Besides being able to build a device that is not much larger than normal glasses and is truly useful, it is to make sure other people don’t feel that their privacy is threatened by the device.”

Privacy concerns

Exposing our private lives to devices full of sensors and cameras raises concerns about privacy, and Pollefeys explains that the HoloLens device is built with privacy in mind. Raw image data never leaves a separate processing unit that applications cannot access.

“Both broadly at Microsoft, and specifically in the HoloLens team, we have ethics teams that are focused on the ethical aspects of AI, and help us reason through these aspects, and figure out what makes sense and not,” he says. Part of the research at his Zurich lab focuses on ways to make data even more abstract to better guarantee privacy.

Pollefeys expects it will be five to ten years before a larger portion of the population embraces mixed reality devices, though early adopters might only be a few years away.

“Part of the evolution will be technological progress, but also, what is the right tradeoff between privacy and utility for the consumer?”

Computer vision research in Zurich

In 2016, Facebook acquired a team of researchersExternal link from the Federal Institute of Technology Zurich (ETHZ) and the University of Zurich to join its virtual reality subsidiary Oculus. The American firm recently announced its intention to double the number of employees in the Swiss city. 

Other acquisitions around computer vision included Apple’s takeover of Zurich-based startup FaceshiftExternal link in 2015. Magic LeapExternal link, an MR startup that raised $2.6 billion (CHF2.5 billion) acquired a division of the startup DracudaExternal link in 2017. 

Google too, has been growing, adding in 15 years close to 4,000 employees to its initial count of two. The Silicon Valley technology giant reportedly operates its biggest engineering research center outside of the US in Zurich.