Varjo is unveiling its Reality Cloud platform for virtual teleportation. That means one person can capture the reality of a space in a particular location and share that reality in extreme detail for a remote person to experience, virtually.
The Varjo Reality Cloud shares the details of a room in photorealistic detail, showing someone remotely located a view of the room in real time. Yes, you read that. Varjo lets one person scan a 3D space and another person experience it virtually at almost the same time, as it can transfer the necessary data in compact streams of 10 megabits-to-30 megabits per second with almost no time delays, the company said.
It’s a pretty amazing technology that comes from the pioneering work that Varjo has done in creating high-end virtual reality and mixed reality headsets for enterprises such Volvo, which uses it to design cars in virtual environments.
The caveat, of course, is if the tech really works as envisioned.
“We are introducing Varjo Reality Cloud, and this is something very different from what you’ve seen from Varjo before,” said CEO Timo Toikkanen in an interview with GamesBeat. “We have been working on a software platform that is the first in the world that enables virtual teleportation.”
The earlier VR and mixed reality tech that Varjo introduced in the past couple of years now uses cameras on a Varjo VR-3 virtual reality headset to capture the environment around a person. Then it transmits that slice of reality to someone else who uses a headset to experience the exact same physical reality, but in a virtual way. If the company can deliver the Varjo Reality Platform with the same quality it shows in its videos, then it will feel like you’re “teleporting” from your real location to a virtual location.
“You can be anywhere in the world,” Toikkanen said. “You can scan your surroundings, not just a 3D object or something like that. You can digitize the world around you if you like. And do that in super high fidelity, through Varjo Reality Cloud, so anybody anywhere in the world can join you in that location and see it exactly the way you see it, in perfect color, with lights and reflections, and so forth.”
It’s no joke, as Varjo has been working on this for years and it has raised $100 million to date from investors including Volvo (via the Volvo Cars Tech Fund), Atomico, NordicNinja, EQT Ventures, Lifeline Ventures, Tesi, and Swisscanto Invest by Zürcher Kantonalbank.
“It’s a sci-fi dream come true. But we are fully grounded in reality. So we have been looking at the at the experience. How can we enable people to have similar interpersonal experience, as you do in real life, and do that remotely,” Toikkanen said. “What really accelerated for us during last year was the realization how world will never be returning to the same after COVID and travel will forever be changed. And we saw that this is one of those moments when world is more ready than ever for the transformation of this nature in the way the communication and interaction is done. This is the right time to begin that change.”
A realistic metaverse
Toikkanen said that this capturing and sharing of reality is like a true-to-life metaverse, the universe of virtual worlds that are all interconnected, like in novels such as Snow Crash and Ready Player One.
He said that you will be able to see in real-time what your friend is seeing in another place through the cloud-based platform. One person can map their reality by looking around in a room, and that view is transported to the cloud and rebuilt as a room. The person that you share this reality with can view it and feel like they’re there, Toikkanen said.
“It’s a metaverse grounded in reality,” he said. “It really is like the science fiction, beaming yourself to the other end of the world and back. And we think we think this is a really big deal. If you think of the economical and ecological drivers in the world today, something like this makes travel unnecessary.”
He said it could pave the way for a new form of human interaction and universal collaboration.
“You can engage on a completely different level than you have ever been in the history of communications,” Toikkanen said. “It really does change things in a big way. Both for businesses as well as for private individuals. You can teleport to other people, to your family, or you can teleport to a work project.”
The system lets anybody scan their surroundings, turning them into 3D imagery using a Varjo XR-3 headset and then transport that 3D space to another person. That person gets to see the exact physical reality, completely bridging the real and the virtual in true-to-life visual fidelity, said chief technology officer Urho Konttori.
“It’s super important that the latency is kept low enough so that you have you feel that the interaction is logical, and that you don’t have like motion-related latency,” said Konttori. “We have put immense amount of effort into making it so that human-eye resolution, fully immersive stream, from the cloud, can be sent in 10 to 30 megabits per second speeds.”
This real-time reality sharing will usher in a new era in universal collaboration and pave the way for a metaverse of the future, transforming the way people work, interact, and play, Konttori said.
For the past five years, Varjo has been building and perfecting the foundational technologies needed to bring its Varjo Reality Cloud platform to market such as human-eye resolution, low-latency video pass-through, integrated eye-tracking and the Lidar capability of the company’s mixed reality headset.
The company has already delivered these building block technologies in market-ready VR products that enterprises use to design their products. And now Varjo is uniquely positioned to combine them with Varjo Reality Cloud to empower users to enjoy the scale and flexibility of virtual computing in the cloud without compromising performance or quality.
Using Varjo’s proprietary foveated transport algorithm, users will be able to stream the real-time human-eye resolution, wide-field-of-view 3D video feed in single megabytes per second to any device. This ability to share, collaborate in and edit one’s environment with other people makes human connection more real and efficient than ever before, eliminating the restrictions of time and place completely.
To further accelerate bringing the vision for Varjo Reality Cloud to life, Varjo today also announced the acquisition of Dimension10, a Norwegian software company that pioneers industrial 3D collaboration.
“We’re big fans of the company and have been for a long time,” Toikkanen said. “They have been pioneering collaboration, 3D models. And we think collaboration is at the heart Varjo Reality Cloud and us joining forces with them expedites progress.”
The Dimension10 virtual meeting suite is designed for architecture, engineering, and construction teams and will become a critical component to making virtual collaboration possible within Varjo Reality Cloud. Dimension10 adds 14 people to Varjo’s team.
Additionally, Varjo added Lincoln Wallen to the company’s board of directors. Wallen currently serves as the CTO at Improbable, and he is a recognized scholar in computing and AI. Wallen has worked as CTO of Dreamworks, where he transitioned global movie production to the cloud, including the development of a cloud-native toolset for asset management, rendering, lighting, and animation.
Varjo Reality Cloud will first be available to existing customers and partners in alpha access starting in the third quarter. For more information about Varjo’s new cloud platform and its vision for the metaverse, tune into a live, virtual event at 9 a.m. Pacific time today via varjo.com.
Above: Varjo lets one person scan a 3D space and another person experience it virtually.Image Credit: Varjo
In a video tech demo, Varjo showed a simplification to show how the world can be captured and streamed in real time as a 3D representation. It shows a time-lapse capture of a scene captured in real-time from a Varjo XR-3 headset. The video is converted into a 3D space that someone with a viewer and access to the Varjo Reality Cloud can use to see that room from any 3D angle.
In the beginning of the video, the user scans the room and then stops to watch Konttori give a talk. While Konttori is speaking, you see the naturalness of the movement, captured with just a Varjo XR-3 headset in the room, no additional cameras or recording devices. The camera is able to move freely as it’s all in 3D and not a flat video.
In a second video, Varjo teleports Konttori to the company’s Varjo HQ in Helsinki in mixed reality. A user wearing the headset sees the teleported Konttori mixed into a physical space at the headquarters. Later they mix the teleported surroundings together with the physical space in the headquarters.
Above: Kia is using Varjo headsets to design cars.Image Credit: Varjo
Varjo was founded in 2016, when other headsets like the Oculus Rift and the HTC Vive first appeared. But instead of targeting entertainment, Varjo went after enterprises with no-compromise technology.
It debuted its first VR headset, the XR-1, in early 2019 with human-eye resolution, or 1,920 pixels-by-1,080 pixels per eye and an 87-degree field of view. That headset cost $10,000, but the company followed it up December 2020 with its XR-3 and VR-3 headsets that combined VR and augmented reality in the same headset.
That generation had twice the performance of the previous generation, with “human-eye resolution” of 1,920 pixels-by-1,920 pixels per eye and a 115-degree field of view. It was also cheaper, ranging from $3,195 to $5,495 and it was available for cheaper enterprise subscriptions.
Now these headsets can be the jumping off point for the Varjo Reality Cloud, as they can connect to the datacenter and upload the scanned environment that someone can see via the cameras that are on the headset. The quality of the headset capture enables high-quality imagery in the cloud, Konttori said.
“We have innovated for the last five years on making that high fidelity possible,” Toikkanen said. “It links directly to the investment we have made on the headset side into gaze-tracking, eye-tracking, if you like, because that enables innovation. We have also invested in transporting the data between the locations, to the cloud and back, so that we can do this ensure high quality or super low latency. So that’s essentially what we are. We think of it as nothing less than the next form of human interaction.”
The hard part
Above: Varjo is targeting professionals such as product designers with its XR/VR headsets.Image Credit: Varjo
“Nobody else is at the place that they have the hardware even near the quality that we have, let alone the software stack that allows us to actually pull this off,” Toikkanen said. “And we have of course be developing this simultaneously. And now is the culmination of all that work.”
Gaze-tracking is important because if you can track where someone’s eyes are moving, then you know what they’re looking at and you can transport that view with low latency. That allows the company to create foveated transport algorithms, which means it only sends the data that you can see and that you are looking at, rather than other data that isn’t needed in real time at that moment.
“It’s a huge undertaking, and so we developed a year and a half ago a new way of doing that transport,” Konttori said. “The video stream focuses at the place that you’re looking at. That’s where we have the full resolution in the video stream. And then the degrades gradually from that towards the edges of the screen. And does that very quickly. It means that we can send the data that we send at the moment on cables from the computer to the headset, which is running at like 20 gigabits per second, and we can send that with our new compression technology at 10 megabits to 30 megabits per second.”
That means it works that you can share imagery with someone 2,000 miles away, Toikkanen said.
Above: Varjo’s new XR-3 and VR-3 headsets.Image Credit: Varjo
It’s a level of quality that is 10 times the resolution difference of other headsets out there, Konttori said.
“You get real-time presence because when we’re scanning, we’re just not just making a 3D model of the surroundings that you’re in and make that a teleport location,” Konttori said. “We’re actually updating that in real time.”
You could have a manager on a factory floor put on a headset. They can create a teleport node, and people from other countries can join and see what the manager sees. It’s all updated in real time and people get a sense they are truly at that location. They can fix the things that the manager is looking at, and then take off a headset and be at home.
“If you want to visit your family, it’s the same thing,” Konttori said. “You can share that physical location, and people can instantly perceive the world as if they were actually there themselves.”
Once you scan a place, you don’t have to scan it again, Toikkanen said. And you can use any headset to teleport to a location, or use a phone and still have the freedom of movement to look around. But the Varjo XR-3 is the only device that can be the teleportation node that broadcasts and streams the 3D space to someone else.
Toikkanen said it’s like moving from the telephone to a video conference, and moving from that to something that is even more transformative.
“We think there are going to be a billion people using this kind of service over the next 10 years or 20 years,” he said. “We are in the alpha phase with real customers and partners this year.”
A cousin of the Omniverse
Above: BMW Group is using Nvidia’s Omniverse to build a digital factory that will mirror a real-world place.Image Credit: Nvidia
I asked if this would be a way to scan the world into Nvidia’s Ominverse, the metaverse for engineers that lets them simulate photorealistic details in a virtual world to test how they will work in reality. BMW is using the Omniverse for creating a “digital twin,” or a car factory it can design in a virtual space before it builds an exact copy in the physical world.
Toikkanen said that both tools are useful for the metaverse and they are complimentary.
“They’re both part of the like, movement towards metaverse, and this teleport functionality is adding a completely new node into the sphere of discussion of a metaverse, which is that one part of that can be the real world itself,” Toikkanen said. “And we make it so that you get the benefits of a metaverse also in real world setting. And we think that’s at least equally transformative as the metaverse which is typically seen only in virtual reality.”