The biggest reveal at Meta’s Connect event was its long-promised AR glasses, Orion. As expected, the prototype, each of which reportedly costs around $10,000, won’t be ready for the public any time soon.

In the meantime, Meta offered a glimpse of its new holographic avatars, which will allow people to talk with lifelike holograms in augmented reality. The holograms are Meta’s Codec Avatars, a technology it’s been working on for several years. Mark Zuckerberg teased a version of this last year when he participated in a podcast interview “in the metaverse.”

That technology may now be closer than we think. Following the keynote at Connect, I sat down with Mark Rabkin, a VP at Meta leading Horizon OS and Quest, who shared more about Meta’s codec avatars and how they will one day come to the company’s VR headsets as well.

“Generally, pretty much everything you can do on Orion you can do on Quest,” Rabkin said. The Codec Avatars in particular have also gotten much easier to create. While they once required advanced camera scans, most of the internal avatars are now created with phone scans, Rabkin explains.

“It’s an almost identical process in many ways in generating the stylized avatars [for VR], but with a different training set and a different amount of computation required,” Rabkin explained. “For the stylized avatars, the model has to be trained on a lot of stylized avatars and how they look and how they move. [It has to] get a lot of training data on what people perceive to look like their picture, and what they perceive to move nicely.”

“For the Codec avatars ... it's the same process. You gather a tremendous amount of data. You gather data from very high-quality, fancy camera scans. You gather data from phone scans, because that's how people will be really creating, and you just build a model until it improves. And one of the challenges with both problems is to make it fast enough and computationally cheap enough so that millions and millions can use it.”

Rabkin said that he eventually expects these avatars to be able to play in virtual reality on the company’s headsets. Right now, the Quest 3 and 3S don’t have the necessary sensors, including eye tracking, necessary for the photorealistic avatars. But that could change for the next-generation VR headset, he said: “I think probably, if we do really well, it should be possible in the next generation [of headset].”

This article originally appeared on Engadget at https://www.engadget.com/ar-vr/metas-orion-holographic-avatars-will-eventually-be-in-vr-too-235206805.html?src=rss

The biggest reveal at Meta’s Connect event was its long-promised AR glasses, Orion. As expected, the prototype, each of which reportedly costs around $10,000, won’t be ready for the public any time soon.

In the meantime, Meta offered a glimpse of its new holographic avatars, which will allow people to talk with lifelike holograms in augmented reality. The holograms are Meta’s Codec Avatars, a technology it’s been working on for several years. Mark Zuckerberg teased a version of this last year when he participated in a podcast interview “in the metaverse.”

That technology may now be closer than we think. Following the keynote at Connect, I sat down with Mark Rabkin, a VP at Meta leading Horizon OS and Quest, who shared more about Meta’s codec avatars and how they will one day come to the company’s VR headsets as well.

“Generally, pretty much everything you can do on Orion you can do on Quest,” Rabkin said. The Codec Avatars in particular have also gotten much easier to create. While they once required advanced camera scans, most of the internal avatars are now created with phone scans, Rabkin explains.

“It’s an almost identical process in many ways in generating the stylized avatars [for VR], but with a different training set and a different amount of computation required,” Rabkin explained. “For the stylized avatars, the model has to be trained on a lot of stylized avatars and how they look and how they move. [It has to] get a lot of training data on what people perceive to look like their picture, and what they perceive to move nicely.”

“For the Codec avatars ... it's the same process. You gather a tremendous amount of data. You gather data from very high-quality, fancy camera scans. You gather data from phone scans, because that's how people will be really creating, and you just build a model until it improves. And one of the challenges with both problems is to make it fast enough and computationally cheap enough so that millions and millions can use it.”

Rabkin said that he eventually expects these avatars to be able to play in virtual reality on the company’s headsets. Right now, the Quest 3 and 3S don’t have the necessary sensors, including eye tracking, necessary for the photorealistic avatars. But that could change for the next-generation VR headset, he said: “I think probably, if we do really well, it should be possible in the next generation [of headset].”

This article originally appeared on Engadget at https://www.engadget.com/ar-vr/metas-orion-holographic-avatars-will-eventually-be-in-vr-too-235206805.html?src=rss

Meta’s AI assistant has always been the most intriguing feature of its second-generation Ray-Ban smart glasses. While the generative AI assistant had fairly limited capabilities when the glasses launched last fall, the addition of real-time information and multimodal capabilities offered a range of new possibilities for the accessory.

Now, Meta is significantly upgrading the Ray-Ban Meta smart glasses’ AI powers. The company showed off a number of new abilities for the year-old frames onstage at its Connect event, including reminders and live translations.

With reminders, you’ll be able to look at items in your surroundings and ask Meta to send a reminder about it. For example, “hey Meta, remind me to buy that book next Monday.” The glasses will also be able to scan QR codes and call a phone number written in front of you.

In addition, Meta is adding video support to Meta AI so that the glasses will be better able to scan your surroundings and respond to queries about what’s around you. There are other more subtle improvements. Previously, you had to start a command with “Hey Meta, look and tell me” in order to get the glasses to respond to a command based on what you were looking at. With the update though, Meta AI will be able to respond to queries about what’s in front of you with more natural requests. In a demo with Meta, I was able to ask several questions and follow-ups with questions like “hey Meta, what am I looking at” or “hey Meta, tell me about what I’m looking at.”

When I tried out Meta AI’s multimodal capabilities on the glasses last year, I found that Meta AI was able to translate some snippets of text but struggled with anything more than a few words. Now, Meta AI should be able to translate longer chunks of text. And later this year the company is adding live translation abilities for English, French, Italian and Spanish, which could make the glasses even more useful as a travel accessory.

And while I still haven’t fully tested Meta AI’s new capabilities on its smart glasses just yet, it already seems to have a better grasp of real-time information than what I found last year. During a demo with Meta, I asked Meta AI to tell me who is the Speaker of the House of Representatives — a question it repeatedly got wrong last year — and it answered correctly the first time.

Catch up on all the news from Meta Connect 2024!

This article originally appeared on Engadget at https://www.engadget.com/wearables/metas-ray-ban-branded-smart-glasses-are-getting-ai-powered-reminders-and-translation-features-173921120.html?src=rss

Meta’s AI assistant has always been the most intriguing feature of its second-generation Ray-Ban smart glasses. While the generative AI assistant had fairly limited capabilities when the glasses launched last fall, the addition of real-time information and multimodal capabilities offered a range of new possibilities for the accessory.

Now, Meta is significantly upgrading the Ray-Ban Meta smart glasses’ AI powers. The company showed off a number of new abilities for the year-old frames onstage at its Connect event, including reminders and live translations.

With reminders, you’ll be able to look at items in your surroundings and ask Meta to send a reminder about it. For example, “hey Meta, remind me to buy that book next Monday.” The glasses will also be able to scan QR codes and call a phone number written in front of you.

In addition, Meta is adding video support to Meta AI so that the glasses will be better able to scan your surroundings and respond to queries about what’s around you. There are other more subtle improvements. Previously, you had to start a command with “Hey Meta, look and tell me” in order to get the glasses to respond to a command based on what you were looking at. With the update though, Meta AI will be able to respond to queries about what’s in front of you with more natural requests. In a demo with Meta, I was able to ask several questions and follow-ups with questions like “hey Meta, what am I looking at” or “hey Meta, tell me about what I’m looking at.”

When I tried out Meta AI’s multimodal capabilities on the glasses last year, I found that Meta AI was able to translate some snippets of text but struggled with anything more than a few words. Now, Meta AI should be able to translate longer chunks of text. And later this year the company is adding live translation abilities for English, French, Italian and Spanish, which could make the glasses even more useful as a travel accessory.

And while I still haven’t fully tested Meta AI’s new capabilities on its smart glasses just yet, it already seems to have a better grasp of real-time information than what I found last year. During a demo with Meta, I asked Meta AI to tell me who is the Speaker of the House of Representatives — a question it repeatedly got wrong last year — and it answered correctly the first time.

Catch up on all the news from Meta Connect 2024!

This article originally appeared on Engadget at https://www.engadget.com/wearables/metas-ray-ban-branded-smart-glasses-are-getting-ai-powered-reminders-and-translation-features-173921120.html?src=rss