Project Astra is a new AI-powered virtual assistant that learns about world around you, with you

Remember J.A.R.V.I.S. from Iron Man? A virtual assistant that could do anything from sending Tony Stark reminders to gifting Ms. Pepper Potts flowers for her birthday. Or perhaps TARS from Interstellar, who could do everything ranging from piloting the Endurance to collecting data from the many worlds they visited. Well, Google hasn’t gotten there quite yet, but it has something that’s perhaps as close as it gets to them for now.

Namely, their AI agent named Project Astra. Announced amidst a flurry of other game-changing AI tech at last night’s I/O keynote, Project Astra particularly caught our eyes. Here’s why.

What Project Astra is, and how it works

Project Astra is a new multimodal AI agent, capable of answering real-time questions, fed to it through text, video, and images and speech, by pulling up the relevant information for it. It does this pulling up information from both, the web, as well as the world it sees around you, through your smartphone camera lens.

ALSO READ: Google I/O 2024: Top 5 AI announcements from Sundar Pichai’s keynote address

How it does this, is by encoding video frames and speech into a timeline, and then caching it for recall. In simpler words, it looks at the world around you through your smartphone camera, perceives, responds, and remembers it, just as we do. And it does so even when the objects it saw earlier are no longer in the camera frame.

What Project Astra can do for you

A demo video by Google shows an employee using Astra on their smartphone and asking it to identify what a particular part of a speaker is called and how it functions. Astra managed to identify it in seconds and explain it in the same tone as a human would. It even managed to identify the area the person was in London, just by a fleeting glimpse out of the window.

There are many more interesting use cases that Astra can help with. Imagine you’re parking your car in a large parking space. Well, all you have to do is let Astra know where it is, and it can literally guide you back to the same spot.

It will even be able to analyse snippets of code and identify just what it’s for. Or say you see a really pretty sunset by the beach and are feeling poetic. Well, Project Astra will be able to write a poem around it for you!

You must think that carrying a phone with its camera on the whole time to teach Astra about the world is a hassle, right? But that’s not entirely true, as Astra is not just limited to smartphones. Google also showed it being used with a pair of smart glasses in a demo video, adding to its ease-of-use and utility all the more.

Virtual assistants, up until now, depended on the information they managed to extract from the web and the information that you fed them, to help you accomplish tasks. Project Astra though, can learn about the world around you, making it as close to a human-assistant-like experience as is possible currently.

How can you access Project Astra?

For now, Project Astra is in its early stages of testing, and there are no specific launch dates. However, Google has hinted that some of its capabilities could potentially be integrated into its other products such as the Gemini app later this year.

Called Gemini Live, it is not unlike the talk-back feature on OpenAI’s ChatGPT. We can’t wait to see what else Google has in store for us with its latest AI-powered virtual assistant and try it out for ourselves.