Google Taps AI to Improve Accessible Camera Features, Live Transcribe

Along with the debut of the Pixel 9 series, Google on Tuesday unveiled a host of Android accessibility updates powered by AI. They include improvements to Guided Frame and Magnifier on Pixel phones, as well as a new Live Transcribe mode for foldable phones running Android and the ability to use Live Caption and Live Transcribe in more languages. 

In recent years, Google -- along with other tech companies like Apple and Amazon -- has been working to bolster accessibility offerings on its devices and platforms. The search giant has introduced apps like Lookout, which is designed to help blind and low-vision users identify objects and read documents, and Sound Notifications, which alerts people with hearing loss about "critical household sounds" like appliances beeping and water running. A handful of tech companies have also teamed up for a research initiative called the Speech Accessibility Project, which aims to make voice recognition more useful for people with a range of diverse speech patterns and disabilities.

Read more: Pixel 9 Pro XL Review: Google Goes All In on AI, for Better or Worse

Google usually shares accessibility updates alongside its big annual events like I/O and Made by Google. Here's how it's refreshing some of its newest tools with the help of artificial intelligence. 

Guided Frame is easier to enable and use

Guided Frame on Pixel is designed to make it easier for people who are blind or low-vision to snap photos and selfies by using audio and haptic cues, which provide feedback as they position their phone. Once they're in the "sweet spot," Guided Frame will automatically take the picture. The feature works with Google's TalkBack screen reader, which speaks aloud text and images that appear on someone's screen. 

Now, you can also turn on Guided Frame directly from your Pixel's camera settings. This can be helpful for people with low and moderate vision who may not typically use on a screen reader. Guided Frame now also touts improved object recognition, smarter face filtering in group photos and better focus on subjects in more complex scenes. 

Magnifier gets smarter with AI

Google is also rolling out an update to Magnifier, an app it debuted last year that zooms in on text and objects to make them easier to decipher. With the help of AI, Magnifier will now let you search for specific words in your surroundings, which can be helpful for tasks like looking for your flight's departure time or searching for a vegetarian dish on a menu. 

You can also choose the best lens for the task at hand in the Magnifier app's settings. For instance, you could choose the macro lens to zoom in on fine print or opt for the wide-angle lens to better understand the scope and context of what you're looking at. Magnifier is available on Google Play for the Pixel 5 and up, not including the Pixel Fold lineup.

Live Transcribe's dual-screen mode is tailored for foldable phones

Live Transcribe, which provides real-time transcriptions of spoken words and sounds, can now take advantage of two screens to make back-and-forth conversation easier, if you've got a foldable phone. Google says it got feedback from users that "it would be helpful for speakers to see their own transcriptions while using Live Transcribe," the company noted in its blog post.

Now, a dual-screen mode will work on foldable phones, including the new Pixel 9 Pro Fold, so you can put your phone in tabletop mode and everyone can follow the conversation. This can be helpful for meetings or when chatting with friends, for instance.

Live Caption and Live Transcribe expand to more languages

Live Caption provides real-time captions across apps on Android devices, so people who are deaf or hard of hearing can access audio content. Google is adding seven new languages: Korean, Polish, Portuguese, Russian, Chinese, Turkish and Vietnamese.

These languages will also be available on-device for Live Transcribe. That means people can use the feature in 15 languages, even when there's no internet connection or they're in airplane mode. With an internet connection, you can access over 120 languages on Live Transcribe. 

Not Just AI: 14 Changes Coming Soon to Android 15