Live Caption is one of the most impressive features coming to Android. It automatically transcribes anything being said in a video or audio on your screen for you to read, and can be an invaluable tool for hearing-impaired users — plus it could avoid awkward glances if you raise the volume too much in public. Now that the feature has launched exclusively on the Pixel 4, everyone has been trying it out and discovering a few nice, and perhaps unexpected aspects of it.
First, Live Caption's persistent notification only shows up when internal audio is detected. The feature will automatically enable and disable itself, saving you from having to look at an annoying notification all the time.
Second, and more importantly, it works when audio is muted — be it in-app or with the volume buttons. That's due to its usage of Android 10's new in-app audio recording API, which means it doesn't rely on the mic to detect audio. We knew this was necessary to not transcribe nearby sounds and speech, but as it turns out, it also enables grabbing audio even when the app itself isn't meant to play any.
Take for example Twitter and YouTube's videos that autoplay while scrolling. Audio is disabled on them, but as you scroll and they play, you'll see the live captions on the screen.
Live Captions when scrolling in Twitter and YouTube. (Our tipster set a small font for captions, sorry.)
The same is true when the phone's sound is entirely muted, as well as when you mute a video inside the app. The Android 10 API still captures the sound that was supposed to come out and captions it.
Left: Live Caption despite sound muted by the phone. Middle & Right: Also when video is muted in-app.
Live Caption is currently an exclusive to the Pixel 4, but will make its way to the 3 and 3a in December. It's certainly one of the features I'm looking forward to the most.
- Anthony Maki