Google's Goggles is all but abandoned now. We've seen Google resurrect apps from the dead and update them after years of neglect, but it's hard to imagine the company putting a fresh coat of paint on Goggles at this point. If only because the app has been superseded by others from Google, with its functionality cut off into little pieces and moved to various places inside the ecosystem.
But that doesn't take away from the fascination and respect that Goggles deserves. It could recognize landmarks before Google Photos, read and translate text before Google Translate, use OCR on images before Now on Tap, and even solve sudoku puzzles, scan and add contacts from a business card, and find and suggest similar products — all options that have yet to be transplanted into any other Google app.
The nostalgic types among us will miss Google Goggles, the others will be happy to see that you don't need a specific app to contextually recognize things, places, people, or words. Most of its functionality is now baked into the Android system and integrated with the apps and screens where it makes sense. With the latest update to Now on Tap, I don't miss Goggles anymore. The new image and character recognition features make Goggles feel a little redundant and a lot antiquated.
Now on Tap is accessible from any screen, it's smart enough to work on images in your browser or your camera viewfinder, in Google Photos or any other gallery app you might be using, in a chat window or an email, and more. And it doesn't just impress time and again with its contextual results, it also succeeds way, way more often than it fails at providing these results. Now on Tap is harnessing the potential of Goggles and supplementing it with some smart awareness, Knowledge Graph cards, and essentially combining many of Google's recent apps, features, and ventures into one (or a couple of) small card that just shows the extent to which Google understands the world around us and everything in it.
I've been testing Now on Tap over the past days on almost every single screen I could think of. I see glimpses of Goggles almost every time, but I also see the prospects of something bigger looming on the horizon. It's scary, fascinating, promising, and ever a tad magical.
Now on Tap and Places
Whether you're just using your camera's viewfinder to get information about something near you or you're pulling up images in your gallery or on the web, Now on Tap can recognize places and offer a lot of information about them.
In the first example below, my friend who's traveling tested Now on Tap in his camera. It knew he was looking at Anaa and suggested not just doing regular searches in some apps but also opening up Trip Advisor for him. In the second and third, I tested it on geotagged images I'd taken in front of Big Ben in the UK and the Paris Las Vegas hotel in the US. Check those ratings and call icons for Places, that news shortcut for the Westminster abbey, and that photosphere icon to look inside the hotel: all of them are perfectly suited for the locations they're appended to.
Even more impressive: I tried Now on Tap on this geotagged image of the Jeff Dunham show's poster taken inside Planet Hollywood in Las Vegas. Sure, I got the hotel's card with an events shortcut to see what it has in store, but I also got Jeff Dunham's card with links to his Instagram account and to buy some of his media on the Play Store. All of that without having to manually trigger any OCR feature.
Now on Tap and Music, Movies, TV shows, etc
I then tried Now on Tap on a couple of different media elements. I first did a Google search for Sam Hunt's Take Your Time and pulled up Now on Tap. It obviously recognized the song, but it surprised me by offering to show it to me inside the Shazam app installed on my phone. It also pulled up Sam's card and based on the Knowledge graph, knowing he's a singer, it suggested quick searches for his images, songs, albums, and upcoming events.
I then tried Now on Tap on a saved image of the Warcraft movie poster and got IMDB ratings and the genre right away along with a few handy links. I can only imagine how useful this will be at the movies when I'm trying to decide between two films.
I finally pulled it up on a Mr Bean meme. The image didn't have anything relevant in its name and the meme didn't mention Mr Bean in the text for it to be recognized. I didn't expect much to be honest, but Now on Tap knew this was a Mr Bean photo and linked it to the TV show with quick access to the cast.
Now on Tap's other new strength isn't just in recognizing famous places and images of things, it's also in reading the text inside images and performing smart searches on it. Take this example of a cartoonish meme and font. After launching Now on Tap, I tapped the new text selection icon and pinpointed it to the bubble. The recognized text showed up right on top.
How about a worse font with more jumbled up letters? Let's try it on this comic. And lo-and-behold, it didn't just get all the words right, it also nailed that apostrophe.
I went ahead and tried a different meme, this time with a few names in it. I didn't need to pinpoint to any letters or words, but still, OCR worked in the background to offer cards about both the character and the actor that were mentioned in it. There's even a Chuck Norris quotes link that I found hilarious. Maybe I'll finally be able to understand all those memes I come across that mention people and characters I don't know...
But Now on Tap's OCR chops don't just stop at recognizing the words and typing them for you. They go beyond that to quickly show you what that word you selected is all about, especially if it's something that Google has in its Knowledge Graph. Take this Xarelto medication box that I just pointed my camera at. It got the name right, weird font and merged 'e' and 'l' be damned, and it quickly gave me the drug card of its generic name Rivaroxaban.
Alright, maybe the Rivaroxaban thing was spotted in the photo. Let's be skeptical and try it on a drug prescription that only mentions the brand name, with the generic nowhere to be found. Success. Now on Tap read through that godawful font and horrible printer ink in the first screenshot, recognized Crestor (the brand name) and offered a Rosuvastatin (generic name) drug card. Impressive.
I even threw at it a couple of throat lozenge boxes from the brand Läkerol and it read through that curved weird font, even offering me to start a product search in eBay. Notice that I didn't get the eBay option for the other drugs, but I got it for Läkerol because you can actually buy it on eBay. Creepy/awesome.
And finally, I threw Now on Tap's OCR under the bus by pointing my camera at another drug box with a few different fonts, labels, and languages. First up, you'll see it recognize "pelliculés" and offer two cards for it: the first is an instant medical card for dandruff (WAT... oh it's because pellicule means dandruff in French) and the second is a translation card offering the more appropriate coated definition for the word. It's true that the first card wasn't contextually appropriate, but I still found it awesome that it offered the medical card directly and didn't just translate the word, leaving you to do another search to find out what it is.
The second word I pointed it to was the company's name, which it recognized despite the white on grey font. The third word is where I stumped Now on Tap's OCR: the drug's name. It got Apromasc instead of Aprovasc, all because of that weird red V letter.
Failures and Misses
Although the success rate of Now on Tap was quite high, there were still instances where I thought it should recognize something and it didn't. Take that Jim Carrey meme/image for example and that Minion one. The first is a renowned actor in a very renowned role (Mr Bean was recognized above, this is kind of similar), the second is a really famous fictional character with a huge franchise that you see everywhere. I know that Now on Tap only said it worked on movies, places, and books, but actors and characters should have been there too at launch.
Then there's this cute puppy golden retriever photo that yielded nothing, although Google has the chops to know what it is (cough Photos cough), the failure to do anything with a QR code although Goggles could handle QR codes and barcodes without a problem, and the Simpsons meme that for some reason gave me something about the Institute of Chartered Accountants of Nigeria. Lolwut.
Potential, All the Potential on Tap
Despite being announced with Marshmallow almost a year ago, Now on Tap has yet to grow into its full potential. These latest updates are just scratching the surface of what it can and will do, moving forward. Google already has image recognition smart enough to categorize hundreds of places and things in Photos and even provide you with search results based on emojis as keywords (try searching ice cream emoji or sunglass emoji, you'll be amazed). It can already provide info cards on actors when you pause a movie or TV show in Google Play Movies. It can do image searches instead of text searches on the web and find similar images of anything you come across. And inside Goggles, it can recognize QR codes, barcodes, business cards, and even sudoku puzzles. Disappointingly, none of these features are available in Now on Tap now, but that doesn't preclude the prospect of them being added soon.
There are also other apps right now that can do even more with visual searches. I've used LikeThat Garden and found it super accurate at telling me the name of each plant's photo that I ran by it. There are similar apps that can do that for fashion items, products, dog breeds, and more. I'm sure Google isn't that far behind on those fronts either.
These are all unexploited territories for Google's Now on Tap. The image recognition, as it is today, is impressive but nowhere near what it can be, should Google open it up fully to all of its algorithms and collected data. There's potential beyond what Goggles was or ever could be, one that pushes the idea of recognizing stuff in your camera viewfinder to all of the things around you and on your screen, one that starts here and can grow infinitely once you start factoring virtual reality into the equation, and one that opens up the knowledge of the entire web to you in an intelligent and situationally aware way... like looking at the world with contextually smart goggles.