- 1 Your Assistant at your service, anytime, anywhere
- 2 From OK Google to Assistant: Bite-sized conversational searches
- 3 From Google Now to Assistant: Subscriptions are dumber Now cards
- 4 From Now on Tap to Assistant: Smarter image recognition
- 5 Assisting with every conversation
- 6 Beyond searches, a real helper with untapped potential
Hi Google, it's me Rita. I believe we've met before. Somewhere between Gmail, Google Photos, and Chrome, you must know a lot about me. Things I might not want others to discover, so hushhhh. (There are thousands of people reading us, let's not tell them about my love for Winnie The Pooh.) But our relationship doesn't feel equal; I barely have any information about you. Your new guy, this Assistant you've sent here to talk to me, I'd like to get to know him better. He looks a lot like the other guys you've sent before, Now and On Tap, but he seems special. Smarter, more interesting, less coy, like a better analogue interface of what I imagine you to be.
Hiding within Google's new Allo messenger is one intriguing feature that was demoed at I/O this year: Assistant. Since the announcement and even now that Allo is accessible to many, there's been a lot of confusion about what this Assistant is, where it works, where it doesn't, what it's capable of, why it lives inside a chat application of all places, and why it had to be known as a different entity from all the other ways you can do Google searches. In the following article, I'll try to answer many of these questions and show the benefits and pitfalls of Assistant today, then discuss its future potential.
Your Assistant at your service, anytime, anywhere
Most of the time people spend using their mobile devices is done communicating with others. Facebook, WhatsApp, SMS, Snapchat, Hangouts, Telegram, are all our hubs for reaching out to friends, family members, coworkers, and other acquaintances, so why not use that same approach to talk to an artificial entity too?
Siri, Cortana, Hound, Google Search, and now Assistant, they all started as smarter search engines, but as time passes, they are becoming more akin to full-fledged assistants, managing your everyday life, tasks, plans, information, etc... And you talk to your assistant, you don't "Google" him/her. That's why I think the chat approach makes more sense in some contexts than doing a plain search.
Assistant is tied to your Google account so it will tailor answers to you.
Although Assistant will not remain within the confines of Allo and will power more of the searches and queries you perform later on outside of Allo, its first appearance as a chat bot persona has a few benefits that you may have dismissed.
Say goodbye to "Can't reach Google at the moment"
Chatting with Google proves to be more handy than what you might think at first. Since this is a chat, it can be asynchronous by default. It doesn't need to be instantaneous nor does it need an internet connection to work. This means that you can type a message to Assistant when you're offline, you can send an Assistant query and let it process in the background while you switch to another app, then get notified when it has an answer for you.
One of the grievances of using OK Google so far, whether it's via your phone or Wear watch, is the dreaded spinning/loading animation when for some reason or another, your phone can't directly talk to Google at that moment. Sometimes you have a poor connection, but other times, there's no logical reason, and in all cases, you have to repeat your sentence again and again.
With the chat approach, that limitation, real or from buggy behavior, doesn't exist anymore. Whether you typed a message or sent it via voice, your query will surely be sent to Google at some point, and Google will reply at some point. So any request that you don't need answered this exact moment can be processed by Assistant, knowing fully well that you won't have to repeat it and it won't fail.
If Google is aiming to invade your home and become the center of your searches and automations, I think this modified behavior from OK Google is crucial. Because who wants to repeat the same command 3 times until Google understands what they're saying?
Searchable unified history
Another problem I often face with OK Google searches is that each one is processed as a new instance, creating countless Google windows in my Recents apps that I either have to delete (and thus re-do to get the results again should I need them) or keep and clutter up my multitasking screen to a high extent.
With Assistant's chat interface, your searches and history are saved automatically inside this one chat window so there's no clutter and you can go back and revisit the results of any question or request you made. You can also search your searches (aka searchception) inside Allo, even when you're offline, to find the definition of that one word you asked Assistant about a few days ago. It's like having access to your Google search history and the results inside one continuous stream.
From OK Google to Assistant: Bite-sized conversational searches
However, let's not fool ourselves: Assistant will not replace a full-fledged Google Search for you. By its mere interface design, Assistant doesn't serve well for queries where you need to consult multiple sources and find the best one or where you're browsing for different types of pages to get a more rounded answer to your question. It's true that Assistant always gives you a shortcut to open the full results, but in case you're doing a more complex search, you're better off starting with a regular Google Search than wasting your time to open Allo, then Assistant, then ask it.
The "Search results" fallback is handy, but it's just faster to do a Google Search from the start.
But there are instances and situations where searching with Assistant can be more fun, more informative, or even slightly more capable than regular Google searches done in the browser or via an OK Google command.
Weather, news, games, translations, actions... these are the things Assistant is good at.
If Google Search can do Tic-Tac-Toe and Solitaire, Assistant one-ups it by letting you play even more games when you're bored. There are emoji games, quizzes about math and geography and entertainment, PAC-MAN, and even a full blown Rubik's Cube to solve. Google is after my heart there...
When I'm bored, I know what to do.
Smart search with a bit of flair
For instance, Assistant excels at bite-sized searches, the kind where you're tapping into general knowledge or objective information like biology, music, movies, translations... you know, the stuff that usually has Knowledge Graph cards in search results. Except in this case, you don't get the whole card, you just get the beginning or the specific piece of information you asked about, then you can follow up with more questions.
1, 2: You have to ask Assistant for each bite of info. 3, 4: Google Search just gives you everything.
Assistant is conversational and understands context: if you said "hello Adele" and got the video of the song, then typed "lyrics," you'd get a link to the song's lyrics followed by suggestions like "what about Sorry" and "hello Lionel Richie." So Assistant is threading together searches based on one or multiple common points then generating its suggestions.
It also has a bit of a personality. From the moment it introduces itself to the quirky or smart answers it gives you sometimes, Assistant is more human-like than the voice behind OK Google ever was until now. And that is despite not having any vocal interface yet.
Quirky Assistant answers.
Everything from your personal stuff to OK Google commands
Beside doing regular Google searches, Assistant can comb through your own personal stuff. It can look for your past or upcoming flights, hotel reservations, agenda, and search for keywords in your email.
Assistant can search your personal Google stuff.
But it can also start music playback of a certain genre or album or artist, call your mother, or even find you images from Google Photos of specific people or things or locations. In that extent, Assistant is very akin to OK Google and its multitude of commands, but it's still limited.
It handles well most "OK Google" commands.
Some commands don't seem to work just yet, like adding notes to Google Keep or some of the easter eggs. Other commands still require Assistant to redirect to a Google Search screen to process, like music playback, placing calls, toggling settings, etc...
Left: Keep commands are no-go. Right: "Remind me daily" becomes a reminder for the word "daily!"
And despite how much Assistant already knows about you through your Google history, it will still like to get to know you better, like your favorite foods, colors, and more. It will also use this information to provide better search results for you if you ask it to.
Assistant can tailor answers based on my favorite color, shape, food.
Search with text, emojis, images, voice messages
One last aspect of Assistant worth pointing out is that you can talk to it via any means possible within Allo. This includes typing the text, sending it a voice message (which is akin to doing OK Google voice commands), sharing pictures with it (something you can't do with a regular Google search on your phone), or even emojis.
Talking to Assistant via voice messages and emojis can be handy and/or fun.
This all makes it so that Assistant isn't exactly a replacement to all Google searches, but a better solution for those times when you just want small specific information. The integration of the same commands as OK Google is still clunky sometimes, but it's functional now and with a bit more polish and faster execution, Assistant could easily be a more reliable helper than what it is right now.
From Google Now to Assistant: Subscriptions are dumber Now cards
Google Now, as a brand, has started to disappear a few days ago. Despite the Google app no longer calling it that, the Now cards have been part of our daily Android experience for several years, showing us the weather, relevant articles to read, sports scores, and more. And the newly named Feed and Dashboard carry most of the same cards and features.
Assistant has a similar functionality, except it's not smart enough to give you the information unless you ask for it first. You search for the weather and ask it to remind you daily about it. The same goes for sports news and scores from your favorite teams, stock values, news about topics you're interested in, or any Google search you can think of.
Just type "send me daily" after any search and Assistant will ask for a time to provide you with that information.
You can subscribe to plenty of searches in Assistant.
It calls these subscriptions and lets you manage them and specify the exact time each day you want to be reminded about them.
Using subscriptions is, in a way, more flexible than relying on Google Now cards because you can ask to be notified about the weather at a specific time, you can get directions to work or home each day without having to open Google Now first, and you can automate Assistant to perform any searches for you daily.
For example, my biggest pet peeve with Google Now was its lack of support for currency exchange cards. Drug prices vary depending on currency values, so my work at the pharmacy requires me to keep an eye on the EUR, GBP, CHF, and a few other exchange rates. I always added these as stocks I'm interested in, but the card never showed up in my Now stream. With Assistant, I was able to set up a subscription for "1 EUR in USD" and I get the exchange rate each morning so I no longer have to rely on Pushbullet for that.
Finally, a way to get the currency exchange every day!
However, subscriptions are inherently less smart than Google Now. The point behind Now was to anticipate your needs before you even knew what they were. Live sports scores for your favorite teams, directions to places you're likely to go at this time on this day, a smorgasbord of news you might be interested in based on your recent search and browsing history, and so on. Assistant isn't that smart yet.
Like a new employee, it will need to be taught everything in order to function and that's either going to deter a lot of people or make them more comfortable knowing they hold the reins and there's no superior intelligence at work who knows their habits better than they do.
Keep in mind though that this is still the first sneak peek we get into Assistant. In the future, it could gain many of Now's smarter abilities either by learning from your behavior as time passes, or by getting updates to its code. But in the meantime, subscriptions aren't going to replace Now cards for many of us.
From Now on Tap to Assistant: Smarter image recognition
While Now on Tap has recently been renamed to "Screen Search," another sign that Google is trying to remove instances of "Now" and lessen confusion as to which search entity does what exactly, it is still pretty much an available option in your Android device. And you might think that there's little overlap between Assistant and Now on Tap — what does image and screen searching have to do with an Assistant that can find you restaurants and set up alarms? — but you'd be mistaken.
Since you can share images with Assistant, there is an immediate — if not obvious — overlap between it and Now on Tap. On the one hand, opening Assistant and taking a live photo of something in front of you seems much more natural than starting your camera then launching Now on Tap (aka Screen Search) on top of the livefeed view. It is more logical to ask an assistant about something by sending him/her an image than by using some obscure method only a handful of Android users know.
And so far, Assistant seems to do well with understanding plenty of images you send it. For example, it can see QR codes, whether they're just links or contacts details.
1: Integrated camera in Allo. 2, 3: Assistant's suggestions are taken from the QR codes.
The smarts of Google Photos at your disposal
But where Assistant really shines is with two categories of images: places and things. It recognizes places from photos based on smart algorithms and not the image's title or EXIF data. I've spent time trying to annoy it by removing any indication of a photo's origin or even adding false information to it, it wouldn't budge. It seems to employ the same black magic detection as Google Photos, which is awesome and super powerful.
That means it also recognizes things, to an almost unnerving degree. I verified that by sharing all sorts of images with Assistant, ones I never thought it would guess. Like that kohlrabi image below. A few months ago, I had to enlist the help of my Twitter friends to know the exact name of that vegetable, but Assistant guessed it in a second.
Assistant is the one who suggested Kohlrabi, so I tapped it and got the info card.
In my time testing and comparing both, there were many instances where Now on Tap was stumped by images that Assistant either easily recognized or at least found the general categories of.
Two examples of image searches where On Tap was clueless but Assistant recognized the subject.
Weakness with text: OCR, movie posters, translations
But Assistant still has a long way to go with analyzing any image or screenshot you send its way. Right now, it can't recognize barcodes nor QR codes that hide sentences (not URLs or contacts).
Yes, I can see images! But I can't tell you anything about them :(
It also doesn't do OCR or recognize memes or GIFs.
Very, very limited OCR capabilities.
And most importantly, it doesn't seem to suggest translations or recognize movie posters, two features where Now on Tap excels.
1,2: Assistant fails with the live view and poster of the movie. 3: On Tap gets it from the live view.
1: Testing with a French article. 2: Assistant fails. 3: On Tap gets the topic + offers to translate.
Until Assistant gains all the abilities of Now on Tap / Screen Search, it's going to be difficult for users to gauge situations where using one would be better than the other. I'd recommend you stick with Screen Search where you know it works, ie. when there's text (typed or embedded in an image), and go to Assistant if you don't get any answer or if you have a photo without any text.
Assisting with every conversation
As discussed above, there is a lot of overlap and some clashes between Assistant and the three search entities that it is attempting to complement/replace, but there's also one context where Assistant stands alone: inside your chats.
Assistant is well involved in each and every chat you have in Allo and it's an integral part of the experience. It will suggest smart replies to you, not just based on text you receive but also images and emojis.
The smart replies even show up as quick-actions in Allo's notification so you can quickly send an answer to your contacts without opening the app and without even typing a single letter.
This does have a lot of advantages. Time-saving for one, because the most obvious replies are only a single tap away. Feigning interest too, because we've all been in conversations where we just had to go through the motions of "oh no" and "yay" and "so cute" and "yummy" in reply to people flooding us with complaints or pics of babies or food.
The problem, as time passes, is that your chats start to feel more and more automated if you rely on Assistant's replies too much. You start doubting whether answers you're getting are genuine enthusiasm and care or a dismissive tap on a button based on a suggestion from an AI. In parallel, you're more and more prone to lazily picking an Assistant reply if it was just remotely close to what you had in mind. That's a deep and complex topic that I'm going to leave open for debate. It's not what we're here to discuss, but I hope you keep it in mind and keep yourself in check when you go around having chats in Allo. Don't let the ease of use take away from your own personality.
Augmenting your chats
The other way Assistant can be useful inside conversations is basically by doing everything we discussed earlier, except inside your chat. By summoning @google, it can give you contextual suggestions if you're planning for a movie, a dinner, event, or trip, and help you check different options together with your friend(s).
And it can also answer any query you send its way, like a regular search. That's handy if you and your friend(s) need to see the sports scores, if you want to show them something specific, if you need a better explanation of a term, or maybe if you're trying to settle a debate about some piece of information.
Assistant can be hellafun to introduce during conversations and its real usefulness won't be clear unless you have a few contacts you talk to very frequently during the day and until you have several days and weeks of use behind you. Testing it for a brief moment is very different from experiencing it many times daily in different contexts.
Beyond searches, a real helper with untapped potential
The first brush with Assistant leaves you with a feeling that it's a good party trick but nothing too interesting or powerful. However, the more you try different things with it, the more you realize how capable it is and how it's laying the foundation for a smarter virtual helper, not just a crude search assistant.
It's true that there's a lot that Assistant can't do in this Preview version. It doesn't yet have a voice and can't talk back to you, nor can it understand languages beyond English, and it surely can't yet fully replace any of Google's other search entities. Google Search is more robust when you're looking for detailed information, OK Google commands are better integrated with our Android devices, Google Now (ie Google Feed) cards are smarter and more anticipatory, and Now on Tap (ie Screen Search) is miles better with text and OCR.
But despite its young age, Assistant manages to not just ape some of their functionalities, it also augments them and provides better results or a more interesting experience in various cases. The conversational and contextual search approach can have you jumping (logically) from one topic to another while providing you with just enough information to answer your questions. The subscriptions are more customizable and their openness to any Google search expands their usefulness beyond the Now cards. And being able to use image recognition akin to Google Photos but on demand has potential I haven't even begun to fully comprehend.
Mix all of these with the benefits of the chat interface as an asynchronous search with one continuous history feed, and the integration with your other conversations with friends, and you have the burgeon of something great.
And as Assistant's personality and abilities trickle down to Google's search and feeds and voice commands, the extent of its potential will become more and more apparent. If Google wants this whole artificial intelligence venture to pan out, from your phone to your home and through your everyday life, Assistant will also have to learn to integrate with other services, whether it's by being reachable from other apps on your phone / computer / Google Home, or by having access to other bots and tapping into automations and information that aren't directly available to Google.
Right now, it's easy to get bored of Assistant. It's confined to one bot inside one chat application that hasn't garnered many fans yet. It doesn't play well with any of Google's other search entities, which is another reason for us to sigh and say, "great, Google, there are now 200 different ways to do the exact same thing!" And I'm not sure even Google knows for certain what it wants to do with Assistant, where exactly it is headed, or what it will morph into as time passes.
We are all standing at the dawn of a new era that's fascinating with its promise of better integration and simpler access to information, intimidating with its reliance on artificial assistants in your everyday life, and maybe just a little scary too. But its power and future are unquestionably beyond the grasp of our comprehension today.