When the Photos app launched in 2015, Google had already been working on computer vision for many years. It was no surprise that object recognition was a major draw for the app, and automatic sorting of pictures by person with the use of facial recognition became a must-have feature. Google is now looking to leverage this technology to make the Photos app a bit more social. A teardown of the latest Photos update reveals the service will soon offer additional features based around facial recognition, and even encourage users to share pictures with the people detected in them.
The features discussed below are probably not live yet, or may only be live for a small percentage of users. Unless stated otherwise, don't expect to see these features if you install the apk.
Since facial recognition is a sensitive privacy topic, it's worth a quick sanity check to reduce confusion and knee-jerk reactions going into this teardown. Everything discussed here is opt-in and the features will not work if users do not select a photo of themselves, and that means nothing can happen accidentally or without granting permission. Furthermore, most of the features are only slightly modified versions of existing aspects of the app and pose no additional risk to privacy. I'll leave it up to each reader to determine how they feel about the remaining functions and their practical implications.
Face recognition for personalization, sharing, and more
Instead of opening with the reason this feature might be useful, I'm going to jump straight to a summary of how Google is going to present it. Quite simply, the Photos app is going to ask users to "personalize" their experience by selecting a picture of themselves from the list of previously identified faces. If you already named yourself, the group might be renamed "Me" after making the selection.
<string name="photos_facegaia_optin_impl_title">Personalize Google Photos by labeling your face</string>
<string name="photos_facegaia_optin_impl_desc">Which face is yours?</string>
<string name="photos_facegaia_optin_impl_picker_face_picker_title">Select your face</string>
<string name="photos_facegaia_optin_impl_confirmation_toast_msg">All set! Try searching for “Me”.</string>
Face recognition isn't just the tool, it's actually part of the pitch to get users interested. Google's explanation is broken up into two parts, the first of which offers to reward users for choosing a face with "personalized search, creations, and more."
<string name="photos_facegaia_optin_impl_detailed_desc0">Get personalized search, creations, and more based on your face?</string>
Personalized search is explained by the toast message above – it just means you can search for "me" to find pictures of yourself. I assume "creations" are the same auto-generated albums, collages, and animations we usually see in the Assistant tab, but obviously focused on you, the user. Both of these sound like things we have now.
<string name="photos_facegaia_optin_impl_detailed_desc1">Allow your contacts' Google Photos apps to recognize your face in photos for faster sharing? <a href=help:>More</a></string>
It's the second part of the explanation where things get interesting. If you have identified your face for Google Photos, you can also give permission to have the app on other people's phones recognize you when pictures are taken. To be clear, this can only happen for people if they are in your contact list and if you've given permission for it to make matches.
<string name="photos_facegaia_optin_impl_dismiss_dialog_title">You're missing out</string>
<string name="photos_facegaia_optin_impl_dismiss_dialog_message">Get personalized features and help friends share faster with you</string>
<string name="photos_facegaia_optin_impl_picker_remove_dialog_face_sharing_off_desc">You'll miss out on personalized search and creations</string>
<string name="photos_facegaia_optin_impl_picker_remove_dialog_face_sharing_on_desc">Your contacts will not receive suggestions to share with you</string>
<string name="photos_facegaia_optin_impl_content_desc_empty_avatar_view">Unnamed person</string>
<string name="photos_facegaia_optin_impl_dismiss_dialog_neg_button">Keep off</string>
<string name="photos_facegaia_optin_impl_dismiss_dialog_pos_button">Turn on</string>
<string name="photos_facegaia_optin_impl_more_faces_button">None of these</string>
<string name="photos_facegaia_optin_impl_picker_remove_dialog_positive_button">Remove label</string>
<string name="photos_facegaia_optin_impl_picker_remove_dialog_title">Remove “Me” label?</string>
<activity android:name="com.google.android.apps.photos.facegaia.optin.impl.picker.MyFacePickerActivity" android:exported="false" android:resizeableActivity="true" android:theme="@style/Theme.Photos.MyFacePicker">
<meta-data android:name="android.support.PARENT_ACTIVITY" android:value="com.google.android.apps.photos.home.HomeActivity" />
<activity android:name="com.google.android.apps.photos.settings.faceclustering.advanced.AdvancedFaceClusteringSettingsActivity" android:exported="false" android:label="@string/photos_settings_faceclustering_face_recognition_setting_title" android:theme="@style/Theme.Photos.Settings.Preferences" />
Newly added layouts:
There are also a couple of settings for managing the new facial recognition features, but they are just there to turn the grouping features on and off, or to change the face you selected in the past – perhaps you tapped on the wrong mug during setup, or maybe had plastic surgery, I guess.
<string name="photos_settings_faceclustering_advanced_face_recognition_setting_title">Face grouping</string>
<string name="photos_settings_faceclustering_advanced_face_recognition_setting_desc">Manage preferences for face grouping</string>
<string name="photos_settings_faceclustering_advanced_setting_detailed_desc">See photos of your favorite people grouped by similar faces <a href=help:>Learn more</a></string>
<string name="photos_settings_faceclustering_face_recognition_off_dialog_message">All face groups for your favorite people will be removed</string>
<string name="photos_settings_faceclustering_advanced_allow_sharing_title">Allow contacts to recognize you</string>
<string name="photos_settings_faceclustering_advanced_allow_sharing_desc">Use face group labeled as “Me” to help your contacts share faster with you <a href=help:>Learn more</a></string>
<string name="photos_settings_faceclustering_advanced_my_face_title">Face labeled as “Me”</string>
<string name="photos_settings_faceclustering_advanced_my_face_no_face_title">No face labeled as “Me”</string>
<string name="photos_settings_faceclustering_advanced_my_face_no_face_desc">Personalize features & help friends share faster with you <a href=help:>More</a></string>
My biggest question is what users will see after they take a picture of somebody else and a match is made. Primarily, will the photographer be prompted to share the photo, will it quietly include some visual marker that somebody was detected, or will it remain quiet and simply make intelligent suggestions if the share button is pressed? A big feature like this is probably not going to trigger quietly, so I expect notifications, but that could get pretty overwhelming after something like an office party or family reunion.
Unfortunately, I'm sure there are many unanswered questions, and since the implementation isn't done and there are missing pieces, there aren't going to be answers quite yet.
The potential problem for features like this is that they tip-toe right up to the line where many people feel uncomfortable about potential privacy issues. It looks like Google has done a good job of designing barriers to avoid some of those issues, and it's clear users are responsible for making their own decisions. It's important that this is an opt-in feature by default.
I do like the idea of using facial recognition to smooth out the rough edges of sharing photos with people, especially if it also acts as a reminder. I admit, I have a bad habit of taking pictures of friends and forgetting to share them later. If this goes well, Google might have the killer feature that turns Photos into a must-use service.
The APK is signed by Google and upgrades your existing app. The cryptographic signature guarantees that the file is safe to install and was not tampered with in any way. Rather than wait for Google to push this download to your devices, which can take days, download and install it just like any other APK.