You may have heard a lot about artificial intelligence (AI) and machine learning (ML) in recent years as the demand for each continues to rise. These two terms in the technology world have been tossed around for one specific reason: They take our device hardware beyond its physical limitations. For example, the best Android phones rely on an AI or ML model to achieve even more impressive photos from the camera. This guide discusses Google ML Kit and how it uses efficient on-device processing to improve our smartphone experience.

Google's long journey with AI and ML features

Google has been one of the biggest supporters of using on-device processing in smartphones, dating back to the original Google Pixel. One of the earliest uses of leveraging AI and ML models was for the camera, which is still one of the leading forces in today's mobile devices. Google's dedication and power in AI and ML are still evolving on their most recent Google Pixel 7 and Google Pixel 7 Pro smartphones. And with the Google Tensor chip powering these devices now, they continue to surpass many hardware limitations, as they emphasized all those years ago.

With Google's massive success and leadership in AI and ML, Google also wanted to offer the same power they use to all mobile app developers. Because without third-party app developers constantly creating new experiences, our smartphones would have less character and uniqueness about them. Since these developers are crucial to the success of the mobile industry, giving them everything they need to succeed makes perfect sense. That's where Google ML Kit comes into play. No longer are these powerful on-device AI and ML features exclusive to Google.

Google ML Kit: An Introduction

Announced at Google I/O 2018, Google ML Kit was designed with ease of use to bring on-device machine learning features to mobile devices. Google wanted the process to be as seamless as possible, allowing developers to focus more on what they do best — innovate. As such, developers on Android and iOS can take the free software development kit (SDK) that Google provides and apply it to their apps. This enables a new world of possibilities with machine learning features, pipelines, and APIs powered by Google's advanced in-house ML models.

The ML Kit SDK is optimized for mobile devices, ensuring you get the best experience. In previous methods, machine learning models required a separate dedicated server, relying on the cloud to communicate with a mobile device. Tapping into the power of on-device functionality using Google ML Kit allows app developers to run these complex operations offline. With hardware acceleration on supported devices, you get more performance gains and less latency. Using the easy-to-implement APIs for Google ML Kit also removes the complicated step of gathering data and training a machine learning model.

Privacy also comes to mind when discussing on-device versus cloud-based computing and machine learning models. By offering these features in a local setting, the data never leaves the mobile device. It works without a network connection. When using Google ML Kit for these computations, an app uses less mobile data since it doesn't rely on a cloud server. Still, ML models and datasets are typically large in scope or size when stored in the cloud. Reducing the overall footprint to run on a mobile device lessens their capabilities, but they're still quite powerful.

What Google ML Kit APIs are available?

As of this writing, mobile developers can effortlessly implement 12 primary turn-key APIs into their apps using Google ML Kit. Combining this with Flutter, another open source framework from Google, further simplifies the app-building process. Google continues to enhance and optimize these pre-trained models and will add new ones when ready for public use. There is also an interactive tutorial to help you design your first on-device ML app, regardless of your machine-learning expertise.

Here's the list of available Google ML Kit on-device APIs:

• Vision APIs: Barcode scanning, face detection, text recognition, image labeling, object detection and tracking, digital ink recognition, pose detection, and selfie segmentation.
• Natural language APIs: Language identification, text translation, smart reply, and entity extraction.

Google also offers additional tools for developers that need more than the default turn-key solutions. Regarding specific use cases beyond the standard APIs or ML models, Google gives developers complete control over their datasets. They can retrain current ML models entirely or train one of their own using a preferred method of choice. For example, you can prepare a custom ML model on TensorFlow and bring it into ML Kit. Then you can import the model to your app with Gradle via Android Studio and configure it for production using the Firebase console.

What is TensorFlow?

TensorFlow is another helpful open source tool by Google that developers can use to manage their machine learning models and data dependencies. As a platform designed for deep learning, TensorFlow can process and deploy any ML model, including the default pre-trained ones or custom user models. When ready for deployment, they can be used on-device, in a web browser, or in the cloud, giving developers as much flexibility as they need for their ML model needs.

Natural language processing is how these ML models can do advanced things like recognizing items, people, or text in a photo. Mimicking the human brain, a neural network uses specific algorithms from datasets to train the models via patterns. And they are typically trained using massive data dependencies of billions of images, words, or other sets of information. It can take days, weeks, or even months to finish this process, depending on the size and scope of the project. TensorFlow allows developers to train their ML models without negatively affecting the speed and performance of the process.

If a developer wants to move beyond the default pre-trained ML models, there are custom TensorFlow Lite models. Like Google ML Kit, TensorFlow Lite focuses on bringing these types of on-device experiences to mobile devices — on Android and iOS. Another option is to use TensorFlow Cloud with Google Cloud, allowing for a complete online-based ML model environment. Hosting, training, and managing ML models in the cloud is becoming more popular with many developers. Relying exclusively on cloud computing allows for greater flexibility without dedicated hardware, reduced overall costs, and a more user-friendly experience.

Google ML Kit samples are available on GitHub

Aside from everything Google ML Kit offers to mobile developers, there are open source app samples that anyone can download and try from GitHub. For each of the 12 turn-key APIs mentioned above, Google has a sample app showcasing what they can do. These quickstart samples are available on Android and iOS, letting you try them on any device you own. Visit the ML Kit samples website, find one you want to demo, then click Try it on Android or Try it on iOS. It takes you to that GitHub listing with instructions on how to use it.

Real-world uses for on-device ML models

We covered what Google ML Kit is and how app developers can easily tap into its features, but how about some real-world examples of it in action? What might be possible when third-party app developers use on-device ML models for mobile devices? You still benefit equally if you're not an app developer using Google ML Kit. As the user and consumer, many of your favorite third-party apps can offer new on-device ML features that weren't possible until now. We'll mention just a few examples below to paint a better picture of what's possible.

We touched on the camera a bit earlier. However, there's so much more that ML models can offer other than taking better photos. One example Google highlights is how Adidas leverages on-device features using Google ML Kit. According to the case study, Adidas is tapping into the object detection and tracking API to create a new shopping experience. Powered by augmented reality, customers can open the Adidas app on their smartphone and scan shoes in real time with the camera. This allows them to find more information about a product at leisure without other interactions.

Another case study mentions how a top-rated Android app called Lose It! enhances the experience of its users by utilizing Google ML Kit. It is a health-based app with a calorie counter and food diary to better track food intake and weight management. The company is tapping into ML Kit to replace its own ML models in the cloud with the new ones that work on-device. This allows them to add an instant offline text scanning feature to let users fill in food nutrition information by scanning a label in real time.

Google ML Kit brings machine learning to all

For years Google has excelled and exceeded expectations regarding its success and leadership in AI and ML. Their secrets were initially locked down and only accessible by Google until 2018, when Google ML Kit officially arrived. Creating, training, and deploying a unique ML model is tedious and costly, but that's no longer the case. Google ML Kit removes the guesswork and gives developers everything they need to deploy ML models on mobile devices. Since on-device machine learning opens the doors for the future, we expect to see more possibilities in the coming years.

With how popular AI-powered chatbots have become, you should learn what Google Bard is and what it can do. We cover all the details you need to know, including where it got its name and how you can try it out today.