Hangouts does some things well, but letting you know when someone new is trying to contact you is not one of them. Google is rolling out several new invitation options to Hangouts that will make things much clearer. What's more, it looks like they're already live in the Android app.

If you head to the Hangout settings you'll see the "How others get in touch with you" heading. Below that is "Customize invites," which is where the new features are. The default behavior in Hangouts when someone new messages you is that you will get an invitation to chat. Now you can set different behaviors for people who have your email, phone number, and everyone else.

The settings for each case allow people to contact you directly (no invite needed), send an invite (the default), or they can't contact you at all. You can set each category individually, so perhaps someone who has your phone number can IM you immediately, but people with only your email still have to send an invitation, and strangers can't even do that. These features should just be in the app right now, no updates needed. If you don't see them yet, the rollout should be done soon. Just make sure you're on the current version of Hangouts.