Tip: If you can't video call your contacts on Duo via Assistant (or Smart Displays), try saving their numbers as '+ country code'
Tip: You can video call your Smart Display at home from your phone (with both on the same Duo account)
Tip: Hide personal results from your Smart Display if you don't want to spoil surprises or share your content recommendations
Having a Google Assistant Smart Display, such as the Lenovo, JBL Link View, LG ThinQ WK9, is very convenient around the house. But if you don't pay attention, you may end up with unwanted privacy concerns because of the mere fact that these displays don't differentiate between people looking at them. They know voices and will only respond to you, but they don't know faces. If there's a personal card on the homescreen, anyone can read it or tap it to see more. Which is how you may end up revealing your surprise party plans to your spouse and your full calendar to boot. There's a simple solution for it though, you just need to know where to look.
In the Google Home app, open the side menu and tap More settings. Scroll down to the Devices section and look for your Smart Display in the list (in my case, it's my "Kitchen Display"). Open it and look for two toggles: Personal results and Home screen & notifications.
If you disable Personal results, the display won't recognize your voice anymore and you'll be treated like a guest. You can't set or see calendar events and reminders, ask it to show your own photos, check your commute time, add items to your shopping list, call your contacts, make payments, trigger your routines, ask about the things you told it to remember, or get any personalized info or recommendations.
If that's what you want, then you're free to toggle it off, but odds are you still want access to all that info, you just want it to be tied to your voice and not visible to anyone else.
In that case, you'll need to keep the first toggle on, but disable Home screen & notifications. With it off, the device will still answer all your queries when it recognizes your voice, but it won't show anything personal on the homescreen for others to see or tap. The Coming up card with your reminders and events will be gone, so will your commute card and your YouTube personalized recommendations. That way other users in your household and guests won't be able to know your schedule or influence your YouTube history. No more spoilt surprises!
If this is a concern for you, I recommend you disable the second toggle. It's a little less convenient, but it helps with your privacy.