What cannot be overcome is, whatever you ask Siri, will be heard by someone else. Even an innocent question to get the sports score can be something to fear asking in public. Someone will hear, and then you will have to converse with that person about your team.
Ben Brooks does an awesome job summing up something I am working on trying to get myself used to. I am a massive Bluetooth lover, and while using BT earbuds/a headset/headphones helps this somewhat, there is still an element of "where am I?" when you are thinking of using Siri, and that is a real limitation.
This is further exacerbated by my lovely fiancée converting to the Apple world, so we now have 2 devices that listen for "Hey, Siri" 100% of the time, and a third that joins the fracas when it is plugged in (my iPad Air 2). It can be fun to pit them against each other, but it would be cooler if they could figure out which is closest to me...