Exclude specific friends or servers from seeing your Game Activity

As the title states, showing your game activity is currently an all-or-nothing toggle. Either everyone, everywhere sees everything you're playing, or no one sees anything at all. As a guy who has both friends, family and coworkers connected to the same account, and is a member of both private and business related servers, it'd be great to be able to toggle my Game Activity visibility on a server and person basis.