Selecting which servers see your stream/connections

There are most servers where I want people to know when I'm live streaming on Twitch, and a few where I'd rather it not be known that I stream on Twitch. I also don't feel safe with people from some servers I'm in to know my Twitter account or Steam account. While I could just turn it off entirely, I still want the people from other servers to know that I'm live via streamer mode or be able to find my Twitter/Steam through my connections.