Exclude users from activity feed based on server.

I'd like the option to exclude certain servers from my activity feed so that users in an excluded server that I do not share another server with won't show. It's a pain to have to click every single user to exclude them just because I started chatting in a new server.