A way to turn off "Mutual Servers" for anyone but your friends

When you click on a members profile, you can see what mutual servers you're in. But there also isn't a way to keep this private, which would be helpful if you wanted to keep your servers private. It would be nice if there was an option maybe in "Privacy & Safety" to make it so only friends can see your mutual servers, or no one can.