Display non-members of private channels

I spend a lot of time in private channels, and the problem I run into is the users listed are obviously only the ones allowed in the channel; I rarely see the entire user list of the server. I'd rather have everyone always be shown, but I guess you could display users as crossed out if they aren't allowed in the channel that the user is currently being shown.