Option to have private discord voice chat within a discord server channel

My friend and I thought it would be a great idea if we could have a private voice chat within a discord voice channel. Since we go in guild dungeon runs where we all have to be muted to hear the guild leader, we can't talk to each other within the channel. And this is where it would be cool if we could somehow privately talk to each other yet still hear the guild leader.