Sometimes when clicking a server, I accidentally drag it, delaying me

Sometimes when I click a server, I accidentally drag it a bit up or down instead, causing it to be temporarily unclickable for like a second.  It's a minor thing but it's kind of annoying and I wish they would fix it.  They could make it not freeze a bit like this, or make it not initiate dragging until the mouse drags off the server icon.