Server moving / Folder creation

Alright, I'm gonna say it, it's way to easy to accidentally move servers in your server list. Oftentimes I click on a server and it just jumps 3 slots up, or into a folder with another server. It's far too finicky. I feel like there should be a buffer for it, where you have to hold the server down briefly before you can move it.