Server 100 Limit

I was sent a request to join a server, but when I clicked on the accept button, it showed a message saying, "You are at a 100 server limit. You must leave a server first to rejoin." I left multiple servers and individually counted as well and I only had around 75 servers. I asked for another invite but it still showed that message.