Spotify Listener Linking

When I start a listening party, my friend will join, and we will finish the song together, but when the next song starts, he's a minute ahead of me. It's just weird.