It is a win/lose situation in my opinion. You need to have the kick option, not only to kick players that are verbally aggressive (you can mute those) but to also kick idle players who mess up your team games (e.g. a teammate of yours is idling in game).
But yeah, I see it every time and happens to me quite often (or well, as often as I win a game). The winner gets a vote to kick. It is frustrating, I can also live with that if it happens 2-3 times per session, but if it happens more, I simply vote the lowest ranked player first (since they won't kick him) before they try to kick me.
That works because you can only have 1 vote to kick per lobby, meaning that if you vote a player to be kicked, the other players cannot vote another one. At least that happens when the first player to get voted, does not get kicked. If he does get kicked, I don't know if the players can vote another.