"I'd Agree With You If You Weren't Such A Jerk"
The "I won't agree with jerks" argument is, if taken at face value, an argument based on privilege. It essentially says that the substance of a political view is unimportant. Political ideologies are just arbitrary markers of group belonging, like baggy vs tight pants or "soda" vs "pop" or the brand of truck you drive. From that viewpoint, what matters is whether you want to associate with a particular group of people. And so their jerkishness becomes an important criterion, in the same way that you wouldn't invite jerks to go out for a beer with you.
"Xists are jerks" may be a reasonable empirical/causal explanation for why Xism isn't more popular (given the factual prevalence of the fallacious "I won't agree with jerks" argument), and so an Xist might reasonably argue that they and their fellow travelers ought to tone down their jerkishness*. But it's not a justification for a non-Xist to remain a non-Xist. The only justification for being a non-Xist is to argue that X is wrong on the merits.
*The empirical claim is, of course, debatable, and there are other considerations that weigh in favor of (certain forms of) jerkishness.