Vegetarianism and Privilege
First, we have to be clear on what the poll showed. The racial breakdown is only given for whether people don't eat meat -- but the poll also asked separately about eating poultry and seafood. So it may be that people of color are more likely than whites to eat poultry and/or seafood but not red meat, and that makes up for their lower likelihood of being true vegetarians. This is especially plausible given that there is probably a disproportionate percentage of people of color who give up meat out of economic necessity rather than ethical commitment.
Second, the claim that "vegetarianism is white privilege" (I've much more often heard it charged with being class privilege**, but a similar set of considerations would apply) is about more than what percentage of what kind of people do it. The dominant presentations of vegetarianism in our society are framed in a white middle-class way. Here I'd draw a parallel with feminism -- it's still fair to say that the dominant presentations of feminism in our society are laden with white and middle/upper-class privilege, even though women of color are no less committed to gender equality and justice. The question of who has the most power to define what vegetarianism is all about remains even if all groups are interested in the underlying principles.
*The poll didn't mention races other than the three largest ones.
**The poll doesn't give a breakdown by income, but it does suggest that higher levels of formal education are associated with (statistically insignificant) higher levels of non-meat-eating.