Good point. But there is a kind of litmus test, as far as American civil rights go. If you were in Group X, you could sit down at the Woolworth's lunch counter in 1955. If you were in Group Y, you couldn't. Members of group X could have some Group Y ancestry, but as long as they were seen as X’ers, they could have had lunch there. Members of Group Y could have passed for members of Group X, in which case they could have sat at the lunch counter. The perception, not the reality, was what mattered.

Race may be difficult to pinpoint (maybe impossible if we see it as a construct). But racISM is pretty unambiguous. Racists use these complex constructs to create a rather simple system of in-group and out-group. From a racist's point of view, the only structure that matters is Us vs. Them.