I think it depends. There are really 2 types of “racism”.

There are racialised class systems created by nation states and imperialist powers to supress wages and fuel industries. This type of racism has literally nothing to do with physical characteristics, and fantasy races poison the well in this case.

But it is also the type of racism that fantasy mostly ignores. Most media focuses on the liberal theory of racism, in which races with differing characteristics cannot reconcile their differences due to a lack of mutual understanding. In this case, making the different races really be different is not necessarily a bad allegory. It is just an exaggerated presentation of what writers perceive the racial conflict to be.