It depends on the context.
If you were just describing someone, for instance you are trying to meet someone for the first time in a crowded place, yup that's cool. However, be sure that it is a descriptor and not a classifier.
Just like how you would point out someone that wears glasses or has red hair.
"I'm looking for this person, he is wearing a brown jacket, a blue hat, about six feet tall, and is South Asian."
The big difference is if the person's race is the only thing that you describe and usually in a negative way.
"Some stupid Brown woman cut in front of me in line at the grocery store."
All about the context, delivery, and purpose.