It turns out this is never ok to say to someone of a different race than you.  Even if they just said that your jacket color was stupid and it seems like the perfect comeback.  It will just come across as racist.