Americans absolutely call people brown. It's all over social media, sociology papers, and common conversation. Just search twitter for "black and brown".