As someone who has lived here for three years and travels around a lot in Singapore, not speaking mandarin can be problematic, particularly at some of the hawkers - but in precisely the same way not speaking Spanish was problematic in redwood city California (near Palo Alto).

Honestly, not being able to understand/speak singlish is probably a larger barrier to fitting in than not speaking mandarin.