That's true if you're talking to police but not if you're on the stand. If you're on the stand you have the opportunity to convince the jury that you're a trustworthy person.

In fact, even talking to police has the possibility to make you better off for example if you can wrongly lead them to not think you're the suspect.