Nope. It's for customs, not immigration. You're already in the country, they just need to figure out whether they need to search your bags for contraband.

So then if you refuse to answer they will just search your bags.

Customs agents are supposed to do that. How else do you think smuggling gets stopped? It's pretty much universal that your belongings can be searched (then there are some countries expand that to searching digital data, but that's a bit of a perversion).

This is being used by literal customs agents.

That's the point. If you refuse then they do a thorough search.

If you answer suspiciously, or refuse to answer, then you get the full search.

This is similar to how you can refuse body scans at the airport, but if you do then you get the full pat down search.