It depends on why you wouldn't give a user sudo. If you're worried that they might get bored and do an immature prank, or do something ill-advised (like changing the root password, or giving sudo to someone else) and render the system insecure/inoperable/unmaintainable, you probably can give them shell access. A good example here would be giving shell access to employees or the like, if their job is aided by it. The time and effort it takes to research a privesc vuln is usually sufficient to deter them, and if it isn't, you just revoke access and fire them if they do it.

If you're worried that someone might be trying to deliberately compromise your security, you can't give that person the ability to run code on your system.

the main reason you don't give users sudo is so they don't do anything stupid, not so much to prevent them from acting maliciously.

Correct.