Like "I am an app, dear HSM, please perform following crypto operation for me."
If an attacker can pretend to be a legitimate HSM user, then she does not need key access, she just asks the HSM to perform a crypto operation on her behalf.
On the other hand, if the HSM needs secrets in order to authenticate legitimate users, then those secrets are prone to those attacks, against an HSM shall protect.
Or dont i get it?
The real benefit of HSMs is that they can make low-level keys accessible via a high-level interface that restricts the type of operations that can be performed at all. For example, an HSM will never just hand raw keys to application servers, but rather only let them perform various operations (based on their permissions). The higher level and specific these permissions are, the better.
To authenticate an application, you should generate a client certificate and share to the application, in order to create a mutual authentication trust. When you request some operations to HSM you need to authenticate yourself with the certificate. Of course the certificate must be kept as a secret and not shared with anybody. There is also a sort of RBAC scheme related to client certificate.
They're most useful if they can perform high-level operations, such as "please validate whether the entered card PIN <encrypted PIN block> here matches <PIN verification value on file>, given <credit card number>". The output of that example operation would only be a single bit of information (yes or no), rather than e.g. leaking the entire correct PIN, or even just the decrypted PIN that was entered at the POS.
But even just a signing/decryption oracle can be a step up from just storing long-lived private and secret keys on your application servers, where you'll never know for sure whether they were exfiltrated at some point.
[1] https://en.wikipedia.org/wiki/1-Click