https://datatracker.ietf.org/doc/draft-ietf-httpbis-message-...
Library support for it is spotty, depending on your framework of choice but they exists.
Your second approach is covered by oAuth refreshable tokens I believe.
I went through the same process as you in an API project, eventually ended up using Keycloak as the auth server.
As for the second approach, yes exactly it's very similar to OAuth2. In fact I guess this is the method used by GCP APIs by service accounts. The clients use their long-lived secrets to get an oauth2 access token, this access token is JWT I guess that also contains authorization information such as scopes.
But after much googling it appears not! Seems like a fairly simple additional variables aws could pass through and condition to support, so you can say “only allow this request if the signing time is < 1min ago”, closest you can get is that the policy is only valid after X date! Is a shame!
I ended up implementing an additional auth API endpoint where you use your long-lived secret to periodically request a short-lived JWT that is only valid for a few hours, this short-lived secret is what is used for the actual APIs to authenticate the users.