Note that not all 24 of those keys are suitable for age usage. The 4 main keys have specific usage definitions in the PIV specification that mean hardware tokens alter how those key slots behave. Only one of them (the KeyManagement slot) has a definition that allows encryption, and even that I was somewhat suspicious of overlapping with, as I couldn't predict how those existing keys were being used, and didn't want to support every possible key type that might be in that slot (which users likely wouldn't be able to alter).
age-plugin-yubikey avoids this complexity by only interacting with the 20 "retired" slots, which have no constraining definitions. (I am considering adding restricted support for the KeyManagement slot specifically for CAC card users who aren't allowed to add new keys to their cards [0], but this would be behind a default-off feature flag to keep the primary UX simple.)
But perhaps age is better suited as a backend in other apps, especially Go apps. A good feature of PIV applet of Yubikey 5 is that it stores 24 keys. compared to 3 in OpenPGP (although any app including GPG could make use of those 24 slots).
What I am looking for these days is a back up tool that uses age. Can an app like restic use age as backend?