I was in this same boat (probably not as fit as you), but got sick a lot (pneumonia twice, bronchitis a couple of times, hospitalized with "mystery virus" twice including once leading to ICU, etc etc) no obvious signs of immune compromise, decent Vitamin D nothing showing up on physicals etc until I went to see a Rheumatologist for chronic pain. He diagnosed me with Ehlers-Danlos syndrome[1], which turns out to explain all sorts of other things that I thought were seperate and unrelated problems. It's worth drilling into these things further because there could be some underlying cause that joins all the dots together and (although there's no real treatment for EDS) you could get help.

https://www.nhs.uk/conditions/ehlers-danlos-syndromes/