Hermit-in-residence doesn't sound so terribly different from artist-in-residence or writer-in-residence.

It seems like more of a framing issue, "garden hermit" feels like an eccentric ostentation, writer-in-residence feels like philanthropic patronage