pleroma's media proxy is works just like that and the user experience is fine. actually, i've run instances that don't have caching configured and it still works fine (if a bit unfriendly to other instances).