T5 was open-sourced on release (up to 11B params): https://github.com/google-research/text-to-text-transfer-tra...
It is also available via Hugging Face transformers.
However, the paper mentions T5-XXL is 4.6B, which doesn't fit any of the checkpoints above, so I'm confused.
T5-XXL looks on par with CLIP so we may not see an open source version of T5 for a bit (LAION is working on reproducing CLIP), but this is all progress.