They almost definitely used existing models for generating it. The human feedback part, however, is the expensive aspect.