Or you could use at(1) or batch(1) to invoke the job once. That seems significantly easier and less error prone than trying to clean up crontab.

at(1) is great, but you'll still have the problem of a different environment to cron (it copies most of your environment variables and working directory).

Or use Puppet/Salt/Chef/ansible to manage the file automatically

I've seen this done previously. It gets arduous to maintain a large list of cron jobs, teach everyone how to add new ones, rebuild the cron box and redeploy on updates to config management.

Just use Jenkins, until you outgrow Jenkins. Then use a distributed scheduler.

Can you recommend one?

We recently had to implement a prototype in house, not that easy as I thought it would be, but we could not find a good solution, e.g. handle 100k jobs, long term (never expiring), short term (depends on an entity's live cycle), high available.

Thanks.

Kubernetes if you're a containers first shop, Hashicorp's Nomad if you're not just containers (can run binaries, etc).

Kubernetes does Cron jobs.