Re:Ask HN: When does it make sense to roll your own X?
by KaiserPro on Sunday July 14, 2019 @10:24AM
(#20433552)
It can't. Not easily.
Job scheduling is trivial. Doing it reliably, tracably and in a way that handled failure predictably is really a pain in the arse.
There are zillions of ETL frameworks out there, just use those. some are unspeakably complex (airflow I'm looking at you) Some are simple, but require glue (AWS batch) some are trendy (argo on kubernetes)
However, most CI tools are effectively job scheduling systems. Jenkins works surprisingly well as a cron replacement for >100 machines
Job scheduling is trivial. Doing it reliably, tracably and in a way that handled failure predictably is really a pain in the arse.
There are zillions of ETL frameworks out there, just use those. some are unspeakably complex (airflow I'm looking at you) Some are simple, but require glue (AWS batch) some are trendy (argo on kubernetes)
However, most CI tools are effectively job scheduling systems. Jenkins works surprisingly well as a cron replacement for >100 machines