Show HN: Dcron – Distributed, fault tolerant job scheduling system

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by dozzie on Monday August 17, 2015 @11:45AM (#10073748)

You may want to rename your project. dcron is Dillon's cron for quite a lot time already.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by feld on Monday August 17, 2015 @04:14PM (#10075802)

Yeah upon reading the title I thought Dillon had done some improvements on dcron.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by victorcoder on Wednesday August 19, 2015 @06:51PM (#10089088)

The project has been renamed to Dkron due to respect to Dillon's cron and to avoid confusion.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by jedisct1 on Monday August 17, 2015 @05:01PM (#10076084)

Yup, thought the same thing.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by victorcoder on Monday August 17, 2015 @05:05PM (#10076124)

Right I know, but if he don't mind, both projects are enough different to not create confusion.

Need more time to see if this will be necessary.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by mattzito on Monday August 17, 2015 @10:30PM (#10077290)

Honestly, it's just the classy thing to do - the impact of changing your name is nominal, the potential confusion with the other Dcron is significant.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by dozzie on Monday August 17, 2015 @05:57PM (#10076414)

No, they are not different enough. Both are cron daemons. Both are in the end supposed to be packaged. It's really enough to make the newer one (newer by decades!) to change its name.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by endymi0n on Tuesday August 18, 2015 @06:31AM (#10078394)

Man, how I wish Kala and Dcron would just work together.

https://github.com/ajvb/kala

https://github.com/victorcoder/dcron

Fusing both would be AWESOME (I'd even like to help).

What Dcron gets right:

- Distributed by design

- Raft for the reliable parts, gossip for lightweight sync, a match made in heaven

- Linux mentality

- Dashboard included

- Good docs, nice website, looks serious

What Kala gets right:

- Built much more directly after Chronos, which has an amazing and battle tested API

- Using ISO durations instead of NIHing their own format

- Dependent jobs

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by schmichael on Wednesday August 19, 2015 @09:04PM (#10089506)

While Metafora doesn't support cron-like syntax for job scheduling or tag-based node targeting (yet!), it does support distributed fault tolerant task running. We hope to build out a webui soon as well.

However since Metafora is written as a Go library and provides a statemachine implementation for task handlers to use, it requires writing your distributed tasks in Go.

https://github.com/lytics/metafora

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by Abundnce10 on Monday August 17, 2015 @06:23PM (#10076532)

I have a bunch of Python scripts that run a various intervals (hourly, daily, monthly, etc). I have trouble keeping track of which jobs fail -- unless I notice an email doesn't show up, or a metric is missing. Is Dcron something I would use to keep track of my cron jobs and whether or not they ran successfully?

At this point I don't need my jobs to be distributed, I'm just looking for a way to keep track of them all and visualize characteristics of the job (start time, length, successful, etc). Ideally, I'd have a way to re-run the job within the UI.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by fortyfivan on Tuesday August 18, 2015 @01:56AM (#10077776)

Disclosure: I work for Iron.io

If you're open to a hosted solution, check out IronWorker: http://www.iron.io/worker

It's an async task processing service with a built-in job scheduler. You can upload your python scripts to Iron.io, then set schedules and other triggers to execute on-demand. We have a dashboard to manage tasks and schedules, see what ran and what failed, and you can visualize the characteristics you're looking for. We do distribute the workloads for you, but sounds like it could be a good fit.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by abrookewood on Monday August 17, 2015 @11:18PM (#10077451)

You can also try cronitor and others like it. You add a CURL command at the end of your script that 'touches' a HTTPS endpoint. If your script doesn't check-in at pre-defined intervals, you get alerted. They are perfect for situations where you don't actually want to set up any infrastructure.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by JonM on Tuesday August 18, 2015 @03:31AM (#10077964)

Hey,

I've been tinkering with this concept recently, it's not quite ready for beta testing, but I'd be happy to take feedback on what I've got so far: http://croncloud.io/

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by victorcoder on Monday August 17, 2015 @07:30PM (#10076766)

Sure, it can do this well. It can run in a single dcron/etcd node if you don't need it to be fault tolerant. At this point you should be checking the job status in the ui as it doesn't have notifications yet.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by yashap on Monday August 17, 2015 @06:33PM (#10076573)

Check out Airflow (open source project from airbnb that does all of that very well).

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by nemothekid on Monday August 17, 2015 @08:54PM (#10077046)

I haven't done any digging, but I'm assuming this uses etcd for membership? How does that work - how does a server know whether its a leader, and how does failover work?

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by victorcoder on Tuesday August 18, 2015 @05:27PM (#10082318)

Nope, it uses serf for membership.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by nodesocket on Monday August 17, 2015 @04:26PM (#10075873)

How do I define jobs? Can they be written in any language? I didn't see anything in the documentation.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by victorcoder on Monday August 17, 2015 @05:01PM (#10076085)

Added to the docs, big omission, sorry! http://dcron.io/docs/api/#create-or-update-a-job

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by nodesocket on Monday August 17, 2015 @05:49PM (#10076377)

Any change that dcron could manage the job scripts itself? I.E. you store the raw command scripts in dcon, then reference them in the create a job endpoint?

For example:

    POST /scripts --data 'node /home/node/myJob.js'

That returns a script id { "script_id": "507f1f77bcf86cd799439011" }

Then:

    POST /jobs --data '{
       "name":"cron_job",
       "schedule":"@every 2s",
       "script": "507f1f77bcf86cd799439011",
       "owner":"foo",
       "owner_email":"[email protected]"
    }'

This way you don't have to manage/create the scripts on each node manually.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by victorcoder on Monday August 17, 2015 @06:13PM (#10076487)

I think you missed the point, jobs are stored in etcd so they're distributed and all dcron (server) nodes will see it. You can even create jobs from any node. Note that, non server nodes doesn't need access to etcd.

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by andyeskridge on Monday August 17, 2015 @05:00PM (#10076082)

I was looking into that too. Based on this job.json[0] definition from the git repo, it looks like it just executes the command. So I believe you could have the job written in any language.

But I think this also means you have to copy every job executable to every server that might have a chance of running the job.

[0] https://github.com/victorcoder/dcron/blob/master/job.json

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by kylequest on Monday August 17, 2015 @04:55PM (#10076053)

Looks like it's defined using the format you see in this json file: https://github.com/victorcoder/dcron/blob/master/job.json

The "command" field is the most important part. What you put there gets executed like this by their agent (on Linux): /bin/sh -c <your_scheduled_command_name>

[ Parent | on HN ]

Re:Show HN: Dcron – Distributed, fault tolerant job scheduli... by carl_ on Monday August 17, 2015 @07:36PM (#10076786)

Cool! Might use this when I renovate http.sh

[ Parent | on HN ]