EDIT: Oh by the way, Ruby folks struggling with cron might appreciate this: https://github.com/pjungwir/cron2english/
I recognize that it is a sore spot for me when people re-invent the wheel when it seems clear they didn't have to and even clearer that the energy spent re-inventing the wheel would have been better (in terms of using the wheel) spent learning why the wheel is the way it is.
It is sort of the Chesterson's Fence of computer science. Don't tell me its wrong, tell me how it is right for what it does and where it falls short for what you want it to do. Cron gets a bad rap here, as did sendmail for that matter. When taken to extremes you replace working systems like init with systems that are borked like systemd. No doubt cron is on the list of things to be absorbed by the systemd borg at some point. Yes, it makes me grumpy.
This has already happened, right? https://wiki.archlinux.org/index.php/Systemd/Timers
Not a systemd fan, myself.
I understand some of the systemd criticism, but I don't at all understand why you want something else to manage your processes than the process management system (which is basically always the init system)?
Perhaps you don't understand one of the fundamental, philosophical criticisms, which is that it violates the Unix tenet of "do one thing and one thing well".
Timing, and scheduling are difficult and (arguably, of course), deserve their own, separate utility.
Cron isn't a process manager, any more than and interactive shell [1]. It just starts processes (and passes output to a local email client, though that's, understandably, brittle).
I wouldn't object to moving the process-starting functionality out of cron and leaving it with only the ability to tell init to "start job named XYZ", but I do object to just integrating its core competency of scheduling into that init.
[1] Arguably less than, considering modern shells' job control and signal facilities
The re-invention of this one is one of my pet peeves, especially since the vast majority of the (legitimate) complaints about early implementations have been addressed in modern (last.. 10ish years?) implementations.
Syslog is lightweight, flexible, and plain text.
Back when the ELK stack first started gaining popularity, I'd get asked in interviews if I had experience with provisioning "high volume" logging. I was reasonably convinced that none of those people had ever seen (or would ever see) a high enough volume to be remarkable.
In hindsight, it was probably the same as thinking one has "big data" if it doesn't fit RAM on ones laptop (or even, more charitably, a server, even a decently large one).
It's ironic considering that syslog is not standardized. Plug any two syslog implementations and they won't work together.
The two main competing protocols are defined in RFC 3164 and 5424, both named syslog, they are incompatible with each other and have their own peculiarities.
Logstash, the L in ELK, only parses a subset of RFC3164 messages with very specific formatting.
I don't find any irony there, at all, since it's reflective of the history of Unix, and consistent with the OC's "like a zillion ways to get telemetry out". More to the point, it's fairer to say that different syslogd implementations were parallel evolution, rather than re-invention.
Now, it seems you're talking about the network protocol, whereas "syslog" the system call seems not to be too different across platforms, certainly not to the point of incompatibility. I call this out, not for pedantry, but to illustrate that, from the app's perspective, it's all the same.
> Plug any two syslog implementations and they won't work together.
Unless, perhaps, one of them understands (or can understand, with plugins) more than one protocol?
Regardless, the solution requiring the least engineering effort is to make sure one has compatible wheels everywhere, not re-inventing the wheel.
> The two main competing protocols are defined in RFC 3164 and 5424
I'm not sure that it's fair to call them competing when 5424 explicitly obsoletes 3164 (and the earlier one is "informational", while the later one is a "proposed standard").
Of course, that says nothing about the reality of the competing implementations, but if there are only two, and the older one is the most popular (which I believe it is), then perhaps there's no real competition.
> Logstash, the L in ELK, only parses a subset of RFC3164 messages with very specific formatting.
That would be an example of a (partial) syslogd re-invention. I have no idea if that input plugin is worth using (e.g. for performance reasons) rather than using something like a pipe plugin for plain text from a local syslogd. I suspect for the vast majority of use cases, it's totally irrelevant.
The two RFC are incompatible, then every implementation is slightly different in the handling of miliseconds/microsecond/timezone, the presence of the hostname field, structured data support, etc...
I am calling out your first post that syslog is lightweight, flexible and plain text. syslog suffers from serious issues.
Agreed, but you now seem to be implying that this just happens because syslog, rather than misusing the tool.
Earlier, you asserted "Plug any two syslog implementations and they won't work together" which I didn't bother to contradict, because it seemed like mere exaggeration. However, at this point, it seems like an extraordinary claim requiring extraordinary evidence.
I have, of course, routinely, in my professional career, spanning decades, successfully used different implementations with each other, usually because (more than) one implementation is embedded and can't be changed to match what's on the servers.
Exactly which syslogd implementations connected to each other over the network will fail to parse messages and drop them?
> The two RFC are incompatible, then every implementation is slightly different
I think you're reversing cuase and effect, especially given both the relatively late date of the earlier RFC in the timeline of Unix history, as well as its enduring popularity over the later RFC.
> I am calling out your first post that syslog is lightweight, flexible and plain text.
I don't believe you've addressed, in any way, the assertion that it's lightweight.
Your challenge to its flexibility is, AFAICT, poor interoperability of only the network protocol, which you haven't backed up with evidence. Even then, that's a weak refutation of the flexibility of the mechanism overall.
Your challenge to its being plain text is, again, primarily based only on network protocol, and, even then, based on a widely-unimplemented RFC.
> syslog suffers from serious issues.
This may be responding to an implied strawman, that syslog somehow perfect or ideal for all use cases, which, of course, I never said. In fact, being plain-text could be considered a "serious issue" for some telemetry.
I've used attitudes towards cron, specifically, as a personal metric for judging someone's skill and output with systems. What this team did is exactly the wrong thing to do.
I would rather use the tool with over 40 years of development work gone into it than whatever this is.
High resource usage so they moved the actual jobs off-server. Not cron related. Crontab is hard to edit so we'll create an arbitrary data structure that'll be both hard to edit and non-standard. Good job. Our jobs are failing so instead of better error handling we'll just run them again manually. Great. And apparently they've never heard of cron logging to syslog.
That's great, what does that have to do with cron?
As a result what I read was:
"We don't understand what cron does, nor do we understand how job scheduling is supposed to work, and we don't understand how to write 'service' based applications, so somebody said 'Just use cron' and we did some stuff and it didn't work how we liked, and we still haven't figured out really what is going on with schedulers so we wrote our own thing which works for us but we don't have any idea why something as broken as cron has persisted as the way to do something for longer than any of us has been alive."
I'm not sure that is the message they wanted to send. So lets look at their problems and their solution for a minute and figure out what is really going on here.
First problem was 'scalability'. Which is described in the blog post as "cron jobs pushed the machine to its limit" and their scalability solution was to write a program that put a layer between the starting of the jobs and the jobs themselves (sends messages to SQS) and they used a new scheduler (AP scheduler to implement the core scheduler).
So what was the real win here? (since they have recreated cron :-)) The win is that instead of forking and execing as cron does allowing things like standard in and what not to be connected to the process, their version of cron sends a message to another system to actually start jobs. Guess what, if they wrote a bit of python code that all it did was send a message to SQS and exit that would run pretty simply. If they did it in C or C++ so they weren't loading an entire interpreter and its run time everytime it would be lighting fast and add no load to the "cron server", this is basically being unaware of how cron works so not knowing what would be the best way to use it.
Their second beef was that cron tabs are hard to edit reliably. Their solution was to write a giant CGI script in a web server that would read in the data structure used by their scheduler for jobs, show it as a web page, let people make changes to it, and then re-write the updated data structure to the scheduler. Guess what, the cron tab is just the data structure for cron in a text form so you can edit it by hand if necessary. Or you can use crontab -e which does syntax checking, or you could even write a giant CGI script that would read in the cron file, display it nicely on a web page, and then re-write as syntactically correct cron tab when it was done.
Problem three was that their jobs were complex and failed often. This forced their poor opsen to log in, cut and paste a complex command line and restart the job. The real problem there is jobs are failing which is going to require someone to figure out why they failed. If you don't give a crap about why they failed the standard idiom is a program that forks the child to do the thing you want done and if you catch a signal from it that it has died you fork it again[1]. But really what is important here is that you have a configuration file under source code control that contains the default parameters for the jobs you are running so that starting them is just typing in the jobname if you need to restart or maybe overriding a parameter like a disk that is too full if that is why it failed. Again, nothing to do with cron and everything to do with writing code that runs on servers.
And finally their is no telemetry, no way to tell what is going on. Except that UNIX and Linux have like a zillion ways to get telemetry out, the original one is syslog, where jobs can send messages that get collected over the network even, of what they are up to, how they are feeling and what, if anything, is going wrong. There are even different levels like INFO, or FATAL which tell you which ones are important. Another tried and true technique is to dump a death rattle into /tmp for later collection by a mortician process (also scheduled by cron).
At the end of the day, I can't see how cron had anything to do with their problems, their inability to understand the problem they were trying to solve in a general way which would have allowed them to see many solutions, both with cron or with other tools that solve similar problems, would have saved them from re-inventing the wheel yet again, giving them the opportunity to experience the bugs that those other systems have fixed given their millions (if not billions) of hours of collective run time.
[1] Yes this is lame and you get jobs that just loop forever restarting again and again which is why the real problem is the failing not the forking.