Re:Why Awk for AI? (1997)
by bcoates on Friday May 17, 2013 @03:48PM
(#5726362)
> I don't think anyone would seriously advocate using awk for complex projects these days
I guess it depends what you mean by 'complex'. Doing basic statistical summaries on a few columns of a TB or so of some logfile in a weird format would be a big project using a lot of the tools I've had to use, but it's more or less 'hello world' for awk. I really like it as part of an "ETL before you ETL" process where you massage your data into a format that makes your official ETL tool with fancy capabilities not choke.
The simplicity of awk also gives it a cultural advantage over things like perl in that you can argue that you're not really introducing another language into your environment, just including some glue logic with a Standardized Tool Everyone Should Know.
The only thing I'm familiar with that does such a good job of keeping simple things simple even when distributed or at largish data volumes is Splunk.
I guess it depends what you mean by 'complex'. Doing basic statistical summaries on a few columns of a TB or so of some logfile in a weird format would be a big project using a lot of the tools I've had to use, but it's more or less 'hello world' for awk. I really like it as part of an "ETL before you ETL" process where you massage your data into a format that makes your official ETL tool with fancy capabilities not choke.
The simplicity of awk also gives it a cultural advantage over things like perl in that you can argue that you're not really introducing another language into your environment, just including some glue logic with a Standardized Tool Everyone Should Know.
The only thing I'm familiar with that does such a good job of keeping simple things simple even when distributed or at largish data volumes is Splunk.