I guess it depends what you mean by 'complex'. Doing basic statistical summaries on a few columns of a TB or so of some logfile in a weird format would be a big project using a lot of the tools I've had to use, but it's more or less 'hello world' for awk. I really like it as part of an "ETL before you ETL" process where you massage your data into a format that makes your official ETL tool with fancy capabilities not choke.
The simplicity of awk also gives it a cultural advantage over things like perl in that you can argue that you're not really introducing another language into your environment, just including some glue logic with a Standardized Tool Everyone Should Know.
The only thing I'm familiar with that does such a good job of keeping simple things simple even when distributed or at largish data volumes is Splunk.
=> 1.upto(100) { |n| puts `curl -s fizzy.heroku.com/#{n}` }
1
2
Fizz
4
Buzz
... from subprocess import call
call(["ls", "-l"])
or import os
os.system("ls -l")Compare
from subprocess import call
call(["ls", "-l"])
to `ls -l` for listing in ls('-l'):
'''etc'''
Here's the library in question - https://pypi.python.org/pypi/pbs - it makes me super happy to do shell work in python.Also, when using Popen, calling .wait() can cause problems if you are expecting large amounts of info back from stdin or stderr. Using .communicate() is generally better.
It's a shame Scheme Shell didn't catch on more. That had a syntax for pipes that simply used the pipe symbol in a way that feels a lot like just using pipes in bash. Unforunately, the project page is down, and the Wikipedia entry doesn't have the best examples.
Sam and vi are both descendants of ed, but go in very different directions. Is there a path forward from awk that focuses on different strengths to perl's choices? (perhaps avoiding perl's move to being multi-paradigm)
Is there something that sits close to unix in the way awk does, but which is stronger?
The editors have had major impact on our languages mainly because regular expressions, called regular sets at the time, where implemented in Thompson's ed which was based on an earlier line editor implementation called qed used on ctss and multics.
One can follow the evolution almost in dialect fairly well.
ed -> grep -> sed -> awk
ed -> em -> ex -> vi
awk -> perl -> ruby
so
ed -> grep -> sed -> awk -> perl -> ruby
The right tool for the right job when all you know is a chainsaw everything looks like a hammer unix process are cheap and all that yada yada philosophical paradigmy finite state automata pipelines vs pointers vs classes vs recursive enumeration iterative parenthesis backtick mind expansions expression logic =P
Never underestimate the power of intellectual inertia, and the lengths people will go to in order to continue using their favorite tool - no matter how poorly suited to a task it might be. ):
from sys import stdin, stdout, stderr
for n,line in enumerate(stdin):
stdout.write(str(n) + '\n')
stderr.write(line)
Is that the kind of reading and writing you are talking about?Compare this to a shell where the combination of &&, ||, and | can fit on a single line what would have taken dozens of lines of python.
For _that_ type of python, I usually use the sh module[0]. It handles piping pretty well, especially if you use StringIO to deal with the pipes. Unfortunately, return code-based conditionals are nowhere as simple as || and && are in shell.
Take Python, which is supposedly a "scripting" language, but requires relatively painful amounts of boilerplate to actually read from or write to pipes, etc. It doesn't force you to keep everything in Python, but it certainly nudges you that way. Without naming names, certain statically typed languages that are obsessed with safety are even worse in this regard.