News for hackers. Stuff that matters.
Re:Why Awk for AI? (1997) by xaa on Friday May 17, 2013 @01:01PM (#5725485)
I don't think anyone would seriously advocate using awk for complex projects these days, but the idea of keeping data really close to the OS/shell is a very powerful one.

Take Python, which is supposedly a "scripting" language, but requires relatively painful amounts of boilerplate to actually read from or write to pipes, etc. It doesn't force you to keep everything in Python, but it certainly nudges you that way. Without naming names, certain statically typed languages that are obsessed with safety are even worse in this regard.

Re:Why Awk for AI? (1997) by bcoates on Friday May 17, 2013 @03:48PM (#5726362)
> I don't think anyone would seriously advocate using awk for complex projects these days

I guess it depends what you mean by 'complex'. Doing basic statistical summaries on a few columns of a TB or so of some logfile in a weird format would be a big project using a lot of the tools I've had to use, but it's more or less 'hello world' for awk. I really like it as part of an "ETL before you ETL" process where you massage your data into a format that makes your official ETL tool with fancy capabilities not choke.

The simplicity of awk also gives it a cultural advantage over things like perl in that you can argue that you're not really introducing another language into your environment, just including some glue logic with a Standardized Tool Everyone Should Know.

The only thing I'm familiar with that does such a good job of keeping simple things simple even when distributed or at largish data volumes is Splunk.

Re:Why Awk for AI? (1997) by tieTYT on Saturday May 18, 2013 @12:44AM (#5727939)
What is Splunk? Are you talking about this: http://en.wikipedia.org/wiki/Splunk
Re:Why Awk for AI? (1997) by dpritchett on Friday May 17, 2013 @01:22PM (#5725588)
I shamelessly reach for the backtick shell notation in any exploratory Ruby code that needs it. I try to replace it with something written in Ruby if the codebase grows beyond my laptop, but backticks never get old.

    => 1.upto(100) { |n| puts `curl -s fizzy.heroku.com/#{n}` }

    1
    2
    Fizz
    4
    Buzz
    ...
Re:Why Awk for AI? (1997) by manish_gill on Friday May 17, 2013 @02:22PM (#5725924)
Ha! That backtick is the only reason most of my "scripts" are being written in Ruby instead of Python. :)
Re:Why Awk for AI? (1997) by sauravc on Friday May 17, 2013 @02:48PM (#5726051)
Seems pretty easy in python:

  from subprocess import call
  call(["ls", "-l"])
or

  import os
  os.system("ls -l")
Re:Why Awk for AI? (1997) by shawabawa3 on Friday May 17, 2013 @06:06PM (#5726919)
The difference between "pretty easy" and "easy" changes everything.

Compare

  from subprocess import call
  call(["ls", "-l"])
to

  `ls -l`
Re:Why Awk for AI? (1997) by bashinator on Friday May 17, 2013 @08:33PM (#5727428)
I really like the python sh library for stuff like this. Would you rather do what you wrote, or this?

    for listing in ls('-l'):
        '''etc'''
Here's the library in question - https://pypi.python.org/pypi/pbs - it makes me super happy to do shell work in python.
Re:Why Awk for AI? (1997) by CatMtKing on Saturday May 18, 2013 @02:00AM (#5728050)
Wow, that's a neat library. You can just import any program and use it like a function (from sh import grep)! Thanks for the link. Also, sh documentation: http://amoffat.github.io/sh/
Re:Why Awk for AI? (1997) by bashinator on Sunday May 19, 2013 @03:06PM (#5734036)
Yeah, it's replaced just a ton of boilerplate `def run(*args):` functions wrapping Popen. I couldn't be happier.
Re:Why Awk for AI? (1997) by mkarttic on Saturday May 18, 2013 @04:42AM (#5728292)
i use Sh to interface with my cmus player. from sh import cmus_remote! :)
Re:Why Awk for AI? (1997) by tomsthumb on Friday May 17, 2013 @03:23PM (#5726253)
While there are several ways to skin that cat, the suggested way is to use subprocess.Popen .

Also, when using Popen, calling .wait() can cause problems if you are expecting large amounts of info back from stdin or stderr. Using .communicate() is generally better.

Re:Why Awk for AI? (1997) by bra-ket on Friday May 17, 2013 @03:43PM (#5726343)
this. python has too many ways to skin the cat.
Re:Why Awk for AI? (1997) by tomsthumb on Friday May 17, 2013 @03:57PM (#5726405)
hehe, while I do agree in this particular situation, it's rather funny to see that in writing.
Re:Why Awk for AI? (1997) by stcredzero on Friday May 17, 2013 @02:29PM (#5725957)
> Take Python, which is supposedly a "scripting" language, but requires relatively painful amounts of boilerplate to actually read from or write to pipes, etc.

It's a shame Scheme Shell didn't catch on more. That had a syntax for pipes that simply used the pipe symbol in a way that feels a lot like just using pipes in bash. Unforunately, the project page is down, and the Wikipedia entry doesn't have the best examples.

http://en.wikipedia.org/wiki/Scsh

Re:Why Awk for AI? (1997) by cturner on Friday May 17, 2013 @01:22PM (#5725600)
What would you put forward as an alternative to awk for the kinds of complex projects the author describes?

Sam and vi are both descendants of ed, but go in very different directions. Is there a path forward from awk that focuses on different strengths to perl's choices? (perhaps avoiding perl's move to being multi-paradigm)

Is there something that sits close to unix in the way awk does, but which is stronger?

Re:Why Awk for AI? (1997) by silentbicycle on Friday May 17, 2013 @01:31PM (#5725665)
I use lua for that, and it seems to be a very good fit. There's awk influence in its history, so it's probably no accident.
Re:Why Awk for AI? (1997) by UNIXgod on Friday May 17, 2013 @06:27PM (#5726988)
Ruby does awk for the most part. I can't comment on strength since awk has a small footprint compared to both ruby and perl clocking in just under 96k. Also both emacs and ed are descendants of TECO =)

The editors have had major impact on our languages mainly because regular expressions, called regular sets at the time, where implemented in Thompson's ed which was based on an earlier line editor implementation called qed used on ctss and multics.

One can follow the evolution almost in dialect fairly well.

ed -> grep -> sed -> awk

ed -> em -> ex -> vi

awk -> perl -> ruby

so

ed -> grep -> sed -> awk -> perl -> ruby

The right tool for the right job when all you know is a chainsaw everything looks like a hammer unix process are cheap and all that yada yada philosophical paradigmy finite state automata pipelines vs pointers vs classes vs recursive enumeration iterative parenthesis backtick mind expansions expression logic =P

Re:Why Awk for AI? (1997) by smalltalk on Saturday May 18, 2013 @01:17AM (#5727989)
ed is not a descendant of TECO.

http://web.mit.edu/kolya/misc/txt/editors

Re:Why Awk for AI? (1997) by smalltalk on Saturday May 18, 2013 @10:33AM (#5729027)
Is this factually incorrect? If so, please cite the source that contradicts the common knowledge that ed descended from QED, developed entirely separate from the MIT environment that led to EMACS. Which explains why EMACS couldn't be less like a good UNIX program.
Re:Why Awk for AI? (1997) by JulianMorrison on Friday May 17, 2013 @02:48PM (#5726052)
Hadoop and friends.
Re:Why Awk for AI? (1997) by voyvf on Friday May 17, 2013 @02:16PM (#5725881)
> I don't think anyone would seriously advocate using awk for complex projects these days,

Never underestimate the power of intellectual inertia, and the lengths people will go to in order to continue using their favorite tool - no matter how poorly suited to a task it might be. ):

Re:Why Awk for AI? (1997) by sampo on Friday May 17, 2013 @07:05PM (#5727136)
One of the first video lectures in Coursera's Machine Learning course, Andrew Ng gives a pretty similar explanation why he chose Octave as the language for the course; in his experience, students get more done in Octave than in any other language he has tried teaching with.
Re:Why Awk for AI? (1997) by sprayk on Friday May 17, 2013 @02:34PM (#5725977)
It has always pretty easy for me to read from and write to pipes in python, assuming you are talking about /std(out|in|err)/.

   from sys import stdin, stdout, stderr
   for n,line in enumerate(stdin):
      stdout.write(str(n) + '\n')
      stderr.write(line)
Is that the kind of reading and writing you are talking about?
Re:Why Awk for AI? (1997) by jjoonathan on Friday May 17, 2013 @02:55PM (#5726086)
I suspect it has more to do with stringing together the inputs/outputs of multiple programs, which is outright painful in python. The syntax is terribly verbose (import subprocess, use subprocess.call, refer to a few enum values in the subprocess package, wire them together which requires a line of code rather than a character). Worse, pipes are tricky to implement correctly. It's easy to deadlock the python process by overflowing a pipe's buffer (wait for exit to read the buffer, buffer fills before subprocess exit, deadlock), so people who don't have an intimate knowledge of the underlying implementation will find themselves stuck with a bug they can't fix.

Compare this to a shell where the combination of &&, ||, and | can fit on a single line what would have taken dozens of lines of python.

Re:Why Awk for AI? (1997) by sprayk on Friday May 17, 2013 @03:35PM (#5726302)
Ah, so using python for stuff bash is usually used for. I thought OP meant writing python scripts to be used as components of bash scripts.

For _that_ type of python, I usually use the sh module[0]. It handles piping pretty well, especially if you use StringIO to deal with the pipes. Unfortunately, return code-based conditionals are nowhere as simple as || and && are in shell.

[0] http://amoffat.github.io/sh/

Re:Why Awk for AI? (1997) by abraininavat on Friday May 17, 2013 @03:14PM (#5726205)
See https://github.com/kennethreitz/envoy

Example:

  >> r = envoy.run('uptime | pbcopy')