News for hackers. Stuff that matters.
Re:Understanding Awk by stevebmark on Thursday September 30, 2021 @01:11PM (#28708643)
There are things I've come to dislike and avoid when programming in general:

- Avoid programming in strings (especially in Bash, where nested quotes are full of pitfalls)

- Avoid magic switches that change behavior (like -F)

- Avoid terse or cryptic variable names (like $NF)

- Avoid terse and magical syntax (sorry Perl, happy to leave you behind me)

- Avoid programs that are hard to read

- Avoid programs that are difficult to debug while writing them

- Avoid programs that ignore types

For these reasons, I prefer to avoid awk for anything except the most trivial of tasks. I think the prevalence of scripting languages and the speed of execution and debugging today has made awk not as necessary as it may have been in the 70s. And as to the first point, I'm aware you can write awk scripts in files, and I feel like if your script has gotten complex enough that you need a file, you're creating something unmaintainable and unreadable that would be better suited in a different programming language.

Edit: I should add this article is great and a good introduction to awk, regardless of my personal taste for the tool.

Re:Understanding Awk by throwaway38941 on Thursday September 30, 2021 @03:01PM (#28709933)
I've been doing systems work for 20 years. Here's why most of those things are actually good:

- Strings are subtly complex, but strings are not variables. You can assign a string, and later handle it as a variable, and not deal with any of the specifics of string-iness. Likewise, you can take a variable, and later treat it as a string (for loosely or not-typed variables).

- Magic switches are not magic, they are options. Virtually every program takes options. Sometimes they impact a lot of things, sometimes a little. Only the context determines how much is "too much".

- Terse/cryptic variables allow you to write complex expressions in a compact form. This allows you to read more in a small space, making it easier to reason about or form complex expressions. Human languages are flush with these, as is mathematics. But you have to balance the terse, cryptic and magical with guilelessness, or it becomes a mess.

- Terse and magical syntax is, again, a feature, not a bug. Using magical syntax I can do in a few characters what would take me many lines with a traditional language, and as we all know, increased number of lines correlates to bugs, in addition to simply making it harder to grok.

- Types aren't ignored, but they may be very loosely enforced. If you want to write a quick program to get something done, typing is a curse. If you want to write a very thorough program, typing is a blessing. In many cases, loosely or untyped programs actually work better than their typed cousins, because they allow for more unexpected behaviors without failing. Failing early and often may be a modern trend, but... it literally means things fail more, and this is often not desirable.

Caveats:

- Programs that are hard to read do indeed suck, and it takes lots of experience to make some kinds of programs easier to read. But that's not an indictment of the program, it's an indictment of the person who wrote it. We don't indict English when somebody writes a document that's impossible to comprehend.

- Interestingly, some of the more popular languages are the worst to debug. Perl is probably one of the easiest languages to debug, not inconsequently because of how good the interpreter is at suggesting to the user what the actual problem was and almost exactly how to fix it.

Re:Understanding Awk by ChuckMcM on Thursday September 30, 2021 @01:52PM (#28709080)
I take it you LOVE ada :-)

There is a lot of wisdom in the things you avoid, however I would ask one question, "How often do you use it?"

For me, the best systems are those that can be wordy and prescriptive but as you get to know them you can use more short hand so they "get out of the way" as it were. A good example of that philosophy is keyboard short cuts. When I'm learning a program I'm happy to pause and sling the mouse around to find the thing I need in the labeled menu stack with an appropriate name which also tells me what the keyboard short cut is for that thing. Then as I get better I can just use the short cut and my workflow gets faster. Once I've internalized the keymap my flow is held up by how fast I can think, not by how fast I can take my hand off the keyboard, move the mouse, click and then put it back on the keyboard.

Awk is one of those things that once you internalize what it can do, you can use it for a lot of stuff, and you can do it quickly.

Re:Understanding Awk by ehsankia on Friday October 01, 2021 @02:06AM (#28715174)
Yep, anything more than 1 line would be a python script. The author vaguely tries to answer that but doesn't really give anything more satisfying than "you can rewrite it in python" and "why not?".

And if it's something you don't know and are experimenting with, you can also open ipython and play around with the data (kinda like he's doing throughout the article) and keep your variables between each run without having to write intermediate values to disk or keep piping over and over.

Let alone the huge stdlib and 3rd party lib you have access to if needed.

Re:Understanding Awk by jrumbut on Thursday September 30, 2021 @02:10PM (#28709248)
The thing that prevents awk from being a major part of my daily routine is that it (amazingly) has poor CSV support. Consider the following:

col1,col2,col3

1,2,3

4,"hello, \"world\"",6

"7 buckets",,9

To get the usual awk experience with this very common file format, exactly the type of thing you want to parse with awk, you first need to install gawk, then use a big FPAT regex that needs to be adjusted for any new CSV variant.

I would love to see awk with "CSV mode", where it intelligently handles formats like this if you just pass a flag. I think awk would do well to differentiate itself with excellent 2d dataset parsing functionality, but at least catchup up to the average scripting language would be great.

I'm half expecting someone to say "just pass -csv it does what you want" and if so I'll be very excited.

Re:Understanding Awk by nickcw on Thursday September 30, 2021 @02:18PM (#28709358)
There is an answer to CSV mode a bit further down the page

https://news.ycombinator.com/item?id=28708145

...but if your files are CSV, there is a CSV extension for gawk

    @include "csv"
    BEGIN { CSVMODE = 1 }
Re:Understanding Awk by jrumbut on Thursday September 30, 2021 @02:30PM (#28709513)
Well there you go, for the sake of my pride at least it's an extension.

It's funny searches for awk CSV seem to yield a bunch of SO questions where the answers are increasingly cumbersome regexes instead of this extension.

Of course, you can't count of this extension being widely installed, but it's great for my own desktop.

Re:Understanding Awk by nmz on Thursday September 30, 2021 @03:31PM (#28710321)
that's because the extension only works in gawk. its not portable anywhere else.
Re:Understanding Awk by nmz on Thursday September 30, 2021 @03:40PM (#28710421)
You can just use https://github.com/Nomarian/Awk-Batteries/blob/master/Units/... and use as so

    awk -f ./ucsv.awk -e '{print $5}'
Also this

> 4,"hello, \"world\"",6

Is incorrect per https://tools.ietf.org/html/rfc4180 so you should just fix it with a sed -i 's/\\"/""/g' and then just parse as normal.

https://github.com/Nomarian/Awk-Batteries/wiki/Formats

Re:Understanding Awk by sk5t on Thursday September 30, 2021 @02:24PM (#28709436)
'miller' and 'xsv' are pretty good tools for wrangling CSV. (And regexp is kind of a terrible tool for it, too many edge cases.)
Re:Understanding Awk by jrumbut on Thursday September 30, 2021 @02:28PM (#28709482)
Yeah, I don't want to have to write a CSV library each time, that's what I'm trying to get at.

I just end up using Python/Perl but I do have a soft spot for awk so it would be cool if good support was built-in.

Re:Understanding Awk by sk5t on Thursday September 30, 2021 @02:52PM (#28709777)
Who's writing a library? Just use xsv or miller to extract the bits you want from the CSV, change the delimiter or escapes to something more convenient, etc., then feed that to awk or other CSV-unaware text processors.
Re:Understanding Awk by jrumbut on Thursday September 30, 2021 @03:01PM (#28709932)
I was agreeing with your point about regexes, that it's good to avoid trying to deal with all the corner cases yourself when you're just trying to write a small script.
Re:Understanding Awk by sk5t on Thursday September 30, 2021 @05:27PM (#28711517)
Ah, understood! CSV is funny, it seems like a more trivial thing than it really is, and its human readability sort of invites broken approaches in a way that something like Parquet would not.

XML is somewhere in the middle--I've seen some horrible abuses of CDATA sections way back when--but at least there are accepted ways to prove what's invalid.

Re:Understanding Awk by dbro on Friday October 01, 2021 @01:13PM (#28721064)
There is a small program I wrote called csvquote[1] that can be used to sanitize input to awk so it can rely on delimiter characters (commas) to always mean delimiters. The results from awk then get piped through the same program at the end to restore the commas inside the field values.

In principle:

  cat textfile.csv | csvquote | awk -f myprogram.awk | csvquote -u > output.csv
Also works for other text processing tools like cut, sed, sort, etc.

[1] https://github.com/dbro/csvquote

Re:Understanding Awk by m463 on Thursday September 30, 2021 @01:50PM (#28709063)
I use awk for one-liners, no more.

Looking at my command history, I mostly use awk to extract a field like this:

   <something> | awk '{print $3}'
(I know "cut" is supposed to do the same thing, but it was never reliable for me - maybe tabs/spaces?)
Re:Understanding Awk by chasil on Thursday September 30, 2021 @02:03PM (#28709174)
Here is a GAWK program of mine that implements outgoing SMTP. While not a one-liner, this is much shorter and less tedious than trying to do it in C.

    $ cat /bin/awkmail
    #!/bin/gawk -f

    BEGIN { smtp="/inet/tcp/0/smtp.yourco.com/25";
    ORS="\r\n"; r=ARGV[1]; s=ARGV[2]; sbj=ARGV[3]; # /bin/awkmail to from subj < in

    print "helo " ENVIRON["HOSTNAME"]        |& smtp;
    smtp |& getline j; print j
    print "mail from: " s                    |& smtp;  smtp |& getline j; print j
    if(match(r, ","))
    {
     split(r, z, ",")
     for(y in z) { print "rcpt to: " z[y]    |& smtp;  smtp |& getline j; print j }
    }
    else { print "rcpt to: " r               |& smtp;  smtp |& getline j; print j }
    print "data"                             |& smtp;  smtp |& getline j; print j

    print "From: " s                         |& smtp;  ARGV[2] = ""   # not a file
    print "To: " r                           |& smtp;  ARGV[1] = ""   # not a file
    if(length(sbj)) { print "Subject: " sbj  |& smtp;  ARGV[3] = "" } # not a file
    print ""                                 |& smtp

    while(getline > 0) print                 |& smtp

    print "."                                |& smtp;  smtp |& getline j; print j
    print "quit"                             |& smtp;  smtp |& getline j; print j

    close(smtp) } # /inet/protocol/local-port/remote-host/remote-port
Re:Understanding Awk by meltedcapacitor on Thursday September 30, 2021 @02:48PM (#28709723)
Cheap fix: the space after MAIL FROM: and RCPT TO: is not standard compliant.
Re:Understanding Awk by jrumbut on Thursday September 30, 2021 @02:17PM (#28709338)
That's a beautiful use of the language, it reminds me of some of the awk CGI efforts out there.

For example: https://www.gnu.org/software/gawk/manual/gawkinet/html_node/...

Re:Understanding Awk by m463 on Thursday September 30, 2021 @03:51PM (#28710526)
IMHO, that's too big for awk, why not python?

for example:

    #!/usr/bin/python
    import smtplib
    from email.mime.text import MIMEText

    msg = 'hi'
    subj='read this!'

    smtp_server='mail.example.com'
    smtp_from='[email protected]'
    smtp_to='[email protected]'

    m = MIMEText(msg)

    m['To'] = smtp_to
    m['From'] = smtp_from
    m['Subject'] = subj

    s = smtplib.SMTP(smtp_server)
    s.sendmail(smtp_from, [smtp_to], m.as_string())
    s.quit()

of course, you seem to think in gawk so if that works for you that's what you should continue doing!

by the way, I hacked this example from another script which attached a logfile:

    with open(arg.logfile) as f:
        log_contents = f.read()

    m = MIMEText(log_contents)
you can also use:

    from email.mime.image import MIMEImage
    from email.mime.text import MIMEText
    from email.mime.multipart import MIMEMultipart
and then:

    m = MIMEMultipart()
    m.attach(MIMEText('\n\n%s\n\n'%xkcd_img_title))
    m.attach(MIMEImage(xkcd_img))
Re:Understanding Awk by chousuke on Thursday September 30, 2021 @04:20PM (#28710859)
Your script doesn't even do the same thing. You are importing a library that implements SMTP, which is missing the point.

The AWK script doesn't need libraries, so it can actually be useful in places where you have awk but not Python.

Re:Understanding Awk by likpok on Thursday September 30, 2021 @02:22PM (#28709401)
Consider the input a b

Awk will treat it as having two columns (by default), while cut will treat each space as it’s own column.

Awk is also a little nicer for whitespace; cut makes specifying the delimiter (with say “-d\ “) a little more vexing.

Re:Understanding Awk by wernsey on Friday October 01, 2021 @06:58AM (#28716822)
In my day job I use Java, and all the items you list are good ideas to maintain my sanity.

But I sometimes use Awk scripts to quickly analyse a log file to get information you need or to generate a configuration file or something.

I think of the strict strongly-typed nature of a language like Java as equivalent to the formal language a lawyer might use in a legal document: There shouldn't be any place for ambiguity or misunderstanding.

Programming in Awk, on the other hand, is more like having an informal conversation with an old friend; those cryptic variable names is like the slang words that both of you understand and the magical syntax is like the dialect you share that may sound a bit strange to outsiders.

I think there is a place for both approaches in the broad space of computing.