- Strings are subtly complex, but strings are not variables. You can assign a string, and later handle it as a variable, and not deal with any of the specifics of string-iness. Likewise, you can take a variable, and later treat it as a string (for loosely or not-typed variables).
- Magic switches are not magic, they are options. Virtually every program takes options. Sometimes they impact a lot of things, sometimes a little. Only the context determines how much is "too much".
- Terse/cryptic variables allow you to write complex expressions in a compact form. This allows you to read more in a small space, making it easier to reason about or form complex expressions. Human languages are flush with these, as is mathematics. But you have to balance the terse, cryptic and magical with guilelessness, or it becomes a mess.
- Terse and magical syntax is, again, a feature, not a bug. Using magical syntax I can do in a few characters what would take me many lines with a traditional language, and as we all know, increased number of lines correlates to bugs, in addition to simply making it harder to grok.
- Types aren't ignored, but they may be very loosely enforced. If you want to write a quick program to get something done, typing is a curse. If you want to write a very thorough program, typing is a blessing. In many cases, loosely or untyped programs actually work better than their typed cousins, because they allow for more unexpected behaviors without failing. Failing early and often may be a modern trend, but... it literally means things fail more, and this is often not desirable.
Caveats:
- Programs that are hard to read do indeed suck, and it takes lots of experience to make some kinds of programs easier to read. But that's not an indictment of the program, it's an indictment of the person who wrote it. We don't indict English when somebody writes a document that's impossible to comprehend.
- Interestingly, some of the more popular languages are the worst to debug. Perl is probably one of the easiest languages to debug, not inconsequently because of how good the interpreter is at suggesting to the user what the actual problem was and almost exactly how to fix it.
There is a lot of wisdom in the things you avoid, however I would ask one question, "How often do you use it?"
For me, the best systems are those that can be wordy and prescriptive but as you get to know them you can use more short hand so they "get out of the way" as it were. A good example of that philosophy is keyboard short cuts. When I'm learning a program I'm happy to pause and sling the mouse around to find the thing I need in the labeled menu stack with an appropriate name which also tells me what the keyboard short cut is for that thing. Then as I get better I can just use the short cut and my workflow gets faster. Once I've internalized the keymap my flow is held up by how fast I can think, not by how fast I can take my hand off the keyboard, move the mouse, click and then put it back on the keyboard.
Awk is one of those things that once you internalize what it can do, you can use it for a lot of stuff, and you can do it quickly.
And if it's something you don't know and are experimenting with, you can also open ipython and play around with the data (kinda like he's doing throughout the article) and keep your variables between each run without having to write intermediate values to disk or keep piping over and over.
Let alone the huge stdlib and 3rd party lib you have access to if needed.
col1,col2,col3
1,2,3
4,"hello, \"world\"",6
"7 buckets",,9
To get the usual awk experience with this very common file format, exactly the type of thing you want to parse with awk, you first need to install gawk, then use a big FPAT regex that needs to be adjusted for any new CSV variant.
I would love to see awk with "CSV mode", where it intelligently handles formats like this if you just pass a flag. I think awk would do well to differentiate itself with excellent 2d dataset parsing functionality, but at least catchup up to the average scripting language would be great.
I'm half expecting someone to say "just pass -csv it does what you want" and if so I'll be very excited.
https://news.ycombinator.com/item?id=28708145
...but if your files are CSV, there is a CSV extension for gawk
@include "csv"
BEGIN { CSVMODE = 1 }It's funny searches for awk CSV seem to yield a bunch of SO questions where the answers are increasingly cumbersome regexes instead of this extension.
Of course, you can't count of this extension being widely installed, but it's great for my own desktop.
awk -f ./ucsv.awk -e '{print $5}'
Also this> 4,"hello, \"world\"",6
Is incorrect per https://tools.ietf.org/html/rfc4180 so you should just fix it with a sed -i 's/\\"/""/g' and then just parse as normal.
I just end up using Python/Perl but I do have a soft spot for awk so it would be cool if good support was built-in.
XML is somewhere in the middle--I've seen some horrible abuses of CDATA sections way back when--but at least there are accepted ways to prove what's invalid.
In principle:
cat textfile.csv | csvquote | awk -f myprogram.awk | csvquote -u > output.csv
Also works for other text processing tools like cut, sed, sort, etc.Looking at my command history, I mostly use awk to extract a field like this:
<something> | awk '{print $3}'
(I know "cut" is supposed to do the same thing, but it was never reliable for me - maybe tabs/spaces?) $ cat /bin/awkmail
#!/bin/gawk -f
BEGIN { smtp="/inet/tcp/0/smtp.yourco.com/25";
ORS="\r\n"; r=ARGV[1]; s=ARGV[2]; sbj=ARGV[3]; # /bin/awkmail to from subj < in
print "helo " ENVIRON["HOSTNAME"] |& smtp;
smtp |& getline j; print j
print "mail from: " s |& smtp; smtp |& getline j; print j
if(match(r, ","))
{
split(r, z, ",")
for(y in z) { print "rcpt to: " z[y] |& smtp; smtp |& getline j; print j }
}
else { print "rcpt to: " r |& smtp; smtp |& getline j; print j }
print "data" |& smtp; smtp |& getline j; print j
print "From: " s |& smtp; ARGV[2] = "" # not a file
print "To: " r |& smtp; ARGV[1] = "" # not a file
if(length(sbj)) { print "Subject: " sbj |& smtp; ARGV[3] = "" } # not a file
print "" |& smtp
while(getline > 0) print |& smtp
print "." |& smtp; smtp |& getline j; print j
print "quit" |& smtp; smtp |& getline j; print j
close(smtp) } # /inet/protocol/local-port/remote-host/remote-portFor example: https://www.gnu.org/software/gawk/manual/gawkinet/html_node/...
for example:
#!/usr/bin/python
import smtplib
from email.mime.text import MIMEText
msg = 'hi'
subj='read this!'
smtp_server='mail.example.com'
smtp_from='[email protected]'
smtp_to='[email protected]'
m = MIMEText(msg)
m['To'] = smtp_to
m['From'] = smtp_from
m['Subject'] = subj
s = smtplib.SMTP(smtp_server)
s.sendmail(smtp_from, [smtp_to], m.as_string())
s.quit()
of course, you seem to think in gawk so if that works
for you that's what you should continue doing!by the way, I hacked this example from another script which attached a logfile:
with open(arg.logfile) as f:
log_contents = f.read()
m = MIMEText(log_contents)
you can also use: from email.mime.image import MIMEImage
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
and then: m = MIMEMultipart()
m.attach(MIMEText('\n\n%s\n\n'%xkcd_img_title))
m.attach(MIMEImage(xkcd_img))The AWK script doesn't need libraries, so it can actually be useful in places where you have awk but not Python.
Awk will treat it as having two columns (by default), while cut will treat each space as it’s own column.
Awk is also a little nicer for whitespace; cut makes specifying the delimiter (with say “-d\ “) a little more vexing.
But I sometimes use Awk scripts to quickly analyse a log file to get information you need or to generate a configuration file or something.
I think of the strict strongly-typed nature of a language like Java as equivalent to the formal language a lawyer might use in a legal document: There shouldn't be any place for ambiguity or misunderstanding.
Programming in Awk, on the other hand, is more like having an informal conversation with an old friend; those cryptic variable names is like the slang words that both of you understand and the magical syntax is like the dialect you share that may sound a bit strange to outsiders.
I think there is a place for both approaches in the broad space of computing.
- Avoid programming in strings (especially in Bash, where nested quotes are full of pitfalls)
- Avoid magic switches that change behavior (like -F)
- Avoid terse or cryptic variable names (like $NF)
- Avoid terse and magical syntax (sorry Perl, happy to leave you behind me)
- Avoid programs that are hard to read
- Avoid programs that are difficult to debug while writing them
- Avoid programs that ignore types
For these reasons, I prefer to avoid awk for anything except the most trivial of tasks. I think the prevalence of scripting languages and the speed of execution and debugging today has made awk not as necessary as it may have been in the 70s. And as to the first point, I'm aware you can write awk scripts in files, and I feel like if your script has gotten complex enough that you need a file, you're creating something unmaintainable and unreadable that would be better suited in a different programming language.
Edit: I should add this article is great and a good introduction to awk, regardless of my personal taste for the tool.