perl -ne "print $_;"
perl -pe "s/foo/bar/"
Slightly more info available in `man perlrun` or https://perldoc.perl.org/perlrun.html
perl -plane 'my $script'
+ (p)rints every processed line back out
+ deals with (l)ine endings, in and out
+ (a)utosplits every line into @F
One small test (with a big file) I executed took 1 minute with Perl and 20 minutes with awk.
And then there are those really complicated formats where awk is just not flexible enough.
Awk is really useful, but it doesn't cover the same problem set as Perl does.
Around 10 years ago I rewrote markdown.pl in awk and it was almost 20 times faster. The speedup came from both a much faster startup time and a much faster (albeit simpler) regexp implementation.
I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them.
Is this a certain kind of engineering job that requires searching through text files so often and requiring specialized tools? I've managed my whole career with ctrl-f, and highlight-all matches.
I think it depends a lot on the sort of software one works with. A main use of awk and other unix tools for me is as-hoc data munging, looking through production logs, taking bits of data from different sources and comparing them. If you make eg gui applications and sell them or work as a contractor developing in-house applications for clients, you probably don’t see a lot of production logs like that or process them that way. Any data that you expect to process ought to go into a database and then you can use sql (where joins work much better than the unix join command and you can use the actual structure of the data instead of trying to tease it out with the smallest simplest code you can think of). You might be using a debugger or backtracks or reproduction in test to deal with/investigate production issues rather than starting with logs. On the other hand, many other companies will have big production systems that run on lots of different virtual machines and produce lots of logs and have issues that are hard to reproduce (or maybe the logs are easier), and have lots of data spread about random places such that it may be easier to do the hacky thing for one-off cases or to prototype or whatever rather than Doing things more thoroughly or properly. And this set up will also lead to tools that are designed to fit into the rest of the unix ETL tack by the way they output or input data.
On the second point, I think that even as a die-hard emacs user I wouldn’t recommend that anyone try to use emacs as an ide for something like java (or maybe c++). I’d probably try to use emacs a myself but I think my java-editing experience would be worse than for those people who do it with an IDE. More realistically, I’d try to avoid writing java if at all possible.
I mention DevOps because it is hard to think of a situation where you’re just coding Java and C# all day that truly requires awk-or-similar. But just the other day I had a situation where I copied the contents of a CD-ROM to a web server — drag and drop on my Mac — and ended up with the files all in upper case. The program I was trying to run was failing because it requested everything with lower-case URLs. There were hundreds and hundreds of files in a bunch of directories, so I fixed it with Perl in about 30 seconds.
I’m curious if that’s the sort of problem you never encounter, or if it is, what you reach for.
I've also used awk to get button presses from an input stream on a MIDI controller. For me, I found that the up front cost of learning a few awk commands quickly made it's return on investment.
I'm not proud to admit it, but I've used awk in a couple places where I should have properly used `expect' instead. Except, I don't know expect well and haven't gotten to the point where it was worth the investment in learning it.
Imagine you have to do this for 100 files.
I've used perl and awk extensively for all sorts of things. Log parsing embedded device logs to generate reports was a big one for me, that's where I really learned both languages. And they are superb tools for that task. We also used awk quite a bit for configuration parsing as a sort of intermediary between different processes and scripts on those devices.
As I moved into web development, I've found myself using both much less. I haven't touched perl more than once in the last 6 years -- I was given a script to maintain recently but it only needed an hour or two of my time to add a small feature to. That perl script is part of a legacy build system. I still use awk every couple of months but it's just part of some pipeline on the command line.
I could set up a database, write multiple layers of code ... but really?
ETL and anything dealing with importing large data sets come to mind.
To answer the serious question, it's often enough for a first order solution to any problem where you have line and field separated text. Pulling instances of "something weird" out of log files, for instance, is a great use for awk, especially if the fingerprint of "something weird" is scattered through fields in a line in a way that makes grep cumbersome. Or if it spans a couple of lines, especially if you're dealing with a log from a multi-threaded app where there might be irrelevant lines interleaved with the ones you care about.
I haven't written much production-grade awk, but I've often used it as a tool to understand a problem well enough to write a production-grade solution to a problem or fix bugs in application code.
I say it's useful on servers because log files, dump files, or whatever text format servers end up dumping to disk will tend to grow large, and you'll have many of them per server. If you ever get into the situation where you have to analyze gigabytes of files from 50 different servers without tools like Splunk or its equivalents, it would feel fairly bad to have to download all these files locally to then drive some forensics on them.
This personally happens to me when some Erlang nodes tend to die and leave a crash dump of 700MB to 4GB behind, or on smaller individual servers (say a VPS) where I need to quickly go through logs, looking for a common pattern.
Awk/sed and the ETL stack that comes with every Unix-like OS (aka od, tr, cut, sort) and all that are superb data wrangling tools. Log file parsing, data cleaning, even actual data science at scale can be done with these tools. They're extremely efficient, as they're designed from an era when pretty much all interesting data was comparable to or much larger than memory size. As such, you can do a lot of stuff with them that most people don't imagine is even possible. FWIIW for high end data scientists; I don't consider knowledge of these tools to be optional at all. Anyone who hasn't used them in their career hasn't worked on serious problems, and is probably the kind of educated idiot who will suggest you do a job in a giant Hadoop cluster that you could easily do on one machine.
Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes with an ecosystem that is much larger still.
I know I can use perl for what I use awk for, and when I've raised this point before, people have been quick to explain how to process input by lines, and conditionally do something. For basic stuff, the fact that that's basically all awk does[0] means there's so many fewer ways to do it wrong. I can't say the same of perl, even when I was more familiar with it.
caveat: awk, nawk, mawk, and gawk don't necessarily share the same set of corner cases, and you may not get error messages that make sense to you when you bump into one with an unfamiliar awk.
[0] I know, not really, and especially for gawk. I've written awk scripts a couple hundred lines in the past. It's true enough for the 20 minute version though.
Edited for clarity.