News for hackers. Stuff that matters.
Re:Awk in 20 Minutes (2015) by scottlocklin on Saturday May 02, 2020 @11:05AM (#23052154)
People (me anyway) used perl back in the dark ages of the 90s for the same sorts of things people use python for now. It was really the only option for open source general purpose interpreters for quite a while. People still use it in biotech I think, or were 5 years ago; the text processing capabilities are useful for genomic data.

Awk/sed and the ETL stack that comes with every Unix-like OS (aka od, tr, cut, sort) and all that are superb data wrangling tools. Log file parsing, data cleaning, even actual data science at scale can be done with these tools. They're extremely efficient, as they're designed from an era when pretty much all interesting data was comparable to or much larger than memory size. As such, you can do a lot of stuff with them that most people don't imagine is even possible. FWIIW for high end data scientists; I don't consider knowledge of these tools to be optional at all. Anyone who hasn't used them in their career hasn't worked on serious problems, and is probably the kind of educated idiot who will suggest you do a job in a giant Hadoop cluster that you could easily do on one machine.

There are no comments on this comment yet.