Re:How to Clean Text Data at the Command Line
by dredmorbius on Monday October 19, 2020 @02:17PM
(#24829171)
I don't have your scale problems. You might care to not impose them where they don't exist.
Re:How to Clean Text Data at the Command Line
by hnlmorg on Tuesday October 20, 2020 @06:03AM
(#24835631)
Don't be so ridiculous. You're the one imposing arbitrary problems by saying "everything should be written in awk" then writing an example that was an order of magnitude longer in both character count and execution time than the examples that you were suggesting was wrong.
All I'm doing is citing a few reasons why someone might prefer a terser pipeline but I hadn't realised this wasn't supposed to be an objective conversation and since I have no interest in engaging in pointless language fanboyism I'm just going to leave you to it.
Re:How to Clean Text Data at the Command Line
by dredmorbius on Tuesday October 20, 2020 @09:39AM
(#24837134)
"everything should be written in awk" is a rather dubious translation of:
a quick translation to straight awk gives cleaner code, more power, easier conversion to a script, and access to a small library of awk-based utilities I've acumulated.
https://news.ycombinator.com/item?id=24822697
You're not discussing what I've actually written.
Good day.
Re:How to Clean Text Data at the Command Line
by hnlmorg on Thursday October 22, 2020 @06:35PM
(#24864010)
I addressed those points directly, which you ignored when you blamed me of arguing use cases specific only to me. Suffice to say your complaint was hypocritical given your statement previous to that was very specific to your use case (as I also pointed out in my aforementioned, but ignored, comment)
500 records isn't a large dataset. Not even close. Large would be orders of millions to billions. And yes, I have had to munge datasets that large on many occasions.
> But a quick translation to straight awk gives cleaner code, more power, easier conversion to a script, and access to a small library of awk-based utilities I've acumulated.
- cleaner: only if you find awk readable. Plenty of people don't. Plenty of people find Go or Python more readable.
- more powerful: again depends. You wouldn't have multithreading builtin like pipelines would. And Python and Go are undoubtedly more powerful than awk. I'm not knocking awk here, just being pragmatic.
- easer conversion to a script: at which point you might as well skip awk entirely and jump straight to Go or Python (or any other programming language)
- and access to a small library of awk-based utilities I've accumulated: that only benefits you. If you're having to sell the benefits of awk to someone then odd are they don't have that small library already to hand ;)
> We've spent far more time discussing this than coding, let alone running, it.
Some of the larger datasets I've had to process have definitely taken longer to munge than my reply here has taken to type :)
Disclaimer: I've honestly not got a problem with awk, I used to use it heavily 20 years ago. But these days its value is diminishing and a lot of the awk evangelists seem to miss the point of why awk isn't well represented in blog posts any more. It's both more verbose than pipelining to coreutils and less powerful than a programming language -- it's that weird middle ground that doesn't provide much value to most people aside those who are already invested into the awk language. Some might see that as a loss but personally I see that as demonstrating the strength of all the other tools we have at our disposal these days.