The real issue is why do we silo ourselves so? It's fucking stupid. I do so much more than Ruby but conveying that to the new breed of tech people is just impossible. All they see is your biggest resume badge, and all they care about is how wide you will spread your legs for them.

short-descriptions and tons of examples.. I had fun and learned a lot doing this

link: https://github.com/learnbyexample/Command-line-text-processi...

Find (take a look at -exec option)

Cut (or awk)

Sed (for simple text/string substitutions)

Xargs

Dd (that beast can do charset transtation from ASCII to EBCDIC to use in mainframes and can also wipe disks)

And bash of course :)

    -P max-procs
    Run up to max-procs processes at a time; the default is 1.
    If max-procs is 0, xargs will run as many processes as
    possible at a time.  Use the -n option with -P; otherwise
    chances are that only one exec will be done.

    -n max-args
    Use at most max-args arguments per command line.
    Fewer than max-args arguments will be used if the size
    (see the -s option) is exceeded, unless the -x option is
    given, in which case xargs will exit.

    find . -name "*.log" | sort | xargs -n 1 -P 8 agg_stats.py | sort | merge_periods.py

gotta SSH into 100 machines and ask them all a simple question? xargs will trivially speed that up by at least 10x, if not better, just by parallelizing the SSH handshakes.

https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_w...

"awk"

brb 15 years

Not a program, but another fun bash thing I've learned about recently is brace expansion: https://www.linuxjournal.com/content/bash-brace-expansion

I've got some similar ones for work that take data csv like data and output sql. A little bit of vim-foo on the csv (yy10000p) and I've got all the test data I need.

[1] https://github.com/TheMozg/awk-raycaster

e.g. testing in CI a JSON file is properly formatted:

  jq . < $f | diff -u $f -

[0]: https://stedolan.github.io/jq/manual/

[1]: https://github.com/antonmedv/fx

https://github.com/nemild/cli_for_full_stack

The problem tends to come along with the visibility that comes when one of these get released publicly. People use these things without doing the same level of due diligence to see if something less complicated fits their needs. Either deliberately (resume driven development) or simply because it doesn't occur to them to do so and "if it's good enough for this unicorn it's good enough for me". Or in some cases they do due diligence, but incorrectly evaluate their needs and rule out simpler solutions.

But the most common reason I've ran into is simply that it's more fun to play with these projects than it is to use crufty old stuff, no matter how tried and tested and potentially appropriate it is.

At a previous company, I had a python script running on a cron job every 5 minutes to do some data processing needs. Once or twice a month, the batch would be so large that it took 6-7 minutes to complete and the cron job would trigger the script again before it finished, causing the second instance of the script to see the lock file, log an error, and exit. It didn't cause any problem for these periodically skipped runs, because the business need only required data to be processed within 24 hours of coming in. The 5 minute cron job was just to even out resource usage throughout the day instead of doing a larger nightly batch job. A piece of data not getting processed for 8 minutes instead of 5 did not make any material impact.

Another team had noticed the errors popping up in the log and were in the process of testing out a whole bunch of real time data pipelines like Kafka, leveraging my error messages to justify the need for a "real time" system without ever even asking me about the errors. After I found out other people were noticing those superfluous errors, I moved the cron job to a 30 minute window to stop them from happening anymore. Turns out they didn't have any other justification for their data pipeline greenfield project and weren't happy to go back to their normal day to day work. I offered to let them maintain and expand my python scripts if they were interested in data processing work, but for some reason they never took me up on that offer. :(

I think this is a dangerous, irritating mythology that programmers permit to their detriment. Skipping work entirely to go play pool or watch a movie is "fun". Evaluating a new technology stack to see if it fits business needs (present or future) might be _intellectually stimulating_, but deriding it as "fun" - and allowing management to write it off as time-wasting - hurts everybody. This isn't "fun", it's research, just the same as particle physics experiments are, and it's a big part of what we went to college to learn to do effectively.

Right, Yahoo made Hadoop because (probably) they needed it. 99% of companies... just don’t.

"This is an absolute classic which although ancient in Computing years is an absolute gem full of relevance yet."

I rewrote a Hadoop job to use Streaming (pipes, basic unix model) from a Java job to native. Just that switch was 10x faster. Mostly because of Hadoop overheads.

If you want to find out what tools are available on the CLI that operate on jpegs, you'd try 'apropos jpeg' and get a list of things that mention jpeg in their man page.