$ awk '{print}/baz/ {sub("baz", "elephant"); print}' jvns.txt
  foo:
    - bar
    - baz
    - elephant
    - bananas

  $awk '
  {print}
  /baz/ {
      sub ("baz", "elephant")
      print
  }
  '

There is superb book, The AWK Programming Language, which teaches a lot about programming in addition to the awk language. Good discussion and link to pdf here: [0]

[0] https://news.ycombinator.com/item?id=13451454

https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...

Hmm, I did not know that (I have a print copy of that book).

> I wouldn't post a link to a pirated anything.

OK. But just in case, someone still wants a physical book, they can get it used from Amazon (can talk about Amazon in US) for ~$3.

Fast, but partly because it's not Unicode-aware: it treats strings as 8-bit character sequences rather than UTF-8. Often that's fine, if non-ASCII characters are only passed through unmodified, but requires some care to avoid problems.

    $ echo $LANG
    en_GB.UTF-8

    $ echo "ÜNICÖDE" | gawk '{print tolower($0)}'
    ünicöde

    $ echo "ÜNICÖDE" | mawk '{print tolower($0)}'
    �nic�de

I'd love to see the mawk compilation technology merged to GNU awk. Or mawk updated with Unicode support and a few of the GNU extensions.

The other item on my awk-like wishlist is CSV support, ie split $1 .. $N by CSV rules instead of just a field separator. I usually end up copying CSVs into postgresql because it is fast and then I can process it very flexibly, but it's a bit heavy for things that could be one line-ers. Also, postgresql won't load malformed CSV, but I suspect awk could be less picky.

There is miller which hits some of these points, but I haven't really grokked it yet and the syntax seems ... awkward.

I have been reading through it and running code snippets from there as I find time. It's awesome!

If anyone's interested, I also update my notes from there on my blog.. https://scripter.co/notes/awk. I plan to have that notes post contain the entire book when I am done.

ed script is one of the formats supported by diff and patch (with their -e or --ed command line switch). (diff generates the script by itself, but patch just pipes it to ed.)

ed scripts are used at Apple to maintain their patches for Python: see e.g. https://opensource.apple.com/source/python/python-97.50.7/2.... and others in https://opensource.apple.com/source/python/python-97.50.7/2....

https://rachelbythebay.com/w/2018/04/05/bangpatch/

This was used for the https://holeybeep.ninja/ April fool joke release.

  go get github.com/orivej/unix/regrep
  regrep s '(\n( *-) baz\n)' $'$1$2 elephant\n' < input.yaml

  go get github.com/orivej/unix/inplace
  find . -name '*.yaml' -exec inplace {} regrep s '(\n( *-) baz\n)' $'$1$2 elephant\n' \;

[0] http://doc.cat-v.org/bell_labs/structural_regexps/

    :args file1.txt file2.txt file3.txt
    :set autowrite
    :argdo norm /- baz/<CR>yypwCelephants

This example searches each HTML file in a directory for a line with a string and then deletes a number of lines:

    $ echo "g/search string/ .,+20 d\nx" >> exscript
    $ for f in *.html
      do
          ex - $f < exscript
      done

   for f in *.html; do
       echo -e "g/search string/ .,+20 d\nx" | ex - "$f"
    done

(I also added quotes to the $f dereference in case of file names containing white space, and the -e flag to echo to expand \n to newline. In case of a Bourne shell without support for -e in echo, I would probably use “{ echo "g/..."; echo "x"; } | ex - ...” instead of using \n.)

Also, in a production script I would probably have used “find . -maxdepth 1 -name "＊.html" -print0 | xargs --null --no-run-if-empty | while read f; do ...; done” instead of a ‘for’ loop from a pathname expansion, in order to guard against there being no html files, in which case a ‘for’ loop from a pathname expansion otherwise would be passing the literal string “＊.html” as the file name argument to ‘ex’.

    diff -e file1 file2 > ed_script

After spending 5 minutes with the manual, I realized it was the best line editor I've ever used.

ed is very similar, but it doesn't come with a nice manual. The info page is chaotic and doesn't start from the beginning. You can cover typical usage in 5 lines, but no, you have to read through 25 pages of stuff just to figure out a sensible command. I just use vim or nano.

In many case it indeed doesn't. Historically, that documentation void was meant to be filled by learn tutorials[1] and a written introduction in volume 2A of the manual[2,3]. Unfortunately, the learn tutorials can't really be made use of in a modern environment; you’d actually have to set up a PDP-11 emulator with V7 (though prebuilt images exist) and work with that, an environment where backspace doesn’t really work out of the box.

OpenBSD ed(1)[4] tries, but it's just not quite there as an introduction.

[1] http://a.papnet.eu/UNIX/v7/files/doc/07_learn.pdf

[2] http://a.papnet.eu/UNIX/v7/files/doc/04_edtut.pdf

[3] http://a.papnet.eu/UNIX/v7/files/doc/05_adved.pdf

[4] https://man.openbsd.org/ed.1

  sed '/baz/{s/.*/&\n&/;s/baz/elephant/2}' input.txt

  sed '/baz/ {
           s/.*/&\n&/
           s/baz/elephant/2
       }' input.txt

This being said, I went ahead and bought the book mentioned in the article [0] - a neat little read.

[0]: https://www.michaelwlucas.com/tools/ed

How to make the one-liner portable without using a sed commands file?

Maybe something like:

  sed 's/baz/elephant/;/^ \{2\}- elephant/{h;G;};/^ \{4\}- elephant/{h;G;};s/elephant/baz/' foo|sed -a wfoo

  1. s/baz/elephant/ 
  2. duplicate that line if two or four space indent
  3. s/elephant/baz/
  4. save

cf. jvns.ca blog:

  1. search for baz
  2. copy that line and paste it
  3. s/baz/elephant/
  4. save and quit

You could also just add the text after the matching line. A little simpler and more straight forward.

    $ cat > /tmp/ed-script
    /baz
    a
      - elephants
    .
    w
    q
    $ cat /tmp/2
    foo:
      - bar
      - baz
      - bananas
    $ cat /tmp/ed-script | ed /tmp/2
    33
      - baz
    47
    $ cat /tmp/2
    foo:
      - bar
      - baz
      - elephants
      - bananas
    $·

> I had one extra weird requirement which was that some of the lines were indented with 2 spaces, and some with 4 spaces. The - elephant line needed to have the same indentation as the previous line.

Maybe there's an old copy of "Unix Power Tools" over in your server room or an abandoned cubicle in the office... the content has not changed much in the ensuing decades!

   echo -e '/-baz\n+1\ni\n-elephant\n.\nw\nq\n'|ed foo

for speed, put $TMPDIR on memory file system

sed requires no temp file

   1.sed:
   /-baz/a\
   -elephant
   
   
   sed -f 1.sed foo|sed -a wfoo

   1.sed:
   s/- baz/- elephant/;
   /^  - elephant/{h;G;}
   /^    - elephant/{h;G;}
   s/- elephant/- baz/;

   sed -f 1.sed foo|sed -a wfoo

   1.sed:
   /^  - baz/a\
     - elephant
   

   /^    - baz/a\
       - elephant
   

   sed -f 1.sed foo|sed -a wfoo

  sed s/baz/baz\\nelephants/

  sed -i.bak s/baz/"baz\\n  - elephants"/ *.txt

Let's say I have:

  key   value
  salt  pepper
  fish  chips
  vodka orange
  rum   cola

<F3> to start recording a macro. Alt →, → to position the cursor at "v" (or just →→→→→→ if this is a fixed width column), then Alt U to uppercase the next word. → to move the cursor one forward, to the start of the next line. <F4> to finish recording the macro.

Then press <F4> five times to run the macro five times.

(Explanation intended for users who've never used Emacs before. Of course, there are optimizations.)

• qq to start recording a macro in register q,

• w to jump to the second word,

• gUaw to "go uppercase a word",

• j to move to next line (↓ works as well),

• q to stop recording,

• 5@q to apply macro in register q five times.

But in this example I would have probably used ex command:

    :%normal wgUaw

I'm genuinely interested in someone showing how to do this kind of trasformation in popular modern editors such as Atom and VSCode. Is there such a flexible way as in the classic editors?

    <F3> M-f M-u C-n C-a C-0 <F4>

  1.spt:

  ; x = "  - baz" 
  ; y = "  - elephant"
  ;a a = input :f(end)
  ;  output = a
  ;  a ? x :s(d)f(a) 
  ;d output = y
  ; :(a)
  ;end

  spitbol 1.spt < foo

Depends on the awk implementation and the task. However even gnu awk (gawk) is very fast and mawk is astonishing.

Here is a simple example: count the lines, words, and characters in a 65MB text file (10 copies of a novel stuck together).

Testing on Ubuntu GNU/linux 16.10 reporting middle of three tries:

  export LANG=ASCII    # avoid differences due to unicode
  $ time -p wc big10.txt 
   1284570 10956950 64886660 big10.txt
  real 0.29
  user 0.28
  sys 0.01
  $ time -p gawk '{l+=1; w+=NF; c+=length($0)+1} END {print l, w, c}' big10.txt
   1284570 10956950 64886660
  real 0.55
  user 0.53
  sys 0.01

  $ time -p mawk '{l+=1; w+=NF; c+=length($0)+1} END {print l, w, c}' big10.txt
   1284570 10956950 64886660
  real 0.35
  user 0.33
  sys 0.01

Just for a check, even python is not terrible at this:

  #!/usr/bin/python
  import sys
  l, w, c = 0, 0, 0
  for line in file(sys.argv[1], "rb"):
      l += 1
      w += len(line.split())
      c += len(line)
  print l, w, c
  
  $ time -p ./wc.py big10.txt 
  1284570 10956950 64886660
  real 0.87
  user 0.86
  sys 0.01

  time -p wc big10.txt
  1284570 10956950 64886660 big10.txt

  real         2.76
  user         2.68
  sys          0.08

Because novice, 2 out of 3 counts are incorrect and probably not the fastest solution used.

Total "words" in the example was simply AWK's NF. But looking at big10.txt there anomalies such as words separated by "--" instead of space.

Here I used non-space character followed by space. Far from accurate but not too far.

  1.k: 
  w:0:"big10.txt";v:,/$w
  m:v _ss "[^ ] " / "word": char followed by space
  #w   / lines
  1+#m / words
  #v   / characters

  time -p k 1

  1284570
  10019630
  63602090

  real         2.70
  user         2.40
  sys          0.28

  time -p wc -l big10.txt
  1284570 big10.txt

  real         0.13
  user         0.06
  sys          0.07

  sed -n '$!d;=' big10.txt
  1284570

  real         0.29
  user         0.19
  sys          0.09

  $ time -p sed -n '$!d;=' big10.txt
  1284570
  real 0.07
  user 0.06
  sys 0.00

  time -p mawk 'END {print NR}' big10.txt
  1284570
  real 0.04
  user 0.03
  sys 0.00

  $ time -p gawk 'END {print NR}' big10.txt
  1284570
  real 0.14
  user 0.13
  sys 0.00

  $ time -p wc -l big10.txt
  1284570 big10.txt
  real 0.02
  user 0.02
  sys 0.00

  w:0:"big10.txt";v:{" ",x}'w;u:{#v[x] _ss " [^ ]"}'!#v;t:{#w[x]}'!#w

  #w / lines
  +/u / words
  +/t / chars

But it appears the script is now faster than wc.

  time -p wc big10.txt

  1284570 10956950 64886660 big10.txt
  real         2.78
  user         2.66
  sys          0.12

  time -p k 1

  1284570
  10956830
  63602090

  real         2.57
  user         2.42
  sys          0.14

[0] http://www.norvig.com/big.txt

As with k, I am lacking in spitbol experience and so the counts are not identical to wc. Also I am using 10MB of big10.txt instead of the entire file.

  1.spt:

  ;* m line count, c word count, o char count 
  ;* p word pattern

  ; n = "0123456789"
  ; w = n &ucase &lcase "-"
  ; p = break(w) span(w)
  ;a a = input :f(c)
  ;  o = o + size(a) 
  ;  m = m + 1
  ;b a ? p = :f(a)
  ;  c = c + 1 :(b)
  ;c output = m ' ' c ' ' o
  ;end

  dd if=big10.txt bs=5m count=2 of=10m.txt

  time -p wc 10m.txt

  201346 1763181 10485760 10m.txt
  real         0.51
  user         0.44
  sys          0.00

  time -p spitbol 1.spt < 10m.txt

  201347 1770831 10235302
  real         0.33
  user         0.31
  sys          0.01

    $ ls -l /bin/ex
    lrwxrwxrwx 1 root root 2 Oct  2  2017 /bin/ex -> vi

(Bill Joy went on to become a co-founder of Sun Microsystems.)

"ex" stood for the extended version of ed.

[1] https://en.wikipedia.org/wiki/Vi

I loved ex's open mode. It's kind of a line-mode version of vi. Great to use over 1200 baud. Looks like vim doesn't have it. Sad.

    1.k:

    /k3
    v:"- baz";u:"  - elephant";t:"foo";s:"\n"
    w:_ssr[,/$0:t;v;v,u]
    w[1_ {x-2}'(&w="-")]:s
    t 0:_ssr[w;s;s," "],s

    k 1