News for hackers. Stuff that matters.
Re:Learn just a little Awk (2010) by j0e1 on Friday June 15, 2018 @04:50PM (#17323447)
Here is an Awk program (not a one-liner) that extracted select snippets of text from lots of docs and worked almost 10x faster than a grep equivalent:

   awk 'BEGIN { ORS=" "} { print $0 }' $2 | awk --re-interval -v pat="$1" '
	{
            cut_content = substr($0, 1, match($0, "==== Refs"))
            orig_content = substr($0, 1, match($0, "==== Refs"))
            idx = 0
            regex = "(^|[^a-zA-Z]{1})" pat "([^a-zA-Z]{1}|$)"
            while (match(cut_content, regex)) {
              if (RLENGTH > 0) {
                 prot = substr(cut_content, RSTART-1, RLENGTH+1)
                 gsub(/[:punct: ]$/, "", prot)
                 gsub(/^[:punct: ]/, "", prot)

                 print prot, "\t", substr(orig_content, idx+RSTART-400, RLENGTH+800)
                 idx += (RSTART + RLENGTH -1)
                 cut_content = substr(cut_content, RSTART+RLENGTH)
              }
	    }
        }'
Re:Learn just a little Awk (2010) by AdmiralAsshat on Friday June 15, 2018 @05:25PM (#17323693)
Looks interesting; can you explain how the hell it works?

I tried writing a script that would extract entries from logs (e.g. if every entry started "====CRITICAL ERROR LOG 2018/06/15====", I wanted it to start there and grab every line below it until it hits the next log entry starting with the "=====" header), but I gave up. Your script might do something similar, if I can figure out what it's looking for.

Re:Learn just a little Awk (2010) by bcbrown on Friday June 15, 2018 @05:52PM (#17323929)
You can use regex ranges to match every line between two regexes: https://www.gnu.org/software/gawk/manual/html_node/Ranges.ht...

Another option might be to use multi-line records (https://www.gnu.org/software/gawk/manual/html_node/Multiple-...). First pre-process the file to add an empty line before each header, then use blank lines as the record separator and newlines for the field separator.

Re:Learn just a little Awk (2010) by bartbes on Saturday June 16, 2018 @07:25AM (#17326986)
Easily solved by just keeping some state:

    /^=====/ { inentry = !inentry; next }
    inentry
Possibly without 'next' if you want the marker lines too.
Re:Learn just a little Awk (2010) by j0e1 on Friday June 15, 2018 @08:02PM (#17324825)
Sure. I'll explain the code and hopefully you'd find enough clues to adapt it for your use-case which, from what you've described, is a subset.

Intent: Given a folder with multiple text files of academic papers from PubMed, remove the contents after the '====Refs' header. Then in each file find occurrences of any term in a keywords file and extract it along with 400 chars before and after it. Print this output as a tsv.

input: $1) A text file containing keywords of interest that need to be searched in the files separated by `|`.

$2) File containing paragraphs of lots of text with headers starting with '===='. There isn't a formatting requirement really. Just that, in my case the files happened to be so.

    # Note: Awk string commands have indexes that start with 1 (not 0).
    # The first Awk command just removes the newlines in file $2 and replaces them 
    # with space. `pat` contains the contents of the file $1: 'keyword1|kewword2|...'.
    # --re-interval allows use of, well, regex intervals.
    awk 'BEGIN { ORS=" "} { print $0 }' $2 | awk --re-interval -v pat="$1" '
	{
            # both variables are initialized with full-content of text file 
            # with the part after '==== Refs' removed.
            cut_content = substr($0, 1, match($0, "==== Refs"))
            orig_content = substr($0, 1, match($0, "==== Refs"))
            
            # index (could be better named as offset, but life is too short to refactor).
            idx = 0
            
            # match the keywords, even if it appears between non-alphabet chars.
            # e.g -keyword1. is a match.
            regex = "(^|[^a-zA-Z]{1})" pat "([^a-zA-Z]{1}|$)"
            
            # loop until matches are found.
            while (match(cut_content, regex)) {
              
              # RLENGTH is the length of the matched keyword.
              if (RLENGTH > 0) {
                 
                 # set the matched keyword to the variable `prot`. 
                 # RSTART is the character position of the matched keyword in the file. 
                 # This is the first occurrence of it in the file when read from the start.
                 prot = substr(cut_content, RSTART-1, RLENGTH+1)

                 # clean-up punctuation and/or space from the beginning and end of `prot`.
                 gsub(/[:punct: ]$/, "", prot)
                 gsub(/^[:punct: ]/, "", prot)

                 # print the `prot` and 400 chars before and after it using idx
                 # as an offset. Separate the results by tab char.
                 print prot, "\t", substr(orig_content, idx+RSTART-400, RLENGTH+800)

                 # update offset to character position right after the current matched keyword
                 idx += (RSTART + RLENGTH -1)

                 # cut-out all the text up until the position of the 
                 # last matched keyword (including it) and on the remaining text
                 # repeat the above steps in the loop. This is so that the search
                 # continues throughout the file and doesn't keep finding the same
                 # keyword over and over again. See `while` condition.
                 cut_content = substr(cut_content, RSTART+RLENGTH)
              }
	    }
        }'

Edit: Formatting. HN doesn't wrap text in code-block.