News for hackers. Stuff that matters.
Re:Learn just a little Awk (2010) by j0e1 on Friday June 15, 2018 @08:02PM (#17324825)
Sure. I'll explain the code and hopefully you'd find enough clues to adapt it for your use-case which, from what you've described, is a subset.

Intent: Given a folder with multiple text files of academic papers from PubMed, remove the contents after the '====Refs' header. Then in each file find occurrences of any term in a keywords file and extract it along with 400 chars before and after it. Print this output as a tsv.

input: $1) A text file containing keywords of interest that need to be searched in the files separated by `|`.

$2) File containing paragraphs of lots of text with headers starting with '===='. There isn't a formatting requirement really. Just that, in my case the files happened to be so.

    # Note: Awk string commands have indexes that start with 1 (not 0).
    # The first Awk command just removes the newlines in file $2 and replaces them 
    # with space. `pat` contains the contents of the file $1: 'keyword1|kewword2|...'.
    # --re-interval allows use of, well, regex intervals.
    awk 'BEGIN { ORS=" "} { print $0 }' $2 | awk --re-interval -v pat="$1" '
	{
            # both variables are initialized with full-content of text file 
            # with the part after '==== Refs' removed.
            cut_content = substr($0, 1, match($0, "==== Refs"))
            orig_content = substr($0, 1, match($0, "==== Refs"))
            
            # index (could be better named as offset, but life is too short to refactor).
            idx = 0
            
            # match the keywords, even if it appears between non-alphabet chars.
            # e.g -keyword1. is a match.
            regex = "(^|[^a-zA-Z]{1})" pat "([^a-zA-Z]{1}|$)"
            
            # loop until matches are found.
            while (match(cut_content, regex)) {
              
              # RLENGTH is the length of the matched keyword.
              if (RLENGTH > 0) {
                 
                 # set the matched keyword to the variable `prot`. 
                 # RSTART is the character position of the matched keyword in the file. 
                 # This is the first occurrence of it in the file when read from the start.
                 prot = substr(cut_content, RSTART-1, RLENGTH+1)

                 # clean-up punctuation and/or space from the beginning and end of `prot`.
                 gsub(/[:punct: ]$/, "", prot)
                 gsub(/^[:punct: ]/, "", prot)

                 # print the `prot` and 400 chars before and after it using idx
                 # as an offset. Separate the results by tab char.
                 print prot, "\t", substr(orig_content, idx+RSTART-400, RLENGTH+800)

                 # update offset to character position right after the current matched keyword
                 idx += (RSTART + RLENGTH -1)

                 # cut-out all the text up until the position of the 
                 # last matched keyword (including it) and on the remaining text
                 # repeat the above steps in the loop. This is so that the search
                 # continues throughout the file and doesn't keep finding the same
                 # keyword over and over again. See `while` condition.
                 cut_content = substr(cut_content, RSTART+RLENGTH)
              }
	    }
        }'

Edit: Formatting. HN doesn't wrap text in code-block.

There are no comments on this comment yet.