Re:Learn to Process Text in Linux Using Grep, Sed, and Awk
by xfguunbb on Saturday January 07, 2023 @12:36AM
(#34285550)
Can you post some of your scripts you use? I’d love to get some inspiration.
Re:Learn to Process Text in Linux Using Grep, Sed, and Awk
by 1vuio0pswjnm7 on Saturday January 07, 2023 @01:25AM
(#34285739)
If provide a text-processing problem I will try to provide a solution.
Re:Learn to Process Text in Linux Using Grep, Sed, and Awk
by 1vuio0pswjnm7 on Tuesday January 10, 2023 @01:20AM
(#34321202)
Here is another variation that does not used "sed -i". It uses a favourite hex editor called ired.
test ! -h $1||exec echo $0: error: symlink
case $# in :)
;;1) x=$(sed -n '$=' $1)
test $x -gt 1||exec echo usage: $0 dst src
od -tx1 -An|sed 's/^/w /'|ired -n $1 /dev/stdin
;;2) printf '0r '$2'\nwq\n'|ed -s $1
esac
Re:Learn to Process Text in Linux Using Grep, Sed, and Awk
by 1vuio0pswjnm7 on Saturday January 07, 2023 @08:22PM
(#34294997)
Here are a couple more, edited for HN of course. These are scripts/snippets that would often be used in or by other scripts. Each illustrates some basic quick and dirty usage of tr, sed and grep. These scripts will last for many years with zero "maintenance". They will probably still be working after I have passed away. It seems that python scripts OTOH may have a shelf life.
Task 1: Transform the BIND format of stub resolver output, i.e., the format used by "dig" and many other stub resolvers, to some other format, like HOSTS file, BIND/tinydns zone file, haproxy map file, etc. The input would typically be catenated stub resolver output for hundreds to thousands of domains. The epoch program is three lines of C.
NB. For large input on Linux, dash is significantly faster than bash. I wrote a C program that does the job of this script, faster than dash, and much faster than python. However I still prefer the shell for testing ideas, quickly.
EPOCH=$(epoch);
tr -d '\12'|tr ';' '\12' \
|sed -n '/ANSWER SECTION/{s/ ANSWER SECTION://;
s/\.[^-0-9a-zA-Z].*IN[^-0-9a-zA-Z]*A[^-0-9a-zA-Z]/ 1 IN A /;
# list domains to exclude
/www.google.com/d;
/^$/d;/^.$/d;p;}' \
|{
exec 2>/dev/null 3>&3 4>&4 5>&5 6>&6 7>&7 8>&8 9>&9;
while read NAME TTL CLASS TYPE IPADDR ;do
echo $NAME $TTL $CLASS $TYPE $IPADDR;
echo $NAME 1 $CLASS $TYPE $IPADDR >&3;
echo $NAME $IPADDR >&4;
echo $NAME $IPADDR \# $EPOCH >&5;
echo =$NAME:$IPADDR:1 >&6;
#_IP_()
# {
# echo $IPADDR|grepcidr -f $1-ips.txt >/dev/null;
# }
#_IP_ cloudflare ...
#_IP_ aws ...
#_IP_ cloudfront ...
#_IP_ fastly ...
#_IP_ akamai ...
done;
drill example.com|1.sh 3>>1.zone
kdig example.com|1.sh 4>>/etc/hosts
drill example.com|1.sh 5>>1.map
kdig example.com|1.sh 6>>data
grep -A1 ANSWER.SEC|tr '\11' '\40'|sed -n '/cloudfront.net\.$/{s/\. .* / /;s/\.$//p;}'
drill blogs.aws.amazon.com|2.sh >>2.map
Generally, that rarely happens. We almost never see someone present a text processing problem (input.txt, expected_output.txt) and ask to see what some solutions using UNIX utilties, so that someone might compare with python, for example. Instead, we see endless criticisms of UNIX utilities, without presenting any specific examples (input.txt, expected_output.txt) to ilustrate the unsubstantiated arguments being made.
Without a working example to illustrate a criticism, these criticisms come across as nothing more than worthless opinions. Programmers commenting online provide a lifetime supply of such opinions.
For me, sed is a "daily driver" for text processing. There are numerous small scripts that I use every day that feature sed. I have a folder with hundreds of scripts that use sed. Some use grep and awk as well. However the best program for text processing for me is neither grep, sed nor awk. It's flex.
If the argument is that python looks like "pseudocode" and sed does not, then I agree. No contest. No examples needed. Someone else long ago decided what "pseudocode" should look like.
If the argument is that python has the best or most libraries, again I would not contest that.
However if the argument is something like "python solutions for text processing
(a) are faster to write,
(b) are smaller,
(c) run faster,
(d) are more robust/reliable,
(e) are quicker/easier to edit ("maintain"),
(f) use less memory/CPU,
(g) have fewer dependencies, or
(h) are easier to read",
then we need to look at the problem in question (input.txt, expected_output.txt). Otherwise we cannot have a meaningful discussion.
Python is just too slow for me. It may be fast enough for someone else, but not for me. Being forced to use python due to a job requirement, using python as a result of pressure from other programmers, or using python because it was "recommended", is not the same as first learning UNIX utilities and then evaluating python. I learned sed and other UNIX utilities first and so any potential "replacement" must offer the same benefits.