News for hackers. Stuff that matters.
Re:Introduction to sed by gjm11 on Friday October 07, 2011 @12:04PM (#3084567)
Flagged for excessive self-promotion. Take out "World's best" or make the article provide strong evidence that it's the world's best beyond the fact that its author thinks it is, and I'll unflag it.

If this didn't have "World's best" in the blog title and someone had added that to the HN title, a moderator would be deleting those words. The fact that you've inserted the same peacock words in your own blog title as well as your HN submission does not make this any better.

(I don't like being so blunt with a highly regarded veteran HN user, but abuse is abuse whoever does it. Sorry, Peteris.)

[EDITED to add: The HN guidelines say not to comment saying "I flagged this". I think this is an unusual case. I appreciate that others may disagree.]

Re:Introduction to sed by pkrumins on Friday October 07, 2011 @12:13PM (#3084613)
Thanks for letting me know. I just removed the "World's best." I just thought I'd name it "World's best" cause I was feeling fantastic today. :)
Re:Introduction to sed by gjm11 on Friday October 07, 2011 @02:06PM (#3085129)
Unflagged. Thanks.
Re:Introduction to sed by acpmasquerade on Saturday October 08, 2011 @01:24AM (#3087211)
But the word "World's Best" is still there in the URL, even in the Page Title and the article title. boo !!
Re:Introduction to sed by nabb on Friday October 07, 2011 @12:08PM (#3084581)
The introduction was alright, but I'm not sure that the hold buffer needs to be introduced straight away. Using the hold buffer can get complicated pretty quickly, and it isn't something that you'll frequently have to do. The example the author gives of getting lines matching a regex with the previous line is more easily done using grep -B1, which doesn't require the mental overhead of sed scripting.

Perhaps more useful for an introduction would be retaining portions of the input back-references (\1 to \9) and `&'. Also useful is is the `g' flag for the `s' command, for global replacement, and the `i' flag (a GNU extension) for case insensitivity. Another note is that you can actually use any character to delimit `s' commands you want, which is useful if you have /'s in your pattern or replacement strings. The last main part of sed which I use regularly is the -r option, for extended regular expressions (basically the same as grep), which lets you use regular expression tokens like + and | without escaping. On that note, the author misses this point in his introduction, and the example of "sed -n '/a+b+/p'" should actually be either "sed -rn '/a+b+/p'" or "sed -n '/a\+b\+/p'".

For a serious introduction to all the extra functionality in sed, there's a great reference at the top google result for sed[1]. In addition, GNU sed adds functionality that you might be useful at some point (e.g. s///e), all of which are described in the GNU sed user's manual[2].

[1] http://www.grymoire.com/Unix/Sed.html [2] http://www.gnu.org/software/sed/manual/index.html

Re:Introduction to sed by pkrumins on Friday October 07, 2011 @12:25PM (#3084697)
Thank you, this is a really great comment! I'll improve my introduction based on your feedback to make it better.
Re:Introduction to sed by hsmyers on Friday October 07, 2011 @11:26AM (#3084350)
Are there any HN reviews of this guys eBooks? I think he has two on SED and I'm a little leery of self promotion although the preview was pretty good.
Re:Introduction to sed by bradleyland on Friday October 07, 2011 @11:56AM (#3084519)
I've got both one-liners books (Sed and Awk). Peteris takes a very practical approach in the book. I almost think of it like a field guide. Think of the times you've google searched for something like 'number lines in file sed'. These books not only provide solutions, but give detailed explanations of what's happening. Pick any given example and there's a good chance you'll gain some insight about how sed/awk work. They're very well written, and very accessible.

Having said that, if you're a long beard who has long been acquainted with these tools, you could maybe skip them.

Re:Introduction to sed by route66 on Friday October 07, 2011 @07:02PM (#3086354)
As the author also explains in the introduction, the compilation is based on the earlier compilation by Eric Pement, available as .txt files on his website.

http://www.pement.org/sed/sed1line.txt

http://www.pement.org/awk/awk1line.txt

Re:Introduction to sed by dbbo on Friday October 07, 2011 @01:26PM (#3084943)
To quote "The Awful Truth about sed" section of the Grymoire, "It is not your fault you don't understand sed."

I do, however, understand Perl, so that's what I use. E.g.: `echo 'fubar' | perl -lpe 's/fu/foo/'`

It might be a few milliseconds slower-- for example, commenting every line of my 352-line .zshrc (via s/^/#/) takes 0.007s total with Perl (so does `s/^/#/ unless /^\s*#/`) and 0.005s with sed. Commenting out every line of /usr/share/dict/american-english (98569 lines) takes 1.124s with Perl and 0.813s with sed.

Since I already know how to do more complicated things with Perl (like conditionals, named backreferences, etc.) it doesn't seem worth it to take the time to learn how to use sed effectively. I can wait the extra second since I'm not on any kind of deadline or under any efficiency constraints.

I am not trying to say that Perl is better than sed or any other text processing tool. I also don't mean to imply that speed is sed's only advantage-- it's just one example. I think that for someone who already knows some Perl, learning another similar tool doesn't make sense. I'm sure there are exceptions. This is only my humble, personal opinion.

For people who do need/want to learn sed, the article did a pretty good job of showing you how to get a lot done without a whole lot of reading.

Re:Introduction to sed by _delirium on Friday October 07, 2011 @02:13PM (#3085167)
On the speed question, I actually find Perl considerably faster than sed in a lot of use-cases, if there's enough processing to dominate the slightly higher startup costs of Perl.

For example, at one point I had reason to take a gigantic single-line textfile, and break it into lines based on a specific 3-letter pattern that didn't occur anywhere else:

   s/ABC/A\nC/g
In whatever sed comes with Debian, this took about 10 minutes, CPU-bound, for a 2-gigabyte file. With Perl: 1.5 minutes, IO-bound. Not too sure why. Maybe sed runs everything through the regex engine, while Perl special-cases constant strings? Perhaps Perl has better buffer management for processing gigabytes of text? I haven't done any real testing.
Re:Introduction to sed by dbbo on Friday October 07, 2011 @02:26PM (#3085227)
The copyright section of sed(1) suggests that the Debian package sed (I'm running wheezy) is Gnu sed 4.2.1[1]. Plan9 sed is also available in the 9base package. I'm running perl 5.12.4 (also from Debian testing). I don't see any significant difference in speed with vanilla 5.14.1 and the commands I mentioned earlier.

[1] http://www.gnu.org/software/sed/

Re:Introduction to sed by dramaticus3 on Saturday October 08, 2011 @07:25AM (#3087681)
Turn on locale UTF-8 for GNU software before benchmarking it to compare with Plan9 software.

GNU stuff gets a speed improvement from assuming single byte characters.

Re:Introduction to sed by there on Friday October 07, 2011 @02:48PM (#3085333)
however, perl has -i, which not all seds do, so those 0.002 seconds you lost to perl will be more than gained by being able to type

      perl -pi -e 's/foo/bar/' somefile
instead of

      sed 's/foo/bar/' somefile > somefile.tmp && mv somefile.tmp somefile
Re:Introduction to sed by telemachos on Friday October 07, 2011 @06:46PM (#3086311)
Rather than -i alone, I always use -i.bak (or whatever you prefer instead of '.bak'). That way, you get (1) in-place editing and (2) a backup. The files you work on are backed up to "file.bak", so that if you make some simple regex mistake with huge negative consequences, you have a ready fix. (And if you do things perfectly, it's easy to run "rm *bak" after you confirm that.)
Re:Introduction to sed by burgerbrain on Friday October 07, 2011 @05:57PM (#3086175)
The utility 'sponge' is useful for doing this in the general case.
Re:Introduction to sed by AlecSchueler on Friday October 07, 2011 @09:16PM (#3086696)
Part of the 'moreutils' package, for the curious. The description for sponge on the project homepage reads "soak up standard input and write to a file."
Re:Introduction to sed by pkrumins on Friday October 07, 2011 @03:19PM (#3085463)
Sed has -i, too! Well, at least GNU sed.
Re:Introduction to sed by there on Friday October 07, 2011 @03:21PM (#3085467)
yes, that's why i said "not all seds".
Re:Introduction to sed by pkrumins on Friday October 07, 2011 @03:25PM (#3085488)
missed it. sorry!
Re:Introduction to sed by dramaticus3 on Saturday October 08, 2011 @07:26AM (#3087683)
because GNU is Not Unix
Re:Introduction to sed by hugh3 on Friday October 07, 2011 @02:16PM (#3085177)
I know how to do exactly one thing in sed, and that's sed 's/blah/blahprime/' somefile.

If sed has capabilities other than that, I don't particularly care... but that's one thing that I need to do frequently which is more painful in awk or python.

Re:Introduction to sed by dbbo on Friday October 07, 2011 @02:36PM (#3085277)
I would guess that that's all a significant number of unix-like OS users do with sed, and in those circumstances, it does make more sense to use sed than a more extensive language. For simple substitutions like that, I'd just use an alias: alias ped='perl -lpe' to save the extra keystrokes.
Re:Introduction to sed by dbbo on Friday October 07, 2011 @09:46PM (#3086745)
Apparently the Debian kernel team agrees. I just noticed this while building a kernel package:

test -n "$k" || perl -pli~ -e 's/\$\{shlibs:Depends\}\,?//g' debian/control

Re:Introduction to sed by freemarketteddy on Friday October 07, 2011 @12:25PM (#3084694)
I dont know if this off topic but is there an utlity that can parse C files and print all the function names or local variable names for example.

I know it can be done through sed,but a perfect regex for parsing functions and nothing else can be complex and I am wondering if someone has already done it.

Re:Introduction to sed by grok2 on Friday October 07, 2011 @02:02PM (#3085109)
Look at what the perl script at http://www.gson.org/egypt/ does -- if you are using gcc, you can compile with certain options to generate info from the source code into an intermediate file and then parse this intermediate file with a script to get the info you need. A bit complex, but useful if you are trying to read through code you are maintaining. A compiler is better than a regex for this, I think, for C.

Also ctags/etags like another comment mentioned as also the cscope utility.

Re:Introduction to sed by freemarketteddy on Sunday October 09, 2011 @02:48AM (#3089869)
I checked out Egypt.It is pretty cool but unfortunately it doesnt seem to have support for Objective C.
Re:Introduction to sed by sequoia on Friday October 07, 2011 @01:40PM (#3084991)
Re:Introduction to sed by tmhedberg on Friday October 07, 2011 @02:01PM (#3085100)
In particular, `ctags -x` generates human-readable tabular output (instead of the normal output designed for parsing by text editors) and annotates each symbol to indicate whether it's a variable, function, macro, etc.

If you want to get a list of all function names from file "foo.c", you could do something like:

    ctags -x foo.c | awk '$2=="function"{print $1}'
Re:Introduction to sed by cenuij on Friday October 07, 2011 @12:53PM (#3084803)
I'll just leave this here...

    s/&/&/g
Sorry! :D