Reference: https://www.gnu.org/software/sed/manual/sed.html#Examples
Sorry, I couldn't miss that one. :) Really, check the example. Sed is most powerful tool and can do astonishing work.
It would however be nice to have a tool with similar power but simpler, more comprehensible syntax. One of the other commenters linked to an interesting document on sam, which has a better control flow but equally arcane syntax.
With Perl 5, Perl essentially repositioned itself as a full-blown, general-purpose programming language, with all the power that entailed, but still stuck with a syntax based on essentially trying to be a more powerful and consistent superset of sed and awk.
These days, I think we could do a lot better with a stream processor with the stream-processing focus and one-liner power of Perl 4 but without the needless (these days) constraint of retaining the syntactic hash of ancient unix utilities.
I'd argue that makes it one of the least useful documents you've ever read :)
A job for life! :-)
Streams are a non-trivial construction, after all.
>leaves a blank line at the beginning and end if there are some already.
#!/usr/bin/sed -f
# on empty lines, join with next
# Note there is a star in the regexp
:x
/^\n*$/ {
N
bx
}
# now, squeeze all '\n', this can be also done by:
# s/^\(\n\)*/\1/
s/\n*/\
/
Arcane. Some would call it archaic too :)
> Streams are a non-trivial construction, after all.
...
> Well, sorry but sed _is_ cryptic. Let me quote an exemple for you (squeezing blank lines):
I agree with you that sed is cryptic, but I don't think that that necessarily means that stream processing languages are doomed to have an archaic syntax. I'd also agree with marvy in saying that the car and house example is very clear as to it's intent:
STDIN | /\w+/{|word|
/house/ {
# when word is house
}
/car/ {
# when word is car
}
{
# default case
}
}
EDITS: formatting.
Yes, splitting and combining. Looking at the code example, I feel like so much opportunity is just being squandered, where something like Rust's mapping constructs would feel so much better. Yours is definitely more in that vein, here's something a little closer:
STDIN | /\w+/{|word|
/house/ => "foo"
/car/ => "bar"
"literal" => "baz"
_ => "dib"
# (where _ is a magical symbol for the default case... could be anything)
}Might also be nice to borrow some of Scala's features for terser code than Ruby, like
ary | { _ + 5 }
ary | { |el| el + 5 } {|el| el + 5}
{ _ + 5 }
Compare:
ary.map(_ * 2)
ary.map(x => x * 2)
Of course, in a more functional language you could perhaps just write
ary.map(* 2)You might check out http://stackoverflow.com/a/8001065/3614122. The interesting thing (to me) is that the majority of those are actually the same feature (lifting functions). It's just such a powerful/flexible one that it's often misinterpreted as an entirely new thing in different contexts.
ary.map(_ * 2)
val f = (_:Int) * 2
ary.map(f)
val f: Int => Int = _ * 2
ary.map(f)
Some of the other examples of the underscore are pattern-matches, which can be used on the LHS of assignment, similar to Erlang. Which comes in handy writing unit-tests. i.e.: If I expect something to return Some[User], and I'm writing a GET->UPDATE integration test, then I'd probably do something like:
"Update should not blow up" in async {
val Some(user) = await(db.get(userId))
val update = user.copy(name = "bob")
val Success(result) = await(db.put(update))
}
"Update should not blow up with a for-comprehension" in {
val test = for {
Some(user) <- db.get(userId)
update = user.copy(name = "bob")
Success(result) <- db.put(update)
} yield result
whenReady(result) { "whatever" }
} Arr.map { $1 * 2 }Wait, isn't that already possible in Scala?
ary.map(2*) ary.map(f(_ * 2))
==> ary.map(x => f(x * 2)) ?
==> ary.map(f(x => x * 2)) ?
scala> val f: Int => Int = _ - 1
f: Int => Int = <function1>
scala> val ary = Array(1,2,3)
ary: Array[Int] = Array(1, 2, 3)
scala> ary.map(f(_ * 2))
<console>:10: error: missing parameter type for expanded function ((x$1) => x$1.$times(2))
ary.map(f(_ * 2))
^
So simplify it: map in this context wants a Function[Int, Int] right? So f(_ * 2) must return a Function[Int,Int]. But it doesn't probably. _ is lifting some argument out of whatever f is. If you assume _ is an Int, does f take a Function[Int,Int]? No. It takes an Int. So there's no way to parse this that makes sense. It's not just "I have a stack of vars, pull one off the stack and bind it every time I write an underscore, reading left to right". That would be some AST generative grammar hack. That's not what the underscore is. It's simpler and more consistent than that.
What you're probably looking for instead is Function Composition. So something like:
scala> ary.map(f compose(_ * 2))
res10: Array[Int] = Array(1, 3, 5)
(Is it correct to say in this context f() is a Monad? I'm not sure, I need to sit down and grok the category stuff sometime...)
Or you can write it the long way (calling f inside a new function). But instead of defining "steps" you'd be creating a new imperative function and driving the stack.
Array((_:Int) * 2)
==> (x:Int) => Array(x * 2) ? (correctly typed)
==> Array((x:Int) => x * 2) ? (also correctly typed)
I mean, I wouldn't say the feature is bad, but it definitely has a DWIM vibe to it that would make me classify it as a hack.
I think this is where I'm not communicating what I'm trying to say very well. What I'm trying to say is that that statement is false.
The underscore isn't a placeholder saying "inject a Function here". You probably didn't mean that exactly of course, but it's a programming language; it pays to be a bit pedantic I think.
It's easier to understand if you work backwards maybe.
Is this valid?
(_:Int) * 2
So now you take that perfectly valid piece of code lifting a method to a function and you pass it to Array.apply:
Array.apply((_:Int) * 2)
What sense does that make? That seems like straight up voodoo.
There's some syntax supporting this feature. But not so much as you think. It's just the "lift". Type-inferencing lets you get away with what looks like a little more sometimes.
I dunno. Maybe that's helpful. Maybe not.
It is. Look at the error message:
error: missing parameter type for expanded function ((x$1) => x$1.$times(2))
> You had a Function[Int,Int] before, but now that you've passed that function value to Array.apply instead of getting an Array[Function[Int,Int]] as you would in every single other case, you get a Function[Int,Array[Int]]?
That's besides the point. The question is, when the parser sees "_", how much of the context does it grab along with it? In other words, I know it's lifting stuff, what I want to know is how much it lifts. Here's another example:
f(_, 2)(3)
==> x => (f(x, 2)(3)) ?
==> (x => f(x, 2))(3) ?
> But not so much as you think. It's just the "lift". Type-inferencing lets you get away with what looks like a little more sometimes.
I still don't see what type inference has to do with this. The error message makes it clear that the lift is done before type inference kicks in. In a dynamic language, you would stop at the lift, but it would otherwise work just the same.
Yes, the inferencer/type-resolver runs after the parser (there's a recent thread on the ML about a Parboiled parser you might find interesting BTW). My point was this is parsed. It's an AST at that point, it just hasn't resolved the types yet. It's not a source preprocessor. To me that implies limitations, but also an expectation of consistency.
> ... what about "1 + _ * 2"? What is it lifting? The asterisk? No. It is lifting more than that.
Yes! Paste your code into the REPL and see what you get:
scala> 1 + _ * 2
<console>:8: error: missing parameter type for expanded function ((x$1) => 1.$plus(x$1.$times(2)))
1 + _ * 2
1.+(_ * 2)
(1 + _) * 2
They're doing exactly the opposite thing right? It's like assuming that the LHS and RHS of an assignment are interchangeable. They're not. One is a label and one is a value.
> how much of the context does it grab along with it?
None. In this position, it's the beginning of an expression. It doesn't escape the scope it's defined in. Just like in the 1.+(_ * 2) example, what comes before it doesn't matter. It's the beginning of an expression.
> I still don't see what type inference has to do with this.
I probably just complicated things bringing it up. ary.map(_ * 2) vs ary.map((_:Int) * 2). I feel like maybe it's not understood that those do the same thing. The only difference is the type is inferred from the signature of map[T] (simplified minus CBF) on the first.
I'm not trying to debate that sometimes it's frustrating that I can't just:
documents foreach(couchdbActor ! Queue(json.merge(_)))
documents foreach(couchdbActor.tell(Queue(JsObject => JsObject))
documents foreach((couchdbActor ! _) compose Queue compose json.merge)
(couchdbActor ! _) // This just lifts the tell method, which gives me a Function[Any,Unit].
Queue.apply // Function[JsObject,Queue]: I can compose that with my lifted tell to get a Function[JsObject,Unit]
json.merge // Function[JsObject,JsObject]: Doesn't change the signature: Function[JsObject,Unit]
When you break it down, you have to keep a "stack" in your head. Which can get hard sometimes (at least for me), but the underlying consistency about what's actually happening makes it simple to do. It's like a basic addition problem:
27 + 34 + 192 + 11
> In a dynamic language, you would stop at the lift, but it would otherwise work just the same.
I was going to attempt to write a Ruby analogue, but I can't actually figure out a clean way to do that.
I dunno. I feel like you probably understand all this just fine. You just don't like it. You'd prefer a sort of source-rewriting approach? So maybe Scala just isn't the language for you. I feel like once you understand the mechanics of the underscore, it's pretty straight-forward. Similar to beginners getting tripped up on for-comprehensions:
val o = Option(1)
val f = Future.successful(List(2,3,4))
for {
n <- o
l <- f
x <- l
} yield n + x
Side note: This is a big part of why I find for-comprehensions mostly useless. They do nothing you can't already do, and they introduce what can look like magic. They're pure sugar, except more often than not, they're also longer to write. Plus they're mostly useless outside of testing. How often do you want a Failure to just throw? Or a Play Action to return a None? More often than not, you're gonna want to fold(), getOrElse(), match { case None => NotFound; case Some(foo) => ... Ok(finalResult) } etc. But I digress. :-)
> 1.+(_ * 2)
No, my point is that this is not how it parses. Look:
ary.map(1 + (_:Int) * 2)
res3: Array[Int] = Array(3, 5, 7, 9, 11)
ary.map(1.+((_:Int) * 2))
error: overloaded method value + with alternatives:
(Double)Double
...
>> how much of the context does it grab along with it?
> None. In this position, it's the beginning of an expression. It doesn't escape the scope it's defined in. Just like in the 1.+(_ * 2) example, what comes before it doesn't matter. It's the beginning of an expression.
If that was true, then "1 + _ * 2" would be invalid, and "f(_)(1)" would be equivalent to "f(1)". But the former is valid and the latter doesn't work like that.
> I was going to attempt to write a Ruby analogue, but I can't actually figure out a clean way to do that.
There is a limited Python analogue:
https://github.com/kachayev/fn.py#scala-style-lambdas-defini...
That only works insofar that the _ placeholder object can seize control over the expression, so "f(_, 2)" won't work unless f explicitly handles _. Implementing the full feature would require language or macro support, but my point was more that a new dynamic language could trivially support it.
STDIN | /\w+/ | {|each match|
/house/ => "foo"
/car/ => "bar"
"literal" => "baz"
_ => "dib"
}
I think this is a more flexible solution, and more modifiers than each could be introduced. For instance, sum of squares could be:
seq(100) | {|each x| x * x} | {|reduce (x = 0, y)| x + y} "~/yourfile.txt" utf8 file-contents
"\\w+" findall [ first second ] map [
{
{ "house" [ "house stuff" print ] }
{ "car" [ "car" print ] }
[ "default stuff: %u\n" printf ]
} case
] each
It shows well how a slight shift on the problem expression can lead to a more powerful, simpler to use and general solution. In this case, the shift is to design stream processors (like `sed` and `awk`) to work on streams of text chunks defined by regular expressions rather on streams of lines matching regular expressions.
Regarding this concern, it's hard to say what is the plan of Matz for Streem with a single code example. But the fact, that this sample code processes a stream of integers, lets hope a tool with more flexibility than `sed` or `awk` to specify what is processed along the streams.
The link was posted on HN some time ago and was generally well received. I didn't have the time to use it much yet, but it looks very nice. Well, unless you hate lisp.
Some ways of doing UNIX-style pipes in Python:
http://jugad2.blogspot.in/2011/09/some-ways-of-doing-unix-st...
And that later inspired me to create an experimental tool called pipe_controller:
Swapping pipe components at runtime with pipe_controller:
http://jugad2.blogspot.in/2012/10/swapping-pipe-components-a...
The above post links (recursively) to a few others, also by me, describing a couple of other ways of using pipe_controller, including this one:
Using PipeController to run a pipe incrementally:
http://jugad2.blogspot.in/2012/09/using-pipecontroller-to-ru...
and also includes the link for it on my Bitbucket account.
There's also plumbum, another such Python module.
I'm playing with the idea of build a language, where the functions are unix-like, with STDIN, OUT & ERR. So, instead of raise a exception, it put data in ERR... and make it easy to compose them.
If you google stream processing language you will also get a whole bunch that are not text specific. They are definitely worth looking at if you are interested in building any kind of stream based language.
As is you would have to put in filters to resplit input into lines and that is very messy for something that you will need/want to do very often.
For example if you wanted to parse by character it would be wonderful to be able to do the following:
Even better would be if you took it a step further and offered something like regex pattern matching for the block input. e.g.