News for hackers. Stuff that matters.
Re:Simple apply/filter/reduce package in Go by inglor on Friday April 03, 2015 @09:12AM (#9315629)
I don't get why someone would rather write a for loop than use declarative data syntax. If I want to get the names of all administrators doing `users.Where(user => user.isAdmin()).Select(user => user.Name)` is so much nicer than using a for loop - or maybe he's suggesting we start writing FOR loops instead of SQL for our databases too?
Re:Simple apply/filter/reduce package in Go by adrusi on Friday April 03, 2015 @09:37AM (#9315763)
Because for loops are special forms to the compiler and don't require general parametric polymorphism to be implemented in the language. Parametric polymorphism adds a lot of complexity to the language, so I guess the go team has decided to just not have it in the language. I'm not sure I agree with that decision, but it is true that parametric polymorphism isn't all that useful for the types of programs go is usually used for.

You don't want to use this implementation because it uses reflection, which makes it slow and means the compiler can't catch your errors.

Re:Simple apply/filter/reduce package in Go by bunderbunder on Friday April 03, 2015 @09:18AM (#9315659)
In most languages, the for loop ends up being faster. Sometimes noticeably so. Me, I generally start with declarative syntax, but the profiler frequently tells me to go back and change it.

Being a systems programmer, wonder if it's easier for him to just use the for loop as a default. Performance demands are always high in systems programming (because there'll be a whole stack of additional software standing on top of your code and depending on it for performance), so one might end up needing to use for loops often enough that it's easier to just use them all the time for consistency's sake.

Re:Simple apply/filter/reduce package in Go by pcwalton on Friday April 03, 2015 @11:41AM (#9316460)
> In most languages, the for loop ends up being faster. Sometimes noticeably so. Me, I generally start with declarative syntax, but the profiler frequently tells me to go back and change it.

The optimization techniques to make higher-order functions compile down into the same code as a for loop are well-known. All you have to do is inline, constant propagate, and maybe SROA. Every optimizing compiler I know of, even JavaScript, has no problem doing this.

Re:Simple apply/filter/reduce package in Go by Veedrac on Tuesday April 07, 2015 @01:50AM (#9332213)
FWIW, PyPy has a hard time with this since Python's iterators are really heavyweight.
Re:Simple apply/filter/reduce package in Go by drcode on Friday April 03, 2015 @09:24AM (#9315685)
That's the essence of Clojure's transducers http://clojure.org/transducers

...they give you the declarative syntax of filter/reduce/etc but can have the same evaluation strategy as for loops, with comparable performance.

Re:Simple apply/filter/reduce package in Go by adrusi on Friday April 03, 2015 @09:46AM (#9315815)
That's not why clojure has transducers. You can use a mapping transducer to express map, but it's actually going to have slightly worse performance than regular map.

There's nothing inherently slow about higher order collection functions, rust has them and they compile to the same machine code as the equivalent loop construct. Its just that most languages implement them on the wrong data structures. Functional languages implement them on lazy lists, which is good, but has overhead. A lot of languages implement them on arrays, which is bad because then it needs to process the whole array at once, and allocate all the memory, even though the next transformation in the pipeline doesn't need all that memory. Rust implements them on iterators, which have all the sane benefits as lazy lists, but fit better into a performance-focused imperative language.

Re:Simple apply/filter/reduce package in Go by bunderbunder on Friday April 03, 2015 @10:46AM (#9316156)
.NET implements them on iterators, too, but that's not an inherently fast approach. .NET iterators are objects that implement a specific interface, and iterating over them involves two function calls per element - one to move to the next item, and one to retrieve the current item.

Those two function calls introduce an overhead that may be significant in a "tight loop", and that may not exist in a hand-coded for loop.

Perhaps .NET's optimizer is smart enough to get things down to a for loop equivalent in some cases, but in general I haven't seen much evidence of that happening.

Re:Simple apply/filter/reduce package in Go by pcwalton on Friday April 03, 2015 @11:43AM (#9316464)
> .NET iterators are objects that implement a specific interface, and iterating over them involves two function calls per element - one to move to the next item, and one to retrieve the current item.

Any optimizing compiler will inline those functions. "Avoid function calls for performance" hasn't been relevant optimization advice for at least a decade.

Re:Simple apply/filter/reduce package in Go by bunderbunder on Friday April 03, 2015 @01:58PM (#9317338)
A good optimizer can inline those functions, but that doesn't necessarily mean it will. .NET JIT is conservative in what it decides to inline. There's even a special hint you can use to ask it to be more aggressive about inlining a method. You can't necessarily apply that hint to every function, though. Anonymous ones, for example.
Re:Simple apply/filter/reduce package in Go by pcwalton on Friday April 03, 2015 @02:34PM (#9317550)
> A good optimizer can inline those functions, but that doesn't necessarily mean it will.

If the optimizer isn't choosing to inline where it is a performance win, it's not a good optimizer.

Especially for trivial functions like Current() and MoveNext(), which are pretty much the textbook example of functions where inlining really helps (not so much because of the function call overhead, but because it enables so many intraprocedural loop optimizations).

Re:Simple apply/filter/reduce package in Go by inglor on Friday April 03, 2015 @11:56AM (#9316561)
In particular - the Microsoft .NET JIT will gladly make them as fast as loops in most cases - there might be overhead involving closures but that's mostly taken care of.
Re:Simple apply/filter/reduce package in Go by drcode on Friday April 03, 2015 @10:57AM (#9316224)
> You can use a mapping transducer to express map, but it's actually going to have slightly worse performance than regular map.

That's a strawman argument- Neither OP nor my comment made any mention of "map". This is a discussion about reduction and filtering. Yes, using transducers in a way that is silly like you suggest will produce disappointing results.

Re:Simple apply/filter/reduce package in Go by adrusi on Friday April 03, 2015 @02:13PM (#9317436)
Bad example, but my point holds.

Transducers aren't performance improvers, they're an abstraction over the process of iteration. They actually degrade performance in the absence of a particularly smart inlining compiler.

Re:Simple apply/filter/reduce package in Go by drcode on Friday April 03, 2015 @02:34PM (#9317548)
> They actually degrade performance in the absence of a particularly smart inlining compiler.

So it sounds like you're saying that iterators in Rust are faster than transducer-based filter/reduce.

I think it's clear that my original point was that transducer-based filter/reduce is faster than lazy-list based filter/reduce, and comparable in performance to a for loop. (in that lazy-list operations are MUCH MUCH slower in most cases for filter reduce operations than the latter two approaches)

Re:Simple apply/filter/reduce package in Go by bjz_ on Friday April 03, 2015 @11:28AM (#9316394)
I'm not sure what languages you are referring to, but Rust and D's iterator methods compile to the same code that you would write if you were using raw loops.
Re:Simple apply/filter/reduce package in Go by mhd on Friday April 03, 2015 @09:29AM (#9315710)
Wasn't one of Pike's earliest published whitepapers basically stating that "most of what you need is binary trees, linked lists and hashes" when it came to data structures/algorithms? (Probably add CSP to that list, given his prior work)

So I'd dare to say he's a bit of a minimalist. Fanboys and fellow travelers aside, one of the few persons with a similar point of view that I can think of would be Niklaus Wirth.

But that's the luxury that research and academia offer to you, if your problems can be approach from a tabula rasa view, you can tailor your tools to be similary pure (his "pure" obviously not being of the mathematical/functional persuasion).

I mean, there's no solid reason why we have several different kinds of screwdrivers. But that doesn't help you when you have to assemble your IKEA Wöbsörwös furniture to earn your living.

Re:Simple apply/filter/reduce package in Go by fosforsvenne on Friday April 03, 2015 @09:36AM (#9315755)
The letter W isn't used in Swedish except in proper names and the occasional loanword.
Re:Simple apply/filter/reduce package in Go by mhd on Friday April 03, 2015 @09:41AM (#9315782)
And just when I felt so smart about using "ö" and not "ø" for fake furniture…
Re:Simple apply/filter/reduce package in Go by fosforsvenne on Friday April 03, 2015 @09:49AM (#9315833)
Actually, I both noticed and appreciated that :). And a lot of IKEA product names are just first names anyway, so it _was_ overly pedantic.
Re:Simple apply/filter/reduce package in Go by seanmcdirmid on Friday April 03, 2015 @09:24AM (#9315686)
Debugging. Try debugging a Select/Where stream in C#; it is a PITA. I often find myself unwinding my list comprehensions into for loops because I need to debug.

There is a reason Haskell's type system has to be so strong :)

Re:Simple apply/filter/reduce package in Go by eropple on Friday April 03, 2015 @09:40AM (#9315777)
I was pretty confused by this post for a minute until I remembered I've been on VS2015 since the first preview release. This problem is greatly reduced there.
Re:Simple apply/filter/reduce package in Go by seanmcdirmid on Friday April 03, 2015 @09:46AM (#9315814)
I saw that in June, and I'm interested in how effective it is, and what the usability characteristics are (it looked complex at the time, but it was an early demo). But going from 2010 to 2013 was a painful, and I don't want to throw this in a VM, so I'll just wait for RTM.
Re:Simple apply/filter/reduce package in Go by inglor on Friday April 03, 2015 @11:55AM (#9316547)
Super useful. Debugging Select/Where is now as easy as debugging for loops. Also remember that the pain point for debugging is the laziness and not the declarative syntax and you can just use `.ToList()` to get rid of it.
Re:Simple apply/filter/reduce package in Go by eropple on Friday April 03, 2015 @10:37AM (#9316089)
2013 to 2015 is a much smaller jump. Unless you are doing weirder things than me (and I mean, that's kind of hard!), I'd bet you'd be party. YMMV, of course.
Re:Simple apply/filter/reduce package in Go by seanmcdirmid on Friday April 03, 2015 @05:54PM (#9318823)
I use a plugin that makes curly braces very small, so that my C# code looks more like Python. it seems to break on every version upgrade, and I can't stand normal sized C# curly braces anymore.
Re:Simple apply/filter/reduce package in Go by eropple on Saturday April 04, 2015 @02:00PM (#9321593)
I'd honestly be surprised if that broke this time around. Every extension I've tried in VS2015 worked fine. Some I had to fight with (VSIX manifests with too-restrictive versions), but it's mostly just drop-in-and-go.

They do run side-by-side just fine, too, FWIW.

Re:Simple apply/filter/reduce package in Go by svisser on Friday April 03, 2015 @09:18AM (#9315660)
It's easy to dismiss an opinion but that doesn't mean it's wrong. For example, if you like filter/reduce you may criticize languages that don't encourage it (Go, Python, etc). But often this leads to copying ideas from one language resulting in code that's hard to maintain in another language which encourages different ways of expressing the same logic.
Re:Simple apply/filter/reduce package in Go by metaphorm on Friday April 03, 2015 @09:24AM (#9315689)
filter and reduce are both builtins in Python and I see nothing discouraging anyone from taking full advantage of them.
Re:Simple apply/filter/reduce package in Go by svisser on Friday April 03, 2015 @09:29AM (#9315714)
In Python 3 reduce was removed and you'd have to use functools.reduce. Similarly lambda functions are a bit of a mistake, in Guido's opinion, as well as his preference to use list comprehensions over filter(). I don't expect to see more functional programming constructs anytime soon in Python. That being said, list comprehensions work quite well in Haskell, Python, etc.
Re:Simple apply/filter/reduce package in Go by kasabali on Friday April 03, 2015 @09:44AM (#9315801)
GvR discourages them if that's a strong enough reason for you :)
Re:Simple apply/filter/reduce package in Go by metaphorm on Friday April 03, 2015 @10:50AM (#9316176)
its not, actually. he seems to discourage them on the grounds that a list comprehension would be better. however, that's just alternate syntax for a logically equivalent operation. personally, I use whichever one is more clear to me based on context. I appreciate having the option. Maybe you say that's un-Pythonic because "there should be just one way to do things", but I'm not particularly dogmatic and not persuaded by that.
Re:Simple apply/filter/reduce package in Go by falcolas on Friday April 03, 2015 @11:06AM (#9316269)
List comprehension produces a different output than reduce (since reduce does not have to produce a list). Reduce and for loops are better compared.

For loops make the logic occurring on each operator explicit and in the current frame of reference, as opposed to reduce, which splits the logic from the point of operation.

Also, when reduce is used to produce an artifact similar to a list comprehension: the list comprehension is going to be better optimized, more comprehensible for many programmers, and with two character changes can be turned into a generator instead of a memory-heavy list.

Re:Simple apply/filter/reduce package in Go by inglor on Friday April 03, 2015 @11:58AM (#9316573)
Python is a bad example since it has declarative data manipulation syntax. You'd use a comprehension and not a for loop which is practically the same thing in my opinion.
Re:Simple apply/filter/reduce package in Go by NateDad on Friday April 03, 2015 @11:47AM (#9316493)

    var admins = users.Where(user => user.isAdmin()).Select(user => user.Name)
vs

    var admins []string
    for _, user := range users {
        if user.IsAdmin() {
            admins = append(admins, user.Name)
        }
    }
So, at the end of the day, these two pieces of code do the exact same thing, and have mostly the exact same meaning. I read them almost identically...

Make a list of admins. iterate through the users, if the user is an admin, append its name to the list of admins.

The unrolled version is a lot easier to modify later. What if you decide that each of these admin users should have a star next to their name? For the go code, you just add one line:

    var admins []string
    for _, user := range users {
        if user.IsAdmin() {
            admins = append(admins, user.Name)
            user.Name = user.Name + "*"
        }
    }
for the C# code, you either have to write a second LINQ query (and now you're iterating the list twice unnecessarily), or you have to unroll it into a normal loop.

So now you've added complexity to the language that is only really good for saving carriage returns, and it makes your code harder to maintain. Sure, it's pretty and interesting from a programming theory POV... but at the end of the day, I don't get paid to write pretty or interesting code. I do get paid to write maintainable code.

Re:Simple apply/filter/reduce package in Go by inglor on Friday April 03, 2015 @11:53AM (#9316535)
I humbly disagree - in the C# code I can pass an IQueryable and reuse the same code to make an SQL query, or a RavenDB query, or an XML query and it works just the same - there is a unified way to access data in the language.

As for your query:

``` var admins = users.Where(u => u.IsAdmin()).Select(u => u.Name); var stars = admins.Select(name => name + "*"); ```

Like you said, you don't get paid per line feed - it's ok to have multiple lines and you can keep the nice syntax. It's ok to not like complicated code - no one likes complicated code - which is why I don't like for loops - I don't _care_ how something is iterated I just want to map every object.

Re:Simple apply/filter/reduce package in Go by NateDad on Friday April 03, 2015 @02:35PM (#9317553)
Well, so, that's actually not doing the right thing. You just changed the names in the list of strings that holds admin names, but not the names on the user objects themselves.

Even if the first half were returning the users themselves to the admins list, you're still iterating over the list twice.

This is yet another reason why the for loop is good - it makes it more clear what is actually going on. This is your code in for loops:

    var admins []string
    for _, u := range users {
        if u.IsAdmin() {
            admins = append(admins, u.Name)
        }
    }
    
    for i := range admins {
        admins[i] = admins[i] + "*"
    }
The problem with the fancy syntax is that it makes it too easy to write suboptimal code that you'd never write if you were writing plain old for loops, like the above.
Re:Simple apply/filter/reduce package in Go by frowaway001 on Sunday April 05, 2015 @06:22AM (#9323634)
> you're still iterating over the list twice

There is nothing about higher-order operations that would require developers to implement xs.map(...).map(...).map(...) or xs.Select(...).Select(...).Select(...) or whatever as multiple traversals over the data structure.

> The problem with the fancy syntax is that it makes it too easy to write suboptimal code that you'd never write if you were writing plain old for loops, like the above.

Maybe Google should just invest in hiring better people? I know it's getting increasingly harder to find people who still haven't evolved from the 1960ies mindset, but still ...

Re:Simple apply/filter/reduce package in Go by djur on Friday April 03, 2015 @12:00PM (#9316593)
I'm not familiar specifically with LINQ but in most languages that implement map/reduce/filter operations, the "*" requirement could be clearly expressed using a map.

What happens when you need an alphabetical list of admins who have logged in during the last week? The for-loop approach becomes progressively more and more cluttered with implementation cruft, while a declarative approach just adds a filter and a sort.

And, to be frank, even the example you present is clearer to me in LINQ than in Go, and I am much more familiar with Go.