> I'm surprised comprehensions in particular haven't spread to more modern languages.

I've come to dislike list comprehensions. Simple ones are ok but they do not handle incremental complexity well. Invariably it means code slowly becomes really unreadable as time goes by because nobody wants to rewrite the list comprehension when one more little tweak stuffed in there will do the job.

I much prefer object-functional chaining style ala Scala, Groovy, Ruby etc:

    some_giant_list
      .findAll { it.foo > 20 }
      .groupBy { it.bar }
      .countBy { it.value.size() > 5 }

They scale much better over time as new constructs get inserted as functional additions in the sequential pipeline.

This assumes some_giant_list is an object that has "findAll", that "findAll" returns and object that has "groupBy", which returns an object that has "countBy".

Looking at that code, I don't even know what the intermediary objects are, but given the names of the operations, they can't be all flat lists.

Besides, you may not put your list comprehension inline. It is often more readable to make it span on several lines, espacially if it's a complex one:

    banned_ip = {
        con.ip 
        for con in connexions 
        if con.ip in blacklist
           or 
           con.type == "internal"
    }

This is exact how I write mine, I find it much more readable personally.

The real challenge I've encountered is typically you can only use one expression. If I need to write a slightly more complex mapping I am forced to write a function, which I normally define just before the comprehension. Even though this works, it introduces boilerplate I'd rather not write.

I will admit this doesn't happen often, but it happens enough to bother me.

> This assumes some_giant_list is an object that has "findAll", that "findAll" returns and object that has "groupBy", which returns an object that has "countBy".

> Looking at that code, I don't even know what the intermediary objects are, but given the names of the operations, they can't be all flat lists.

In regards to this, I would say sure, but does that really matter. Most IDEs will give you enough inference to list the operations you can perform and the return types they give. If anything the "super collections" make a developers life far easier, I think Kotlin does a particularly good job of this with a vast set of extension functions. Have a scroll of this documentation https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collecti... mapIndexedNotNull is a great example

Functional chaining (aka “fluent interfaces”) is very bad for dependency injection / mocking.

In f(g(x)) you can directly access or patch f and g as top level names.

In x.g().f() you have to patch methods internal to other structures, and this can lead to problems if the patching should only happen in a certain local scope.

I encountered this recently with differences between pathlib.Path and os in Python.

Consider

    def my_mkdir(pathname):
        pathlib.Path(pathname).mkdir()
        # or
        os.mkdir(pathname)

From the pov of testing and decomposition, the second option is much nicer, because I can patch os.mkdir directly, and not patch pathlib.Path.mkdir (and also worry about controlling when instance creation happens to use the patch when I need it, but let other possible pathlib.Path objects my tests interacts with be constructed normally). mkdir is even a very simple example since pathlib.Path.mkdir is an instance method but only relies on the string data the instance has. Imagine how much harder if pathlib.Path.mkdir has complex interaction with the internal object structure or other instance methods.

Obviously you _can_ solve it either way, but the fluent interface does nothing except require more code.

On this balance I think list comprehensions (or just fmap, which is all comprehensions are) are much, much better than chaining.

Also if you want to operate on data structures just operate on them with module functions.

I think pandas really messes up on this.

    agg(groupby(df, cols), funcs)

is way better than

    df.groupby(cols).agg(funcs)

A very valuable insight, thank you.

Obviously bad practice can ruin any good thing, but I much prefer list comprehensions for smaller operations and will refactor them as they become unwieldy. If you're in an environment where "nobody really wants to" rewrite stuff that's becoming hard to work with (for whatever reason) it probably doesn't matter which language feature dooms you.

They're great until you find something like this in code:

[x for y in z for x in zz if y in x]

Then you begin to wonder how great they really are.

To me that looks super easy to read. You just go left to right.

    for y in z:
        for x in zz:
            if y in x:
                yield x

It’s extremely easy to sight read. If the lines become long, just split them up and it’s even more obvious.

    [x
     for y in z
     for x in zz
     if y in x]

There’s nothing tricky about multiple loops and conditions in comprehensions in most languages that support them. Just go left to right and it mirrors outer to inner loops/conditionals.

Agree.

Also, most data scientists presumably have some advanced math background so it's akin to reading set-builder notation (albeit with listcomps the sets are sometimes nested). Your listcomp reads:

{x : y ∈ z, x ∈ zz | y ∈ x}

If you've read enough statistics/ML papers, this becomes second nature.

> If you've read enough statistics/ML papers, this becomes second nature.

As someone who breathes that style of notation, there is excellent support in vim and emacs for custom display for readability. Conceal syntax in vim¹(builtin for rust/help/$few_others and external such as vim-cute-python²), and pretty-mode³ in emacs. I'm sure similar things exist in other editors too, but I don't use those ;)

You don't get the exact representation in your example with the things I've mentioned, but if you're used to a more "mathy"-style they really are quite nice. YM[and taste]MV.

1. http://vimdoc.sourceforge.net/htmldoc/syntax.html#conceal

2. https://github.com/ehamberg/vim-cute-python (moresymbols branch for er… more symbols)

3. https://github.com/akatov/pretty-mode

Interesting. I didn't know that!

This is only delightful when you are doing synchronous computation or you have async/await in the language. Until Loom is available, using this same pattern with CompletableFuture breaks in the face of try-with-resources.

Also I don't think it's possible in Python using await since await is a prefix keyword so in your example .groupBy is not a method on Future so it won't work.

I love ReactiveX as much as anyone but these issues cheese me off.

Rust has await as a postfix operator so this pattern works flawlessly when mixing async and sync code.

Yes, as a dev who started in Python and has done more JS recently, I’ve really come to believe that the chaining of in-line functions is a better approach in the end. Python theoretically has the advantage of working with lists or iterators, but in practice, all you really use are lists.

> all you really use are lists.

The more you get experience in Python, and the less you use specifically lists.

In fact, you can spot people getting confortable with the language when they start importing itertools, unless they are data scientists, of course.

To be fair many tools use iterator types that are not lists like Django querysets.

I feel the nice thing in Python is generator expressions, so for lots of code you can decrease the memory footprint and speed some code up through easy lazy evaluation.

I do appreciate the clarity of function chaining, and I think Rust made particularly good decisions around iteration in that way.

Function chaining ideally needs good ways to break iteration, skip and things, and JS isn't very feature rich beyond the basic map, reduce, forEach. It feels very incomplete compared to a lot of languages that offer that stuff. Hence the popularity of underscore and then lodash.

If you’re doing a list comprehension with a Django queryset, you’re risking having a million accidental DB requests

> I'm surprised comprehensions in particular haven't spread to more modern languages.

Comprehensions become unreadable quite quickly - even simple ones have to be read inside-out, and complex (especially nested) ones are much worse.

Most modern languages seem to prefer functional-style methods like `map`, `filter` etc, which are more readable in all but the simplest cases.

> which are more readable in all but the simplest cases.

Except in practice, the vast majority of comprehensions are simple.

It doesn't really help that the Go developers are stuck in the early 80's still.

I hope this succeeds and forces some changes.

I'm here to tell you that comprehension in this form is not everyone's thing. Scala's comprehension is more generic and it's actually monad binding. Comprehension as seen in python is only readable for simple computation, so you end up having to use map reduce sometime. That makes 2 ways to do the same thing and the "zen of python" a lie in this instance.