News for hackers. Stuff that matters.
Re:Specific ways to write better Python (2017) by Groxx on Friday June 26, 2020 @06:00PM (#23657555)
Isn't that just the opposite tradeoff for code you call though?

E.g. lists are strictly more flexible than iterators since they are multi-pass and can index. If you take a return from lib func A and pass it to lib funcs B and C (possibly in a different lib), if A returns an iterator then you need to know what B and C are going to do with it. If A returns a list, you do not - B and C cannot affect each other in this way unless they directly mutate the argument, which is relatively rare and usually has very strong documentation and/or naming patterns to make it not surprising. And B and C and all their calls can change without you needing to change your code between them and A.

You can of course `list(A())` before passing it to B and C, but that's arguably defensive programming unless you know A returns an iterator.

tbh I'd say I see multi-iteration and indexing several times more often than I see iterator-only use in most code, and almost never see memory issues except in large or embedded systems. Gigabytes of RAM are standard now, and dumping a few thousand files into memory before processing will quite often out-perform a more memory-efficient streaming approach. In the rare cases where you do exceed those memory bounds, you're likely in a niche area (gigabytes of files, tiny memory space, significant computation cost) and are likely using niche-oriented libraries that benefit more from coordinating tightly than from being perfectly general.

Re:Specific ways to write better Python (2017) by xapata on Friday June 26, 2020 @06:38PM (#23657847)
> never see memory issues

In contrast, I see them all the time (damn you, Pandas). Beyond out-of-memory errors, generators can be vastly more compute-efficient because of the CPU cache.

Re:Specific ways to write better Python (2017) by Groxx on Friday June 26, 2020 @10:00PM (#23658915)
Pandas I'd definitely categorize as being niche :) A moderately-sized one, but still a niche. And there you use whatever performs the way you need it, yeah - there are special constraints on the system, and pandas is built around dealing with them.
Re:Specific ways to write better Python (2017) by hansvm on Friday June 26, 2020 @10:19PM (#23658997)
Haha, that one hits close to home. Iirc Pandas was the author's first big project, and memory use is one of the key things he wants to improve if he gives it another go. It's still an impressive piece of software.