Re:Python Practices for Efficient Code: Performance, Memory,...
by fnl on Sunday August 27, 2017 @05:01AM
(#15109827)
So the conclusion is: Have an existing list to join? Use str.join(list); Otherwise, operator+ might be your friend, but f-strings are probably cleaner to handle (but you only get those in Py3.6+). As to str.format(): Nah, not good.
Didn't expect str.format() to turn out to be "so" poor, though.
But this conclusion is taken w/o looking into the "old" formatting approach, `"%s%s%s%s" % (a, b, c, d)`, though.
Re:Python Practices for Efficient Code: Performance, Memory,...
by dom0 on Friday August 18, 2017 @05:19PM
(#15049804)
The reason join is fast for these cases is because join basically converts all iterables to a list first and figures out how much it has to join exactly.
Re:Python Practices for Efficient Code: Performance, Memory,...
by dr_zoidberg on Monday August 21, 2017 @12:49PM
(#15065836)
It walks the iterables (no need to convert them to lists), but its main trick is actually knowing how much memory it'll need for the end result, so no re-allocations happen, simply a one pass (through multiple iterables) copy.
Re:Python Practices for Efficient Code: Performance, Memory,...
by dom0 on Monday August 21, 2017 @04:54PM
(#15067789)
No, it really converts all iterables to lists (there is a utility function in the C API to do just that, because it's kinda hairy to do by hand. PySequence_ToList or so). Otherwise it would not be possible to calculate how much space is required beforehand, since iterables cannot be walked twice.
Timings are now closer, but the f-string is still a bit faster.
Edit 2: (final edit?) still for string concatenation, "".join(...) is the king:
Notice that in [19] %timeit is also measuring the time to build the list in the first place, which is why I also tested with a prebuilt list in [21]. Building the list in the first place is a cost you have to pay, so it's not entirely free.Still, all this optimizations make more sense for really long strings and fragments (substrings that you want to join). Extreme case of 256 1-long strings: