News for hackers. Stuff that matters.
Re:Python's Mutable Default Problem by sophacles on Sunday February 20, 2011 @11:18AM (#2242184)
I'm not sure this constitutes any problem other than a lack of understanding of the python runtime. What the author describes as:

"the mutable default parameter quirk is an ugly corner worth avoiding"

could also be described as:

"a natural outcropping of python's late binding, "names are references" variable model, and closure mechanisms, which provide a consistency to the language that is often crufted up in others"

I do somewhat agree with the author that this particular functionality should be a "use only when needed" feature. I don't think it should be avoided at all costs tho, because there are times where the mutable default allows for a lot of saved code. In fact in a few cases the code to work around using mutable defaults can get into some serious voodoo because frequently the writer is actually trying to work around the bigger mutable/immutable objects and names are references "issues" in python.

This also reminds me of something I was reading on the front page today about the old 'use the whole language' vs 'simplicity is king' holy war.

Re:Python's Mutable Default Problem by statictype on Sunday February 20, 2011 @11:54AM (#2242291)
"a natural outcropping of python's late binding, "names are references" variable model, and closure mechanisms, which provide a consistency to the language that is often crufted up in others"

hm... that's debatable. The implementation could have just as easily chosen to evaluate the default arguments each time the function is invoked and that decision wouldn't have broken any of the existing mental models of variable binding/closures.

Re:Python's Mutable Default Problem by sjs on Sunday February 20, 2011 @11:41AM (#2242259)
Precisely because Python has late binding you would expect the parameters to be evaluated on each call to the function.

One thing Python lacks is the ability to use preceding arguments in defaults, e.g. you cannot do this:

    def f(a=3, b=a+1):
        return (a + b) / 2
    
    NameError: name 'a' is not defined
Oops.
Re:Python's Mutable Default Problem by beza1e1 on Sunday February 20, 2011 @12:59PM (#2242491)
There is no order for named arguments. You could call the function like this after all:

  f(b=3)
Re:Python's Mutable Default Problem by wingo on Sunday February 20, 2011 @02:55PM (#2242823)
Of course; in that case the default value expression for `b' would not be evaluated.

Common Lisp does this right.

Re:Python's Mutable Default Problem by crystalis on Sunday February 20, 2011 @01:34PM (#2242617)
That hides default logic in the signature. Why would you favor that over

  def f(a=3, b=None):
    b = b or a+1 # or use a more explicit version
    return (a + b) / 2
Re:Python's Mutable Default Problem by chc on Sunday February 20, 2011 @04:38PM (#2243178)
That is code is logically convoluted — that's one reason not to love it. Why do you think is the more straightforward statement, just in general?

- "Let B be one greater than A unless otherwise specified."

- "We have no default value for B. If B has a value, then let B be equal to that value. If B does not have a value, then let B be one greater than A."

Re:Python's Mutable Default Problem by sjs on Sunday February 20, 2011 @08:05PM (#2243775)
Because that can be said more succinctly. It is even easier to read as there is less to read, which I realize is mostly subjective.
Re:Python's Mutable Default Problem by crystalis on Sunday February 20, 2011 @09:33PM (#2244009)
I'm pretty sure this is the same argument used to make Perl a bad guy.
Re:Python's Mutable Default Problem by sjs on Monday February 21, 2011 @01:22AM (#2244478)
I disagree, but we've gotten a bit off topic.

I think Python's behaviour is confusing and basically never what anyone actually wants. Regardless of whether or not you can use other params in defaults, the defaults should be evaluated on each call.

Re:Python's Mutable Default Problem by neutronicus on Sunday February 20, 2011 @11:39AM (#2242255)
"a natural outcropping of python's late binding, "names are references" variable model, and closure mechanisms, which provide a consistency to the language that is often crufted up in others"

My mileage varies.

I'd prefer default parameters to honor referential transparency, whatever hoops the runtime has to jump through to make this happen.

Re:Python's Mutable Default Problem by jerf on Sunday February 20, 2011 @11:54AM (#2242288)
That would make that the only place in which Python has referential transparency, though. It may be a quirky side-effect of consistency, but it is consistent.
Re:Python's Mutable Default Problem by eru on Sunday February 20, 2011 @05:21PM (#2243329)
Strings are also non-mutable, and thus have referential transparency. And so do numbers in Python.
Re:Python's Mutable Default Problem by jerf on Monday February 21, 2011 @12:28AM (#2244374)
Referential transparency is a property of functions, not data (unless you're in a lambda mood and treat them as functions of zero arguments, but in that case you're not in Python so it's not relevant here). Even a function to "concatenate two strings" could be passed an object that overloads the addition operator to cause arbitrary modifications:

    >>> class Evil(object):
            def __init__(self):
                self.evil = 1
            def __add__(self, other):
                result = ("%s" % self.evil) + other
                self.evil += 1
                return result


    >>> def referentially_transparent_concat(a, b):
            return a + b

    >>> e = Evil()
    >>> print referentially_transparent_concat(e, "hi")
    1hi
    >>> print referentially_transparent_concat(e, "hi")
    2hi
You can program in a referentially-transparent style with Python, but you'll have to do it by adding your own restrictions to the code you write. Python will not help you with that.
Re:Python's Mutable Default Problem by eru on Friday February 25, 2011 @04:58PM (#2263948)
In your code the devil is in the data-type.
Re:Python's Mutable Default Problem by aston on Sunday February 20, 2011 @11:58AM (#2242303)
As noted in the comments, DON'T use

  stuff = stuff or []
because if you pass an empty list, you'll get a new one rather than mutating the one you passed.

  stuff = stuff if stuff is not None else []
is wordy, but at least it's correct.
Re:Python's Mutable Default Problem by btilly on Sunday February 20, 2011 @12:28PM (#2242379)
I prefer the usual alternative

  if stuff is None:
      stuff = []
Re:Python's Mutable Default Problem by dilap on Sunday February 20, 2011 @04:59PM (#2243258)
It's sad that such a common and concise idiom as "x or y" is so perniciously, subtly broken in Python, and that there is no satisfyingly concise equivalent.

If I were being cavalier and had an extra wish to burn, I'd request

  x else y
to mean x unless x is None.
Re:Python's Mutable Default Problem by eru on Sunday February 20, 2011 @05:18PM (#2243322)
Make yourself a function.
Re:Python's Mutable Default Problem by dilap on Sunday February 20, 2011 @06:20PM (#2243484)
Not so helpful in a strict language (since you want y to be evaluated only when x is None)
Re:Python's Mutable Default Problem by eru on Monday February 21, 2011 @04:28AM (#2244840)
Yes, that's true. You would need to wrap it in a lambda, but then that's looks horrible and you might as well use an if.
Re:Python's Mutable Default Problem by itsnotvalid on Sunday February 20, 2011 @12:53PM (#2242471)
So indeed the auth of the article is really too clever on this.
Re:Python's Mutable Default Problem by xiongchiamiov on Sunday February 20, 2011 @01:20PM (#2242583)
And as usual, being clever in Python equates to getting yourself in trouble.

Dammit, people - in Python, "That's clever" is an insult. Learn the philosophy of a language and you'll be much happier using it.

Re:Python's Mutable Default Problem by code_duck on Sunday February 20, 2011 @03:19PM (#2242899)
That's exactly why I prefer Python over Ruby: not the language, but the philosophy of the community. The Ruby community seems to adore 'clever', while the Python community explicitly shuns it. I view the latter as being more mature and borne of experience.
Re:Python's Mutable Default Problem by chc on Sunday February 20, 2011 @04:25PM (#2243141)
Python itself is pretty "clever" compared to many other languages (GC, lack of obvious 1:1 correspondance between code written and code executed, dynamic typing, significant indentation, decorators, generators, etc.). If Python programmers were really opposed to cleverness, they'd be writing in straightforward assembly or a very thin veneer over it.
Re:Python's Mutable Default Problem by billmcneale on Sunday February 20, 2011 @06:11PM (#2243459)
A language that forces me to say "self" every other word doesn't strike me as particularly clever.

Especially when that keyword is not necessary and when it breaks standard programmer expectations (if I declare a method with 3 parameters, I should call it with 3, not 2).

Re:Python's Mutable Default Problem by xiongchiamiov on Sunday February 20, 2011 @07:10PM (#2243619)
Ah, but you are calling it with 3 parameters:

  foo.bar(baz, sputz)
  ^1      ^2   ^3
When you write a class method, it has more information available to it than a similar subroutine; thus, another parameter.

GvR brings up the deeper reasons for explicit self at http://neopythonic.blogspot.com/2008/10/why-explicit-self-ha... .

Re:Python's Mutable Default Problem by billmcneale on Sunday February 20, 2011 @10:25PM (#2244131)
Yes, this is exactly what I mean by "breaking programmer's expectations". Except maybe Modula, no language works like that at all. The parameters are in the parentheses, period. If you need to pass this, you do so in the declaration and in the invocation. If this is passed implicitly, you don't declare it in the parameters and you don't pass it at the call site.

Python is doing this totally weird stuff that sits in the middle and that makes no logical sense at all.

The fact that Python forces you to declare the "self" parameter is simply due to the fact that it's old, old, old. Nothing wrong with that, but post rationalizing it by saying it's okay to declare a method with 3 parameters but calling with just 2 is just silly.

Re:Python's Mutable Default Problem by xiongchiamiov on Monday February 21, 2011 @04:38AM (#2244866)
Simply because it does not fit your expectations does not mean that it makes "no logical sense at all". As faulty beings, we often have expectations that really are quite far from logical.
Re:Python's Mutable Default Problem by code_duck on Sunday February 20, 2011 @11:17PM (#2244239)
I like 'self'. I don't mind it at all.

The error messages related to `self` in method definitions caused me some confusion when starting out with python. For instance if you define a method with no arguments, calling it results in "TypeError: your_method() takes no arguments (1 given)".

Re:Python's Mutable Default Problem by fleitz on Sunday February 20, 2011 @07:13PM (#2243625)
This is why I love F#.

It lets you define the name of the this parameter in exactly the same way you pass it.

  type Bar(p) =
    member x.Foo(bar) = printf "%s %d" bar p 

  let x = new Bar(10)
  x.Foo("Foo bar")
Output:

  Foo bar 10
And yes, the variables bar and p are type inferred from the %s and %d respectively
Re:Python's Mutable Default Problem by code_duck on Sunday February 20, 2011 @11:11PM (#2244230)
You can rename self to whatever you want in Python by changing it in the function definition.

    class Flip(object):
        foo=4
        def flop(x,num):
            x.bar=x.foo+num
Re:Python's Mutable Default Problem by xiongchiamiov on Monday February 21, 2011 @04:37AM (#2244863)
Right, but his point was that the self variable is declared in front of the method name, similarly to how it looks when you call said method.
Re:Python's Mutable Default Problem by jbri on Sunday February 20, 2011 @05:04PM (#2243273)
Python is "clever" so that the Python programmer doesn't have to be.
Re:Python's Mutable Default Problem by eru on Sunday February 20, 2011 @05:19PM (#2243328)
Actually Python has pretty close correspondence between source code and compiled byte code.
Re:Python's Mutable Default Problem by tswicegood on Sunday February 20, 2011 @02:46PM (#2242799)
You can also do:

    stuff = stuff is not None and stuff or []
That said, the `if stuff is not None:` is the most Pythonic way of doing it.
Re:Python's Mutable Default Problem by wiredfool on Sunday February 20, 2011 @10:37PM (#2244155)
I didn't realize you could write perl in python.
Re:Python's Mutable Default Problem by perlgeek on Sunday February 20, 2011 @12:07PM (#2242331)
FWIW, Perl 6 implicitly treats default values as closures, and calls them when no argument is passed that could bind to the optional argument.

That way you get a fresh array each time, and you can even use defaults that depend on previous arguments:

    sub integrate(&integrand, $from, $to, $step = ($to - $from) / 100) { ... }
Re:Python's Mutable Default Problem by emehrkay on Sunday February 20, 2011 @11:50AM (#2242279)
Pylint (or maybe it was pep8) has told me not to make dicts default arguments when running it against my code, but didn't explain why. Thanks for the post.
Re:Python's Mutable Default Problem by riobard on Sunday February 20, 2011 @01:03PM (#2242510)
You just need to fully understand when Python does evaluation.

    import types
    def function(item, stuff = lambda: []):
        if type(stuff) == types.FunctionType:
            stuff = stuff()
        stuff.append(item)
        print stuff

    function(1)
    # prints '[1]'

    function(2)
    # prints '[2]'

In Scala it has a better syntax because of typed function object:

    trait Map[A, B] {
        …
        def getOrElse (key: A, default: ⇒ B): B
        …
    }

the `default` parameter is a function, so when you do

    getOrElse(someKey, defaultValue)
The `defaultValue` becomes a function that generates the value you put there when called.
Re:Python's Mutable Default Problem by neutronicus on Sunday February 20, 2011 @11:37AM (#2242249)
"stuff = stuff or []"

This idiom is baked into the perl community. It's kind of funny to see a Pythonista deciding it's a good idea. (You can tell it hurts him too).

Re:Python's Mutable Default Problem by jacobolus on Sunday February 20, 2011 @12:00PM (#2242306)
It’s not a good idea in Python. There are legitimate reasons to pass falsy objects as function arguments.
Re:Python's Mutable Default Problem by robinhouston on Sunday February 20, 2011 @01:46PM (#2242645)
Funnily enough, Perl (since version 5.10) actually has a solution to this. The expression

  $foo // "default"
evaluates to "default" if $foo is undefined, and to $foo otherwise (even if it is defined but false).
Re:Python's Mutable Default Problem by dagw on Sunday February 20, 2011 @02:16PM (#2242710)
I've been programming python for on and off 12 years, and full time for the past 3, and this is the first time I've seen anyone recommending that idiom. So on the whole the python community does not think it's a good idea.
Re:Python's Mutable Default Problem by code_duck on Sunday February 20, 2011 @03:21PM (#2242909)
JavaScript, too:

    stuff = stuff||[]
since JavaScript doesn't even support default argument values.
Re:Python's Mutable Default Problem by spenrose on Sunday February 20, 2011 @11:08AM (#2242144)
Yes, it's a wart. You learn the patterns he mentions pretty quickly.
Re:Python's Mutable Default Problem by code_duck on Sunday February 20, 2011 @03:22PM (#2242911)
I knew the pattern well from other languages, but would have assumed that it wasn't necessary in Python. So, this is good to note.
Re:Python's Mutable Default Problem by mark-r on Sunday February 20, 2011 @09:03PM (#2243920)
I keep seeing this "problem" come up, but I don't understand how it's realistic. If you have a function that modifies a parameter as a side effect, why would you have a default value for the parameter?

And since the site's comments seem to be taken over by link spam, is this mention on Hacker News just a clever way to juice the Google rank of said spam?

Re:Python's Mutable Default Problem by ylem on Sunday February 20, 2011 @09:27PM (#2243989)
I saw it bite some people where I work. It's now used for discussion with potential hires.
Re:Python's Mutable Default Problem by clay on Sunday February 20, 2011 @12:13PM (#2242344)
"stuff = stuff or []"

This would fail to have expected behaviour here:

fill_list = []

stuff = function(info, full_list)

print fill_list

use the if list is None: paradigm

Re:Python's Mutable Default Problem by dustingram on Sunday February 20, 2011 @05:39PM (#2243373)
This 'problem' can actually come in handy when used with a regex callback function.

See if you can determine what this does:

    def cbk(match, nb = [0] ):
        if len(match.group())==len(nb):
            nb[-1] += 1
        elif  len(match.group())>len(nb):
            nb.append(1)
        else:
            nb[:] = nb[0:len(match.group())]
            nb[-1] += 1
        return match.group()+' '+('.'.join(map(str,nb)))
    
    str = re.compile('^(#+)',re.MULTILINE).sub(cbk,str)
Re:Python's Mutable Default Problem by d0mine on Sunday February 20, 2011 @09:00PM (#2243912)
The code converts:

  ##
  #
To:

  ## 0.1
  # 1
str is builtin, don't use it as a variable name especially if you use it in its original role.
Re:Python's Mutable Default Problem by dustingram on Monday February 21, 2011 @10:44AM (#2245766)
Correct, & thanks for the tip! But don't worry, I changed the variable name from my copy & paste and just didn't give it a second thought.
Re:Python's Mutable Default Problem by billmcneale on Sunday February 20, 2011 @06:10PM (#2243452)
The problem is not so much about mutable, it's that default parameters escape the scope of their method.

Which is very, very messed up (but not the first thing Python messed up).

Re:Python's Mutable Default Problem by wyuenho on Sunday February 20, 2011 @12:38PM (#2242407)
It's on SO's Python FAQ too. This is the only thing in Python that's really bitten me. I remember it took me like a week to figure this out when I was tearing down my algorithm bit by bit to find out whether my prove was wrong or the code.

http://stackoverflow.com/questions/1132941/least-astonishmen...

Re:Python's Mutable Default Problem by pedro3005 on Sunday February 20, 2011 @01:13PM (#2242558)
You can use this trick for memoization. Example: http://paste.pocoo.org/show/341849/
Re:Python's Mutable Default Problem by code_duck on Sunday February 20, 2011 @03:23PM (#2242914)
The article is right, though, in stating that later viewers of your code may not be expecting this behavior, and it culd lead to problems.
Re:Python's Mutable Default Problem by underwater on Sunday February 20, 2011 @09:18PM (#2243969)
Memorization is an implementation detail and shouldn't be part of the arguments list.
Re:Python's Mutable Default Problem by viraptor on Sunday February 20, 2011 @12:50PM (#2242458)
Not sure if that's a "problem" if the only other way to implement function-static variables would be to add a variable visible for the whole module, or tricks with decorators...

    @statics(blah=[])
    def foo(normal_args, **kwargs):
    # or
    def foo(normal_args, blah):
It messes up the idea of looking at the definition to find the function signature.
Re:Python's Mutable Default Problem by baq on Sunday February 20, 2011 @11:54AM (#2242286)
it's a gotcha but it makes perfect sense once you understand why it works that way - i.e. the difference between evaluating a function definition and calling it.
Re:Python's Mutable Default Problem by jfm3 on Sunday February 20, 2011 @02:57PM (#2242825)
The Pythonist doth protest too much, methinks.
Re:Python's Mutable Default Problem by drstrangevibes on Sunday February 20, 2011 @03:12PM (#2242877)
i think its clearer and more pythonic in this case to do

def function(item, stuff): .... blah blah def function(item): function(item, [])