python can't access nonlocal variable before local variable is defined with same name

Question

Possible Duplicate:
Python nested functions variable scoping

I've used decorators before and so I was surprised to find a bug in my code:

def make_handler(name, panels):
    def get(self):
        admin = True
        keys = [ndb.Key('Panel', panel) for panel in panels]
        panels = zip(ndb.get_multi(keys), panels)
        panels = [(panel.panel_html if panel else get_default_content(panel_id), panel_id) for panel, panel_id in panels]
        templates = {'panels': panels, 'admin': admin}
        self.render_template('panel_page.html', **templates)
    return type(name, (BaseHandler,), {'get': get})

"""
Traceback (most recent call last):
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1536, in __call__
    rv = self.handle_exception(request, response, e)
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1530, in __call__
    rv = self.router.dispatch(request, response)
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "C:\Users\Robert\PycharmProjects\balmoral_doctors\main.py", line 35, in get
    keys = [ndb.Key('Panel', panel) for panel in panels]
UnboundLocalError: local variable 'panels' referenced before assignment
"""

My fix is to change panel to panel2 beyond the first usage:

def make_handler(name, panels):
    def get(self):
        admin = True
        keys = [ndb.Key('Panel', panel) for panel in panels]
        panels2 = zip(ndb.get_multi(keys), panels)
        panels2 = [(panel.panel_html if panel else get_default_content(panel_id), panel_id) for panel, panel_id in panels2]
        templates = {'panels': panels2, 'admin': admin}
        self.render_template('panel_page.html', **templates)
    return type(name, (BaseHandler,), {'get': get})

Now everything works fine and I'm wondering why.

This is what I guess happens but I don't know:

panels = zip(..)

means that panels is a local variable. That means the function doesn't look in the outerscope for panels.

This is done before the get() function is run, and not midway through?

I thought it would firstly grab panels from the outer function, and then when panels gets defined in the inner function after that, it would from then on use the new local panels variable.

Am I on the right track?

wim · Answer 1 · 2020-01-03T03:15:45.183

You are more-or-less correct, and you found the correct resolution. Your problem is equivalent to this:

bars = range(10)

def foo():
    thing = [x for x in bars]
    bars = 'hello'

foo()
# UnboundLocalError: local variable 'bars' referenced before assignment

At function definition time, it is determined that bars is local scope. And then at function run time, you get the problem that bars hasn't been assigned.

Chris Morgan · Accepted Answer · 2012-11-08T05:41:27.273

Yes.

Python's scoping rules indicate that a function defines a new scope level, and a name is bound to a value in only one scope level in a scope level—it is statically scoped (i.e. all scoping is determined at compilation time). As you understood, you're trying to violate that by reading from a non-local declaration and writing to a local variable. As you observe, the interpreter objects violently to this by raising an UnboundLocalError: it has understood that panels is a local variable (because it can't be that and non-local at the same time), but you haven't assigned (bound) a value to the name, and so it fails.

More technical details

The decision was made in Python to keep track of where variables are at compilation time in the bytecode (to be specific for this case, it's in a tuple get.__code__.co_varnames for local variables), meaning that a variable can be only used in a single scope level in a certain scope. In Python 2.x, it is not possible to modify a non-local variable; you have either read-only access to a global or non-local variable, or read-write access to a global variable by using the global statement, or read-write access to a local variable (the default). That's just the way it's been designed (probably for performance and purity). In Python 3, the nonlocal statement has been introduced with a similar effect to global, but for an intermediate scope.

Binding the modified variable to a different name is the correct solution in this case.

score 1 · Answer 3 · answered Nov 08 '12 at 05:38

Many people don't realise this, but Python is actually statically scoped. When Python sees a read from a bare name (i.e. not an attribute of some object), it can determine precisely where that name read will go to purely from compile-time analysis.

If a name is ever assigned to in a function, then that name is a local variable in that function¹, and it's scope extends over the entirety of the function's body, even over the lines before the assignment.

If the name is not assigned to in a function, then the name is a non-local variable. Python can check the static scope of any surrounding def blocks to see if it's a local variable in any of those. If not, then that name must be a reference to a module global or a built-in (the choice between global and built-in is resolved dynamically, so you can shadow a built-in with a global that is declared dynamically).

I believe this is done mainly for efficiently. It means that the set of local variables of a function can be known when the bytecode for the function is compiled, and local variable access can be converted into simple index operations. Otherwise Python would have to perform dictionary lookups to access local variables, which would be slower.

So because your get function contains a line of the form panels = ..., then panels is a local variable in the entirety of the body of get. The assignment to keys is looping over the local variable panels before it has been assigned.

¹ Unless that name is declared global or nonlocal, but that's still statically known.

score 0 · Answer 4 · edited May 23 '17 at 11:44

If a variable is assigned within a function it is assumed to be referring to the local name, unless you explicitly declare it global or, in Python 3.x, nonlocal. If declared global, the variable must be defined in the module's globals, not covering the above case. You've found one Python 2.x solution; another may be to add panels as an argument to get and use functools.partial:

def make_handler(name, panels):
    def get(self, panels):
        admin = True
        keys = [ndb.Key('Panel', panel) for panel in panels]
        panels = zip(ndb.get_multi(keys), panels)
        panels = [(panel.panel_html if panel else get_default_content(panel_id), panel_id) for panel, panel_id in panels]
        templates = {'panels': panels, 'admin': admin}
        self.render_template('panel_page.html', **templates)
    return type(name, (BaseHandler,), {'get': functools.partial(get, panels=panels)})

See also: Closures in Python, Python nonlocal statement

redman · Answer 5 · 2012-11-08T06:03:28.687

0

Python compilers first scan the whole code inside a function to determin the set of local variables (passed as prameters or assigned a new val inside the function). That is why you can't use the variable assigned a new value inside the function before the assignment.

If the compilers didn't do that then it would have been imposible to create a closure for the function. Closures need the have a reference to all free or non local variables. If a var is free for part of the function and local for the other part....it wouldn't make sense :)

edited Nov 08 '12 at 06:03

answered Nov 08 '12 at 05:53

redman

173
2
9

"it would have been impossible to create a closure for the function": that's not true. Consider this alternative model, which is what is done in JavaScript engines*: just keep track of the stack of scopes, and then, for each variable reference, look up a variable in each scope in turn. It'll typically be slower, but that's the trade-off. (* Or at least used to be; I'm not sure how true it is any more - but you can have a reference to a non-local variable which is then overridden by a local variable later, which you can't do in Python.) – Chris Morgan Nov 08 '12 at 06:24
Ok. I guess i don't understand what a closure is then. Doesn't a closure need to be defined before it is used? What if an assignement happens inside an if statement? I get your point though. – redman Nov 08 '12 at 06:33
I guess i just agree with how python does it and i disagree with how javsscript does it :) – redman Nov 08 '12 at 06:46

python can't access nonlocal variable before local variable is defined with same name

5 Answers5

More technical details