6

I have a wrapper class similar to this (strongly simplified) example:

class wrap(object):
    def __init__(self):
        self._data = range(10)

    def __getitem__(self, key):
        return self._data.__getitem__(key)

I can use it like this:

w = wrap()
print w[2] # yields "2"

I thought I could optimize and get rid of one function call by changing to this:

class wrap(object):
    def __init__(self):
        self._data = range(10)
        self.__getitem__ = self._data.__getitem__

However, I receive a

TypeError: 'wrap' object does not support indexing

for the print w[2] line with the latter version.

The direct call to the method, i.e., print w.__getitem__(2), works in both cases...

Why does the assignment version not allow indexing?

NichtJens
  • 1,421
  • 15
  • 25
  • Pro tip: If you didn't know why this breaks, you shouldn't try optimizing it. Its overhead is probably not as significant as you think it is. – MisterMiyagi Jun 30 '16 at 21:07
  • 1
    But, If I hadn't tried, I wouldn't have learned how it works, which wouldn't have allowed me to do it. OTOH, if I had known it breaks, I'd not have tried it ... Also, you probably don't know, how significant I thought it to be. – NichtJens Jun 30 '16 at 21:26
  • Note that you *can* actually solve this by creating a new class for every type. Basically you need a factory function for this. Which can be transparently implemented via `__new__`. It's some black magic, though... Of course I can just speculate how significant you think it its; for reference, you can safe roughly 0.1 usec per call. – MisterMiyagi Jun 30 '16 at 21:34
  • 1
    Care to write down some example code for that? And, please excuse my snarky comment, I meant to demonstrate that you are implying that learning how something works by breaking it is not a good strategy (which I think it is!) ... – NichtJens Jun 30 '16 at 21:46
  • 1
    To be quite honest with you, I didn't know this was possible either, before your snark motivated me. See my answer below on how to do it (and why it's actually a bad idea for other reasons ^^). Keep up the good snark and never stop breaking things. ;) – MisterMiyagi Jun 30 '16 at 21:54

2 Answers2

5

Special methods (essentially anything with two underscores on each end) have to be defined on the class. The internal lookup procedure for special methods completely skips the instance dict. Among other things, this is so if you do

class Foo(object):
    def __repr__(self):
        return 'Foo()'

the __repr__ method you defined is only used for instances of Foo, and not for repr(Foo).

NichtJens
  • 1,421
  • 15
  • 25
user2357112
  • 235,058
  • 25
  • 372
  • 444
  • Ah! I see. Could you give a link to the documentation for this? And, I don't really see the connection between your first paragraph (which answers my question) and your example... – NichtJens Jun 30 '16 at 21:09
  • 1
    @NichtJens: If special method lookup looked at the instance dict, then `repr(Foo)` would pick up the `Foo.__repr__` method you defined and use that. That's not what you want, because that method is only supposed to handle instances of `Foo`, not `Foo` itself. – user2357112 Jun 30 '16 at 21:10
  • @NichtJens: [Here's](https://docs.python.org/3/reference/datamodel.html#special-lookup) the documentation, which uses a similar example. – user2357112 Jun 30 '16 at 21:12
  • Great... Solved in ~5minutes. Thanks! – NichtJens Jun 30 '16 at 21:16
1

You can actually solve this by creating a new class for every type. If you want this to work transparently, __new__ is the place for it.

import weakref


class BigWrap(object):
    def __new__(cls, wrapped):
        wrapped_type = type(wrapped)
        print('Wrapping %s (%s)' % (wrapped, wrapped_type))
        # creates a new class, aka a new type
        wrapper_class = type(  # new_class = type(class name, base classes, class dict)
            '%s_%s_%d' % (cls.__name__, wrapped_type.__name__, id(wrapped)),  # dynamic class name
            (
                cls,  # inherit from wrap to have all new methods
                wrapped_type,  # inherit from wrap_type to have all its old methods
            ),
            {
                '__getitem__': wrapped.__getitem__,  # overwrite __getitem__ based on wrapped *instance*
                '__new__': wrapped_type.__new__,  # need to use wrapped_type.__new__ as cls.__new__ is this function
            })
        cls._wrappers[wrapped_type] = wrapper_class  # store wrapper for repeated use
        return cls._wrappers[wrapped_type](wrapped)

    # self is already an instance of wrap_<type(wrapped)>
    def __init__(self, wrapped):
        self.__wrapped__ = wrapped

Initial "solution":

import weakref
class wrap(object):
  _wrappers = weakref.WeakValueDictionary()  # cache wrapper classes so we don't recreate them

  def __new__(cls, wrapped):
    wrapped_type = type(wrapped)
    print('Wrapping %s (%s)' % (wrapped, wrapped_type))
    try:
      return object.__new__(cls._wrappers[wrapped_type])  # need to use object.__new__ as cls.__new__ is this function
    except KeyError:
      print('Creating Wrapper %s (%s)' % (wrapped, wrapped_type))
    # creates a new class, aka a new type
    wrapper_class = type(  # class name, base classes, class dict
      '%s_%s' % (cls.__name__, wrapped_type.__name__),  # dynamic class name
      (cls,),  # inherit from wrap to have all its method
      {'__getitem__': wrapped_type.__getitem__})  # overwrite __getitem__ based on wrapped class
    cls._wrappers[wrapped_type] = wrapper_class  # store wrapper for repeated use
    return cls._wrappers[wrapped_type](wrapped)

  # self is already an instance of wrap_<type(wrapped)>
  def __init__(self, wrapped):
    self._data = wrapped

Be careful however! This will do what you want - use the wrapped class' __getitem__. However, this doesn't always make sense! For example, list.__getitem__ is actually built into CPython's CAPI and not applicable to other types.

foo = wrap([1,2,3])
print(type(foo))  # __main__.wrap_list
foo[2]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-31-82791be7104b> in <module>()
----> 1 foo[2]

TypeError: descriptor '__getitem__' for 'list' objects doesn't apply to 'wrap_list' object
MisterMiyagi
  • 36,972
  • 7
  • 82
  • 99
  • 1
    I see. So, even though we can trick the "special lookup" procedure into actually considering the wrapped's `__getitem__`, the trick does not work consistently. That's a little bit of a bummer... Apart from that, I feel your example might be re-written as a decorator, taking the to-be-wrapped methods as argument. But, because of the inconsistencies above, it's probably not worth it. – NichtJens Jul 05 '16 at 03:28
  • @NichtJens Yeah, at least directly using the methods seems not to be worth it. The same approach could be used to dynamically change the inheritance of the wrapper - i.e. make it an actual subclass of the wrapped class. – MisterMiyagi Jul 05 '16 at 06:59