Does declaring variables in a function called from init still use a key-sharing dictionary?

Question

I've tried to always declare class attributes inside the __init__ for clarity and organizational reasons. Recently, I've learned that strictly following this practice has extra non-aesthetic benefits too thanks to PEP 412 being added for Python 3.3. Specifically, if all attributes are defined in the __init__, then the objects can reduce space by sharing their keys and hashes.

My question is, does object key-sharing happen when attributes are declared in a function that is called by __init__?

Here's an example:

class Dog:
    def __init__(self):
        self.height = 5
        self.weight = 25

class Cat:
    def __init__(self):
        self.set_shape()

    def set_shape(self):
        self.height = 2
        self.weight = 10

In this case, all instances of Dog would share the keys height and weight. Do instances of Cat also share the keys height and weight (among each other, not with Dogs of course).

As an aside, how would you test this?

Note that Brandon Rhodes said this about key-sharing in his Dictionary Even Mightier talk:

If a single key is added that is not in the prototypical set of keys, you loose the key sharing

Not answering your question directly, but PyCharm has inspections which warn if you create new attributes outside of `__init__`. So they, perhaps, consider it a bad practice. — wim, Jun 22 '17 at 17:16

Dimitris Fasarakis Hilliard · Accepted Answer · 2017-07-02T17:26:12.597

6

does object key-sharing happen when attributes are declared in a function that is called by __init__?

Yes, regardless of where you set the attributes from, granted that after initialization both have the same set of keys, instance dictionaries use a shared-key dictionary implementation. Both cases presented have a reduced memory footprint.

You can test this by using sys.getsizeof to grab the size of the instance dictionary and then compare it with a similar dict created from it. dict.__sizeof__'s implementation discriminates based on this to return different sizes:

# on 64bit version of Python 3.6.1
print(sys.getsizeof(vars(c)))
112
print(getsizeof(dict(vars(c))))
240

so, to find out, all you need to do is compare these.

As for your edit:

"If a single key is added that is not in the prototypical set of keys, you loose the key sharing"

Correct, this is one of the two things I've (currently) found that break the shared-key usage:

Using a non-string key in the instance dict. This can only be done in silly ways. (You could do it using vars(inst).update)
The contents of the dictionaries of two instances of the same class deviating, this can be done by altering instance dictionaries. (single key added to that is not in the prototypical set of keys)

I'm not certain if this happens when a single key is added, this is an implementation detail that might change. (addendum: see Martijn's comments)

For a related discussion on this see a Q&A I did here: Why is the __dict__ of instances so small in Python 3?

Both these things will cause CPython to use a 'normal' dictionary instead. This, of course, is an implementation detail that shouldn't be relied upon. You might or might not find it in other implementations of Python and or future versions of CPython.

edited Jul 02 '17 at 17:26

answered Jun 22 '17 at 17:11

Dimitris Fasarakis Hilliard

136,212
29
242
233

You do still have to add the attribute in `__init__` first though, right? – Nathaniel Rivera Saul Jun 22 '17 at 17:16
1

@NathanielSaul if you want to. Point is, it makes no difference where you add them from as long as they have string keys and the dictionary of two instances doesn't deviate much in content, you get a shared-key dict. – Dimitris Fasarakis Hilliard Jun 22 '17 at 17:17
Thanks for the follow up. Sounds like as long as initialization happens somewhere in the `__init__`, key-sharing works. But if it happens after the `__init__`, key-sharing breaks. – Nathaniel Rivera Saul Jun 22 '17 at 17:29
@NathanielSaul it all has to do with the contents for two given instances. After `__init__`, if you don't have any conditional branches that might lead to different attributes getting set, you get a shared-key dict. If you deviate, you'll fall-back to a plain dict (which actually isn't that bad memory wise with `3.6` that gives us a [memory efficient implementation](https://stackoverflow.com/questions/39980323/dictionaries-are-ordered-in-python-3-6/39980744#39980744)) – Dimitris Fasarakis Hilliard Jun 22 '17 at 17:36
1

It will happen with a single key. What happens is: *any mutation* to the set of keys for the dictionary will cause it to be 'combined' (separated from the per-type shared keys). But if that dictionary is the *only* `__dict__` instance namespace, it'll immediately be re-split. So as long as there is only one instance, it doesn't matter how often you mutate the keys (add or remove), nor does the type of key matter. It'll be re-split each time. – Martijn Pieters Jun 22 '17 at 17:50
Correction: it is a **resize** that causes the dict to recombined. Adding new keys can still be fine, until you trigger the need for a larger table. – Martijn Pieters Jun 22 '17 at 18:01
Hah, indeed, adding a non-string key can kill sharing instantly. But only for that one instance. A resize kills it for **all** instances. – Martijn Pieters Jun 22 '17 at 18:21
@MartijnPieters Yes, I also remember finding out that resizing also causes consequent instances to use a combined table. – Dimitris Fasarakis Hilliard Jun 22 '17 at 18:30

Martijn Pieters · Answer 2 · 2019-02-10T07:38:48.833

I think you are referring to the following paragraph of the PEP (in the Split-Table dictionaries section):

When resizing a split dictionary it is converted to a combined table. If resizing is as a result of storing an instance attribute, and there is only instance of a class, then the dictionary will be re-split immediately. Since most OO code will set attributes in the __init__ method, all attributes will be set before a second instance is created and no more resizing will be necessary as all further instance dictionaries will have the correct size.

So a dictionary keys will remain shared, no matter what additions are made, before a second instance can be created. Doing so in __init__ is the most logical method of achieving this.

This does not mean that attributes set at a later time are not shared; they can still be shared between instances; so long as you don't cause any of the dictionaries to be combined. So after you create a second instance, the keys stop being shared only if any of the following happens:

a new attribute causes the dictionary to be resized
a new attribute is not a string attribute (dictionaries are highly optimised for the common all-keys-are-strings case).
an attribute is inserted in a different order; for example a.foo = None is set first, and then second instance b sets b.bar = None first, here b has an incompatible insertion order, as the shared dictionary has foo first.
an attribute is deleted. This kills sharing even for one instance. Don't delete attributes if you care about shared dictionaries.

So the moment you have two instances (and two dictionaries sharing keys), the keys won't be re-split as long as you don't trigger any of the above cases, your instances will continue to share keys.

It also means that delegating setting attributes to a helper method called from __init__ is not going to affect the above scenario, those attributes are still set before a second instance is created. After all __init__ won't be able to return yet before that second method has returned.

In other words, you should not worry too much about where you set your attributes. Setting them in the __init__ method lets you avoid combining scenarios more easily, but any attribute set before a second instance is created is guaranteed to be part of the shared keys.

As for how to test this: look at the memory size with the sys.getsizeof() function; if creating a copy of the __dict__ mapping results in a larger object, the __dict__ table was shared:

import sys

def shared(instance):
    return sys.getsizeof(vars(instance)) < sys.getsizeof(dict(vars(instance)))

A quick demo:

>>> class Foo:
...     pass
...
>>> a, b = Foo(), Foo()  # two instances
>>> shared(a), shared(b)  # they both share the keys
(True, True)
>>> a.bar = 'baz'  # adding a single key
>>> shared(a), shared(b)  # no change, the keys are still shared!
(True, True)
>>> a.spam, a.ham, a.monty, a.eric = (
...     'eggs', 'eggs and spam', 'python',
...     'idle')  # more keys still
>>> shared(a), shared(b)  # no change, the keys are still shared!
(True, True)
>>> a.holy, a.bunny, a.life = (
...     'grail', 'of caerbannog',
...     'of brian')  # more keys, resize time
>>> shared(a), shared(b)  # oops, we killed it
(False, False)

Only when the threshold was reached (for an empty dictionary with 8 spare slots, the resize takes place when you add a 6th key), did the dictionaries loose the shared property.

Dictionaries are resized when they are about 2/3rd full, and a resize generally doubles the table size. So the next resize will take place when the 11th key is added, then at 22, then 43, etc. So for a large instance dictionary, you have a lot more breathing room.

@NathanielSaul: I've clarified my answer. The point is that the 'cut-off point' is creating a second instance, at which point all keys that already exist in the shared table are seen as the 'low water mark'. Any new keys added to this can cause the shared dictionary to be turned into a regular (combined) dictionary, but only if it is resized. — Martijn Pieters, Jun 22 '17 at 17:47
Thinking about the 'cut-off point' like that makes it much clearer. Thank you! — Nathaniel Rivera Saul, Jun 22 '17 at 18:04

Does declaring variables in a function called from __init__ still use a key-sharing dictionary?

2 Answers2

Does declaring variables in a function called from init still use a key-sharing dictionary?