Skip to content

Dynamic Attributes and Propertiesđź”—

  • Data Attributes and methods are collectively called as attributes in python. Methods are just callable attributes.
  • Dynamic attributes present same interface as data attributes but are computed on demand following Bertrand Meyer’s Uniform Access Principle.

Data Wrangling with Dynamic Attributesđź”—

  • given following JSON Record
{ "Schedule":
  { "conferences": [{"serial": 115 }],
    "events": [
      { "serial": 34505,
        "name": "Why Schools Don´t Use Open Source to Teach Programming",
        "venue_serial": 1462,
        "description": "Aside from the fact that high school programming...",
        "speakers": [157509],
        "categories": ["Education"] }
    ],
    "speakers": [
      { "serial": 157509,
        "name": "Robert Lefkowitz"
      }
    ],
    "venues": [
      { "serial": 1462,
        "name": "F151",
        "category": "Conference Venues" }
    ]
  }
}

import json
with open('data/osconfedd.json') as fp:
  feed = json.load(fp)

sorted(feed['Schedule'].keys())
# Output : ['conferences', 'events', 'speakers', 'venues']
feed['Schedule']['speakers'][-1]['name']    # syntax is bad
# Output : 'Carina C. Zona'
  • above syntax is cumbersome. In JS, we could get the data using feed.Schedule.events[40].name. It’s easy to implement a dict like class that does similar in python
  • There are many ways to do this, with our FrozenSet its quite simple
import json
raw_feed = json.load(open('data/osconfeed.json'))
feed = FrozenJSON(raw_feed)
len(feed.Schedule.speakers)

from collections import abc

class FrozenJSON:
    """A read-only façade for navigating a JSON-like object
       using attribute notation
    """

    def __init__(self, mapping):
        self.__data = dict(mapping) # make dict from mapping arg

    def __getattr__(self, name): # its called only when there's no attribute with name
        try:
            return getattr(self.__data, name) # nested search
        except AttributeError:
            return FrozenJSON.build(self.__data[name]) # raise error

    def __dir__(self): # supports dir() built-in, supporting auto-completion
        return self.__data.keys()

    @classmethod
    def build(cls, obj): # alternate constructor
        if isinstance(obj, abc.Mapping): # if obj is mapping build frozenjson
            return cls(obj) # nested jsons
        elif isinstance(obj, abc.MutableSequence): # it must be a list
            return [cls.build(item) for item in obj]    # recursively pass each item to obj recursively
        else: # not a dict or a list return item
            return obj
  • NOTE: FrozenJSON instance has __data private instance attributes stored under _FrozenJSON__data. Attempts to access this triggers __getattr__

The Invalid Attribute Name Problemđź”—

FrozenJSON doesn’t support Python keywords as attributes name.

student = FrozenJSON({'name': 'Jim Bo', 'class': 1982})
student.class # invalid syntax error

# but this words
getattr(student, 'class')
  • A solution is to check whether key in the mapping given to FrozenJSON.__init__ is a keyword, and if so, append _ to it, so the attribute can be read like student.class_
    def __init__(self, mapping):
        self.__data = {}
        for key, value in mapping.items():
            if keyword.iskeyword(key):
                key += '_'
            self.__data[key] = value

Flexible Object Creation with __new__đź”—

  • __init__ is referred as constructor because of adopted jargon from other languages. NOTE: how __init__ is called with self meaning it has already been initialised. Also __init__ cannot return anything, so its really initialiser, not a constructor.
  • python actually calls __new__ to create a new instance. Its a class method but gets special treatment, so the @classmethod decorator is not applied to it. Python takes the instance returns by __new__ and then passes it as the first argument of self of __init__.
  • If necessary, the __new__ method can also return an instance of a different class when that happends interpretor doesn’t call __init__
# pseudocode for object construction
def make(the_class, some_arg):
    new_object = the_class.__new__(some_arg)
    if isinstance(new_object, the_class):
        the_class.__init__(new_object, some_arg)
    return new_object

# the following statements are roughly equivalent
x = Foo('bar')
x = make(Foo, 'bar')

from collections import abc
import keyword

class FrozenJSON:
    """A read-only façade for navigating a JSON-like object
       using attribute notation
    """

    def __new__(cls, arg): # passing class here
        if isinstance(arg, abc.Mapping):
            return super().__new__(cls) # default behaviour is to delegate to __new__ of a superclass
        elif isinstance(arg, abc.MutableSequence):
            return [cls(item) for item in arg]
        else:
            return arg

    def __init__(self, mapping):
        self.__data = {}
        for key, value in mapping.items():
            if keyword.iskeyword(key):
                key += '_'
            self.__data[key] = value

    def __getattr__(self, name):
        try:
            return getattr(self.__data, name)
        except AttributeError:
            return FrozenJSON(self.__data[name])

    def __dir__(self):
        return self.__data.keys()

Computed Propertiesđź”—

We have used @property decorator used to make attributes read-only.

{ "serial": 33950,  # here this is linked to venue and speakers db
  "name": "There *Will* Be Bugs",
  "event_type": "40-minute conference session",
  "time_start": "2014-07-23 14:30:00",
  "time_stop": "2014-07-23 15:10:00",
  "venue_serial": 1449,
  "description": "If you're pushing the envelope of programming...",
  "website_url": "http://oscon.com/oscon2014/public/schedule/detail/33950",
  "speakers": [3471, 5199],
  "categories": ["Python"] }
  • We will implement a Event class with venue and speaker properties to return the linked data automatically. (dereferencing the serial number)

1. Data Driven Attribute Creationsđź”—

  • fields from the original JSON can be retrieved as Record instance attributes
import json

JSON_PATH = 'data/osconfeed.json'

class Record:
    def __init__(self, **kwargs):
        self.__dict__.update(kwargs) # shortcut to build an instance with attributes created from keyword arguments

    def __repr__(self):
        return f'<{self.__class__.__name__} serial={self.serial!r}>' # using serial field to build custom Record Representation

def load(path=JSON_PATH):
    records = {}
    with open(path) as fp:
        raw_data = json.load(fp) # parse json
    for collection, raw_records in raw_data['Schedule'].items():
        record_type = collection[:-1] # list name without last char (remove s from suffixes)
        for raw_record in raw_records:
            key = f'{record_type}.{raw_record["serial"]}' # Events.some_val
            records[key] = Record(**raw_record) # create record objects
    return records
  • This is much better than previous recursive FrozenJSON implementation.

2. Property to Retrieve Linked Recordđź”—

  • The goal of this next version is: given an event record, reading its venue property will return a Record
event = Record.fetch('event.33950') # instance of Event class
event   # There will be Bugs
event.venue # <Record serial=1449> # returns a Record instance
event.venue.name # Portland 251
  • Event is a subclass of Record adding a venue to retrieve linked records, and a specialized __repr__ method.
# Record Class
import inspect # used for load
import json

JSON_PATH = 'data/osconfeed.json'

class Record:

    __index = None # private class attribute eventually holding reference to dict returned by load

    def __init__(self, **kwargs):
        self.__dict__.update(kwargs)

    def __repr__(self):
        return f'<{self.__class__.__name__} serial={self.serial!r}>'

    @staticmethod # static method
    def fetch(key):
        if Record.__index is None: # populate the Record.__index if necessary
            Record.__index = load()
        return Record.__index[key]

 # Event Class
class Event(Record): # Event extends Record

    def __repr__(self):
        try:
            return f'<{self.__class__.__name__} {self.name!r}>' # if instance has a name attribute it is used to produce a custom representation or else delegate to `__repr__` from Record
        except AttributeError:
            return super().__repr__()

    @property
    def venue(self):
        key = f'venue.{self.venue_serial}'
        return self.__class__.fetch(key) # build a key from venue.serial_num

# load function
def load(path=JSON_PATH):
    records = {}
    with open(path) as fp:
        raw_data = json.load(fp)
    for collection, raw_records in raw_data['Schedule'].items():
        record_type = collection[:-1]
        cls_name = record_type.capitalize()
        cls = globals().get(cls_name, Record) # search for class based on key name like Event
        if inspect.isclass(cls) and issubclass(cls, Record): # check if subclass of Record
            factory = cls # bind factory name to new class
        else:
            factory = Record # or bind to Record
        for raw_record in raw_records:
            key = f'{record_type}.{raw_record["serial"]}'
            records[key] = factory(**raw_record) # store the record and store in records which is contructed by factory
    return records

3. Property Overriding an Existing Attributeđź”—

    @property
    def speakers(self):
        spkr_serials = self.__dict__['speakers'] # directly retrieve from __dict__
        fetch = self.__class__.fetch
        return [fetch(f'speaker.{key}')
                for key in spkr_serials] # return list of all records with key corresponding to the numbers in spkr_serials

4. Bespoke Property Cacheđź”—

  • Caching property is a common need because there is an expectation that expression like event.venue should be inexpensive.
  • The handmade caching is straightforward, but creating an attribute after the instance is initialized defeats the PEP 412—Key-Sharing Dictionary optimization, as explained in “Practical Consequences of How dict Works”. Depending on the size of the dataset, the difference in memory usage may be important.
class Event(Record):

    def __init__(self, **kwargs):
        self.__speaker_objs = None  # code this to quikly check if speakers are there or not
        super().__init__(**kwargs)

# 15 lines omitted...
    @property
    def speakers(self):
        if self.__speaker_objs is None:
            spkr_serials = self.__dict__['speakers']
            fetch = self.__class__.fetch
            self.__speaker_objs = [fetch(f'speaker.{key}')
                    for key in spkr_serials]
        return self.__speaker_objs

5. Caching using functoolsđź”—

    @cached_property
    def venue(self):
        key = f'venue.{self.venue_serial}'
        return self.__class__.fetch(key)
  • The @cached_property decorator does not create a full-fledged property, it creates a nonoverriding descriptor. A descriptor is an object that manages the access to an attribute in another class.
  • differences between cached_property and property from a user’s point of view.
    • The mechanics of cached_property() are somewhat different from property(). A regular property blocks attribute writes unless a setter is defined. In contrast, a cached_property allows writes.
    • The cached_property decorator only runs on lookups and only when an attribute of the same name doesn’t exist. When it does run, the cached_property writes to the attribute with the same name. Subsequent attribute reads and writes take precedence over the cached_property method and it works like a normal attribute.
    • The cached value can be cleared by deleting the attribute. This allows the cached_property method to run again

@cached_property has some important limitations:

  • It cannot be used as a drop-in replacement to @property if the decorated method already depends on an instance attribute with the same name.
  • It cannot be used in a class that defines __slots__.
  • It defeats the key-sharing optimization of the instance __dict__, because it creates an instance attribute after __init__.

Despite this @cached_property addresses a common need in a simple way and it is thread-safe. Alternate solution that we can use with speaker is to stack @property and @cache decorators

    @property # note this order is important speaker= property(cache(speaker))
    @cache
    def speakers(self):
        spkr_serials = self.__dict__['speakers']
        fetch = self.__class__.fetch
        return [fetch(f'speaker.{key}')
                for key in spkr_serials]

Using a Property for Attribute Validationđź”—

  • Besides computing attribute values, properties are also used to enforce business rules by changing a public attribute into a protected attribute by getter and setter.

LineItem Take #1: Class for an Item in an Orderđź”—

class LineItem:
  def __init__(self, desc, weight, price):
    self.description = desc
    self.weight = weight
    self.price = price

  def subtotal(self):
    return self.weight * self.price

raisins = LineItem('Golden raisins', 10, 6.95)
raisins.subtotal()
# Output : 69.5
raisins.weight = -20  # garbage in...
raisins.subtotal()    # garbage out...
# Output : -139.0

LineItem Take #2: A Validating Propertyđź”—

class LineItem:

    def __init__(self, description, weight, price):
        self.description = description
        self.weight = weight
        self.price = price

    def subtotal(self):
        return self.weight * self.price

    @property  # decorate getter
    def weight(self):
        return self.__weight # actual value stored in __weight

    @weight.setter # setter property
    def weight(self, value):
        if value > 0:
            self.__weight = value # only update property when val > 0
        else:
            raise ValueError('value must be > 0')

A proper look at Propertiesđź”—

  • Although often used as a decorator, the property built-in is actually a class.
  • In Python, functions and classes are often interchangeable, because both are callable and there is no new operator for object instantiation, so invoking a constructor is no different from invoking a factory function. And both can be used as decorators, as long as they return a new callable that is a suitable replacement of the decorated callable.
# property signature
property(fget=None, fset=None, fdel=None, doc=None)
  • A much simpler approach compared with decorator is creating set_weight and get_weight functions and then defining class attribute weight = property(get_weight, set_weight)

Properties Override Instance Attributesđź”—

  • Properties are always class attributes, but they actually manage attribute access in the instance of the class.
  • Instance attributes shadow same named class attributes.
class Class:
  data = 'the class data attr'

  @property
  def prop(self):
    return 'the prop value'
obj = Class()
vars(obj)   # {}
obj.data    # 'the class data attr'
obj.data = 'bar' # shadow by instance attributes
vars(obj)   # {'data': 'bar'}
obj.data    # Output : 'bar'
Class.data  # 'the class data attr'

# let's try to shadow class property
New class property shadows the existing instance attribute

Class.prop  # <property object ...>
obj.prop    # 'the prop value'
obj.prop = 'foo'    # AttributeError becuase directly executes property getter
# lets modify the object data dict
obj.__dict__['prop'] = 'foo'
vars(obj)   # {'data': 'bar', 'prop':'foo'} # this prop is instance attribute
obj.prop    # 'the prop value'  # this still runs property getter, so its not shadowed
Class.prop = 'baz'  # class property destroyed, now its class attribute
obj.prop    # 'foo' # shadow now works, obj.prop retrieves the instance attribute

# New class property shadows the existing instance attribute

obj.data    # 'bar'
Class.data  # the class data attr
Class.data = property(lambda self: 'the data prop value')
obj.data    # property is called, instance attribute is shadowed
del Class.data
obj.data    # 'bar'
  • The main point of this section is that an expression like obj.data does not start the search for data in obj. The search actually starts at obj.__class__, and only if there is no property named data in the class, Python looks in the obj instance itself.

Property Documentationđź”—

When tools such as the console help() function or IDEs need to display the documentation of a property, they extract the information from the __doc__ attribute of the property.

If used with the classic call syntax, property can get the documentation string as the doc argument:

    weight = property(get_weight, set_weight, doc='weight in kilograms')

# Documentation Example
class Foo:

    @property
    def bar(self):
        """The bar attribute"""
        return self.__dict__['bar']

    @bar.setter
    def bar(self, value):
        self.__dict__['bar'] = value

Coding a Property Factoryđź”—

We will create a factory to create quantity properties - so named because the managed attributes represent quantities that can’t be negative or zero in the application.

class LineItem:
    weight = quantity('weight') # factory calls
    price = quantity('price')

    def __init__(self, description, weight, price):
        self.description = description
        self.weight = weight    # here the property works automatically
        self.price = price

    def subtotal(self):
        return self.weight * self.price

def quantity(storage_name): # where data for each property is stored like 'weight' or 'price'

    def qty_getter(instance): # getter
        return instance.__dict__[storage_name] # return the property

    def qty_setter(instance, value):
        if value > 0:
            instance.__dict__[storage_name] = value
        else:
            raise ValueError('value must be > 0')

    return property(qty_getter, qty_setter) # return the property

Handling Attribute Deletionđź”—

  • We can not only delete variables but attributes as well using del
  • In practice, deleting attributes is not something we do every day in Python, and the requirement to handle it with a property is even more unusual.

Essential Attributes and Function for Attributes Handlingđź”—

Special Attributes that Affect Attribute Handlingđź”—

  • __class__ : A reference to the object’s class. Python looks for special methods such as __getattr__ only in an object’s class and not in the instance themselves.
  • __dict__: A mapping that stores the writeable attributes of an object or class. An object that has a __dict__ can have arbitrary new attributes set at any time. If a class has a __slots__ attribute, then its instances may not have a __dict__.
  • __slots__ : An attribute that may be defined in a class to save memory. __slots__ is a tuple of strings naming the allowed attributes

Built-in Functions for Attribute Handlingđź”—

  • dir([object]) : lists most attributes of the object. The official docs say dir is intended for interactive use so it does not provide a comprehensive list of attributes, but an “interesting” set of names. dir can inspect objects implemented with or without a dict. The dict attribute itself is not listed by dir, but the dict keys are listed. Several special attributes of classes, such as mro, bases, and name, are not listed by dir either.
  • getattr(object, name[, default]) : Gets the attribute identified by the name string from the object. The main use case is to retrieve attributes (or methods) whose names we don’t know beforehand. This may fetch an attribute from the object’s class or from a superclass.
  • hasattr(object, name) : Returns True if the named attribute exists in the object, or can be somehow fetched through it (by inheritance, for example).
  • setattr(object, name, value) : Assigns the value to the named attribute of object, if the object allows it. This may create a new attribute or overwrite an existing one.
  • vars([object]) : Returns the __dict__ of object; vars can’t deal with instances of classes that define __slots__ and don’t have a __dict__ (contrast with dir, which handles such instances). Without an argument, vars() does the same as locals(): returns a dict representing the local scope.

Special Methods for Attribute Handlingđź”—

  • __delattr__(self, name) : Always called when there is an attempt to delete an attribute using the del statement;
  • __dir__(self) : Called when dir is invoked on the object, to provide a listing of attributes; dir(obj) triggers Class.__dir__(obj). Also used by tab-completion in all modern Python consoles.
  • __getattr__(self, name) : Called only when an attempt to retrieve the named attribute fails, after the obj, Class, and its superclasses are searched. The expressions obj.no_such_attr, getattr(obj, 'no_such_attr'), and hasattr(obj, 'no_such_attr') may trigger Class.__getattr__(obj, 'no_such_attr'), but only if an attribute by that name cannot be found in obj or in Class and its superclasses.
  • __getattribute__(self, name) : Always called when there is an attempt to retrieve the named attribute directly from Python code (the interpreter may bypass this in some cases, for example, to get the __repr__ method). Dot notation and the getattr and hasattr built-ins trigger this method.
  • __setattr__(self, name, value) : Always called when there is an attempt to set the named attribute. Dot notation and the setattr built-in trigger this method