Welcome, guest | Sign In | My Account | Store | Cart

Any serious user of metaclasses has been bitten at least once by the infamous metaclass/metatype conflict. Here I give a general recipe to solve the problem, as well as some theory and some examples.

Python, 54 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 import inspect, types, __builtin__

 ############## preliminary: two utility functions #####################

 def skip_redundant(iterable, skipset=None):
    "Redundant items are repeated items or items in the original skipset."
    if skipset is None: skipset = set()
    for item in iterable:
        if item not in skipset:
            skipset.add(item)
            yield item


 def remove_redundant(metaclasses):
    skipset = set([types.ClassType])
    for meta in metaclasses: # determines the metaclasses to be skipped
        skipset.update(inspect.getmro(meta)[1:])
    return tuple(skip_redundant(metaclasses, skipset))

 ##################################################################
 ## now the core of the module: two mutually recursive functions ##
 ##################################################################

 memoized_metaclasses_map = {}

 def get_noconflict_metaclass(bases, left_metas, right_metas):
     """Not intended to be used outside of this module, unless you know
     what you are doing."""
     # make tuple of needed metaclasses in specified priority order
     metas = left_metas + tuple(map(type, bases)) + right_metas
     needed_metas = remove_redundant(metas)

     # return existing confict-solving meta, if any
     if needed_metas in memoized_metaclasses_map:
       return memoized_metaclasses_map[needed_metas]
     # nope: compute, memoize and return needed conflict-solving meta
     elif not needed_metas:         # wee, a trivial case, happy us
         meta = type
     elif len(needed_metas) == 1: # another trivial case
        meta = needed_metas[0]
     # check for recursion, can happen i.e. for Zope ExtensionClasses
     elif needed_metas == bases: 
         raise TypeError("Incompatible root metatypes", needed_metas)
     else: # gotta work ...
         metaname = '_' + ''.join([m.__name__ for m in needed_metas])
         meta = classmaker()(metaname, needed_metas, {})
     memoized_metaclasses_map[needed_metas] = meta
     return meta

 def classmaker(left_metas=(), right_metas=()):
    def make_class(name, bases, adict):
        metaclass = get_noconflict_metaclass(bases, left_metas, right_metas)
        return metaclass(name, bases, adict)
    return make_class

I think that not too many programmers are familiar with metaclasses and metatype conflicts, therefore let me be pedagogical ;)

The simplest case where a metatype conflict happens is the following. Consider a class A with metaclass M_A and a class B with an independent metaclass M_B; suppose we derive C from A and B. The question is: what is the metaclass of C ? Is it M_A or M_B ?

>>> class M_A(type):
...     pass
>>> class M_B(type):
...     pass
>>> class A(object):
...     __metaclass__=M_A
>>> class B(object):
...     __metaclass__=M_B
>>> class C(A,B):
...     pass
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

The correct answer (see the book "Putting metaclasses to work" for a thoughtful discussion) is M_C, where M_C is a metaclass that inherits from M_A and M_B, as in the following graph, where instantiation is denoted by colon lines:

<pre>

         M_A     M_B
          : \   / :
          :  \ /  :
          A  M_C  B
           \  :  /
            \ : /
              C

</pre>

However, Python is not that magic, and it does not automatically create M_C. Instead, it raises a TypeError, warning the programmer of the possible confusion. The metatype conflict can be avoided by assegning the correct metaclass to C by hand:

>>> class M_AM_B(M_A,M_B): pass
...
>>> class C(A,B):
...     __metaclass__=M_AM_B
>>> C,type(C)
(<class 'C'>, <class 'M_AM_B'>)

In general, a class A(B, C, D , ...) can be generated without conflicts only if type(A) is a subclass of each of type(B), type(C), ...

It is possible to automatically avoid conflicts, by defining a smart class factory that generates the correct metaclass by looking at the metaclasses of the base classes. This is done via the classmaker class factory, wich internally invokes the get_noconflict_metaclass function.

>>> from noconflict import classmaker
>>> class C(A,B):
...     __metaclass__=classmaker()
>>> C
<class 'C'>
>>> type(C) # automatically generated metaclass
<class 'noconflict._M_AM_B'>

In order to avoid to generate twice the same metaclass, they are stored in a dictionary. In particular, when classmaker is invoked with the same arguments it returns the same metaclass.

>>> class D(A,B):
...     __metaclass__=classmaker()
>>> type(D)
<class 'noconflict._M_AM_B'>
>>> type(C) is type(D)
True

Another example where classmaker() can solve the conflict is the following:

>>> class D(A):
...     __metaclass__=M_B
Traceback (most recent call last):
  File "<string>", line 1, in ?
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

Here the problem is that since D inherits from A, its metaclass must inherit from M_A and cannot be M_B.

classmaker solves the problem by automatically inheriting both from M_A and M_B:

>>> class D(A):
...     __metaclass__=classmaker(right_metas=(M_B,))
>>> type(D)
<class 'noconflict._M_AM_B'>

In some case, the user may want M_B to have the priority over M_A. This is easily done:

>>> class D(A):
...     __metaclass__ = classmaker(left_metas=(M_B,))
>>> type(D)
<class 'noconflict._M_BM_A'>

classmaker automatically skips unneeded metaclasses,

>>> class M0(type): pass
...
>>> class M1(M0): pass
...
>>> class B: __metaclass__=M0
...
>>> class C(B): __metaclass__=classmaker(right_metas=(M1,))
...
>>> print C,type(C)
<class 'C'> <class 'M1'>

i.e. in this example where M1 is a subclass of M0, it returns M1 and does not generate a redundant metaclass _M0M1.

classmaker also solves the meta-metaclass conflict, and generic higher order conflicts:

>>> class MM1(type): pass
...
>>> class MM2(type): pass
...
>>> class M1(type): __metaclass__=MM1
...
>>> class M2(type): __metaclass__=MM2
...
>>> class A: __metaclass__=M1
...
>>> class B: __metaclass__=M2
...
>>> class C(A,B): __metaclass__ = classmaker()
...
>>> print C,type(C),type(type(C))
<class 'C'> <class 'noconflict._M1M2'> <class 'noconflict._MM1MM2'>

I thank David Mertz for help in polishing the original version of the code. The second version has largerly profited from discussion with Phillip J. Eby. The version reported here is the closest to the printed version in the second edition of the Python cookbook and contains many insights from Alex Martelli and Anna Ravenscroft.

These examples here have been checked with doctest on Python 2.4.

7 comments

Graham Horler 18 years, 3 months ago  # | flag

Typo perhaps. What is makecls() in the examples? (Perhaps I missed something)

Very interesting/enlightening stuff.

Michele Simionato (author) 18 years, 3 months ago  # | flag

A typo indeed. makecls should read classmaker (it was called makecls in the first version, classmaker in the printed version)

Stephen Chappell 11 years, 11 months ago  # | flag

Are you going to correct the typographical mistake?

Michele Simionato (author) 11 years, 11 months ago  # | flag

Typo fixed.

kristi 11 years, 10 months ago  # | flag

The indentation is off: there are extra spaces at the front of some lines.

twangist 9 years, 7 months ago  # | flag

I assume that _generatemetaclass is also vestigial, and should be classmaker, yes?

Michele Simionato (author) 9 years, 7 months ago  # | flag

Yes, fixed, thanks.

Created by Michele Simionato on Sat, 7 Jun 2003 (PSF)
Python recipes (4591)
Michele Simionato's recipes (12)

Required Modules

  • (none specified)

Other Information and Tasks