4
<TITLE>Metaclasses in Python 1.5</TITLE>
7
<BODY BGCOLOR="FFFFFF">
9
<H1>Metaclasses in Python 1.5</H1>
10
<H2>(A.k.a. The Killer Joke :-)</H2>
14
(<i>Postscript:</i> reading this essay is probably not the best way to
15
understand the metaclass hook described here. See a <A
16
HREF="meta-vladimir.txt">message posted by Vladimir Marangozov</A>
17
which may give a gentler introduction to the matter. You may also
18
want to search Deja News for messages with "metaclass" in the subject
19
posted to comp.lang.python in July and August 1998.)
23
<P>In previous Python releases (and still in 1.5), there is something
24
called the ``Don Beaudry hook'', after its inventor and champion.
25
This allows C extensions to provide alternate class behavior, thereby
26
allowing the Python class syntax to be used to define other class-like
27
entities. Don Beaudry has used this in his infamous <A
28
HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> package; Jim
29
Fulton has used it in his <A
30
HREF="http://www.digicool.com/releases/ExtensionClass/">Extension
31
Classes</A> package. (It has also been referred to as the ``Don
32
Beaudry <i>hack</i>,'' but that's a misnomer. There's nothing hackish
33
about it -- in fact, it is rather elegant and deep, even though
34
there's something dark to it.)
36
<P>(On first reading, you may want to skip directly to the examples in
37
the section "Writing Metaclasses in Python" below, unless you want
38
your head to explode.)
44
<P>Documentation of the Don Beaudry hook has purposefully been kept
45
minimal, since it is a feature of incredible power, and is easily
46
abused. Basically, it checks whether the <b>type of the base
47
class</b> is callable, and if so, it is called to create the new
50
<P>Note the two indirection levels. Take a simple example:
60
Take a look at the second class definition, and try to fathom ``the
61
type of the base class is callable.''
63
<P>(Types are not classes, by the way. See questions 4.2, 4.19 and in
64
particular 6.22 in the <A
65
HREF="http://www.python.org/cgi-bin/faqw.py" >Python FAQ</A>
66
for more on this topic.)
72
<LI>The <b>base class</b> is B; this one's easy.<P>
74
<LI>Since B is a class, its type is ``class''; so the <b>type of the
75
base class</b> is the type ``class''. This is also known as
76
types.ClassType, assuming the standard module <code>types</code> has
79
<LI>Now is the type ``class'' <b>callable</b>? No, because types (in
80
core Python) are never callable. Classes are callable (calling a
81
class creates a new instance) but types aren't.<P>
85
<P>So our conclusion is that in our example, the type of the base
86
class (of C) is not callable. So the Don Beaudry hook does not apply,
87
and the default class creation mechanism is used (which is also used
88
when there is no base class). In fact, the Don Beaudry hook never
89
applies when using only core Python, since the type of a core object
92
<P>So what do Don and Jim do in order to use Don's hook? Write an
93
extension that defines at least two new Python object types. The
94
first would be the type for ``class-like'' objects usable as a base
95
class, to trigger Don's hook. This type must be made callable.
96
That's why we need a second type. Whether an object is callable
97
depends on its type. So whether a type object is callable depends on
98
<i>its</i> type, which is a <i>meta-type</i>. (In core Python there
99
is only one meta-type, the type ``type'' (types.TypeType), which is
100
the type of all type objects, even itself.) A new meta-type must
101
be defined that makes the type of the class-like objects callable.
102
(Normally, a third type would also be needed, the new ``instance''
103
type, but this is not an absolute requirement -- the new class type
104
could return an object of some existing type when invoked to create an
107
<P>Still confused? Here's a simple device due to Don himself to
108
explain metaclasses. Take a simple class definition; assume B is a
109
special class that triggers Don's hook:
117
This can be though of as equivalent to:
120
C = type(B)('C', (B,), {'a': 1, 'b': 2})
123
If that's too dense for you, here's the same thing written out using
127
creator = type(B) # The type of the base class
128
name = 'C' # The name of the new class
129
bases = (B,) # A tuple containing the base class(es)
130
namespace = {'a': 1, 'b': 2} # The namespace of the class statement
131
C = creator(name, bases, namespace)
134
This is analogous to what happens without the Don Beaudry hook, except
135
that in that case the creator function is set to the default class
138
<P>In either case, the creator is called with three arguments. The
139
first one, <i>name</i>, is the name of the new class (as given at the
140
top of the class statement). The <i>bases</i> argument is a tuple of
141
base classes (a singleton tuple if there's only one base class, like
142
the example). Finally, <i>namespace</i> is a dictionary containing
143
the local variables collected during execution of the class statement.
145
<P>Note that the contents of the namespace dictionary is simply
146
whatever names were defined in the class statement. A little-known
147
fact is that when Python executes a class statement, it enters a new
148
local namespace, and all assignments and function definitions take
149
place in this namespace. Thus, after executing the following class
158
the class namespace's contents would be {'a': 1, 'f': <function f
161
<P>But enough already about writing Python metaclasses in C; read the
163
HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> or <A
164
HREF="http://www.digicool.com/papers/ExtensionClass.html" >Extension
165
Classes</A> for more information.
171
<H2>Writing Metaclasses in Python</H2>
173
<P>In Python 1.5, the requirement to write a C extension in order to
174
write metaclasses has been dropped (though you can still do
175
it, of course). In addition to the check ``is the type of the base
176
class callable,'' there's a check ``does the base class have a
177
__class__ attribute.'' If so, it is assumed that the __class__
178
attribute refers to a class.
180
<P>Let's repeat our simple example from above:
188
Assuming B has a __class__ attribute, this translates into:
191
C = B.__class__('C', (B,), {'a': 1, 'b': 2})
194
This is exactly the same as before except that instead of type(B),
195
B.__class__ is invoked. If you have read <A HREF=
196
"http://www.python.org/cgi-bin/faqw.py?req=show&file=faq06.022.htp"
197
>FAQ question 6.22</A> you will understand that while there is a big
198
technical difference between type(B) and B.__class__, they play the
199
same role at different abstraction levels. And perhaps at some point
200
in the future they will really be the same thing (at which point you
201
would be able to derive subclasses from built-in types).
203
<P>At this point it may be worth mentioning that C.__class__ is the
204
same object as B.__class__, i.e., C's metaclass is the same as B's
205
metaclass. In other words, subclassing an existing class creates a
206
new (meta)inststance of the base class's metaclass.
208
<P>Going back to the example, the class B.__class__ is instantiated,
209
passing its constructor the same three arguments that are passed to
210
the default class constructor or to an extension's metaclass:
211
<i>name</i>, <i>bases</i>, and <i>namespace</i>.
213
<P>It is easy to be confused by what exactly happens when using a
214
metaclass, because we lose the absolute distinction between classes
215
and instances: a class is an instance of a metaclass (a
216
``metainstance''), but technically (i.e. in the eyes of the python
217
runtime system), the metaclass is just a class, and the metainstance
218
is just an instance. At the end of the class statement, the metaclass
219
whose metainstance is used as a base class is instantiated, yielding a
220
second metainstance (of the same metaclass). This metainstance is
221
then used as a (normal, non-meta) class; instantiation of the class
222
means calling the metainstance, and this will return a real instance.
223
And what class is that an instance of? Conceptually, it is of course
224
an instance of our metainstance; but in most cases the Python runtime
225
system will see it as an instance of a a helper class used by the
226
metaclass to implement its (non-meta) instances...
228
<P>Hopefully an example will make things clearer. Let's presume we
229
have a metaclass MetaClass1. It's helper class (for non-meta
230
instances) is callled HelperClass1. We now (manually) instantiate
231
MetaClass1 once to get an empty special base class:
234
BaseClass1 = MetaClass1("BaseClass1", (), {})
237
We can now use BaseClass1 as a base class in a class statement:
240
class MySpecialClass(BaseClass1):
245
At this point, MySpecialClass is defined; it is a metainstance of
246
MetaClass1 just like BaseClass1, and in fact the expression
247
``BaseClass1.__class__ == MySpecialClass.__class__ == MetaClass1''
250
<P>We are now ready to create instances of MySpecialClass. Let's
251
assume that no constructor arguments are required:
256
print x.__class__, y.__class__
259
The print statement shows that x and y are instances of HelperClass1.
260
How did this happen? MySpecialClass is an instance of MetaClass1
261
(``meta'' is irrelevant here); when an instance is called, its
262
__call__ method is invoked, and presumably the __call__ method defined
263
by MetaClass1 returns an instance of HelperClass1.
265
<P>Now let's see how we could use metaclasses -- what can we do
266
with metaclasses that we can't easily do without them? Here's one
267
idea: a metaclass could automatically insert trace calls for all
268
method calls. Let's first develop a simplified example, without
269
support for inheritance or other ``advanced'' Python features (we'll
276
def __init__(self, name, bases, namespace):
277
"""Create a new class."""
279
self.__bases__ = bases
280
self.__namespace__ = namespace
282
"""Create a new instance."""
283
return Instance(self)
286
def __init__(self, klass):
287
self.__klass__ = klass
288
def __getattr__(self, name):
290
value = self.__klass__.__namespace__[name]
292
raise AttributeError, name
293
if type(value) is not types.FunctionType:
295
return BoundMethod(value, self)
298
def __init__(self, function, instance):
299
self.function = function
300
self.instance = instance
301
def __call__(self, *args):
302
print "calling", self.function, "for", self.instance, "with", args
303
return apply(self.function, (self.instance,) + args)
305
Trace = Tracing('Trace', (), {})
307
class MyTracedClass(Trace):
308
def method1(self, a):
313
aninstance = MyTracedClass()
315
aninstance.method1(10)
317
print "the answer is %d" % aninstance.method2()
320
Confused already? The intention is to read this from top down. The
321
Tracing class is the metaclass we're defining. Its structure is
328
<LI>The __init__ method is invoked when a new Tracing instance is
329
created, e.g. the definition of class MyTracedClass later in the
330
example. It simply saves the class name, base classes and namespace
331
as instance variables.<P>
333
<LI>The __call__ method is invoked when a Tracing instance is called,
334
e.g. the creation of aninstance later in the example. It returns an
335
instance of the class Instance, which is defined next.<P>
339
<P>The class Instance is the class used for all instances of classes
340
built using the Tracing metaclass, e.g. aninstance. It has two
347
<LI>The __init__ method is invoked from the Tracing.__call__ method
348
above to initialize a new instance. It saves the class reference as
349
an instance variable. It uses a funny name because the user's
350
instance variables (e.g. self.a later in the example) live in the same
353
<LI>The __getattr__ method is invoked whenever the user code
354
references an attribute of the instance that is not an instance
355
variable (nor a class variable; but except for __init__ and
356
__getattr__ there are no class variables). It will be called, for
357
example, when aninstance.method1 is referenced in the example, with
358
self set to aninstance and name set to the string "method1".<P>
362
<P>The __getattr__ method looks the name up in the __namespace__
363
dictionary. If it isn't found, it raises an AttributeError exception.
364
(In a more realistic example, it would first have to look through the
365
base classes as well.) If it is found, there are two possibilities:
366
it's either a function or it isn't. If it's not a function, it is
367
assumed to be a class variable, and its value is returned. If it's a
368
function, we have to ``wrap'' it in instance of yet another helper
371
<P>The BoundMethod class is needed to implement a familiar feature:
372
when a method is defined, it has an initial argument, self, which is
373
automatically bound to the relevant instance when it is called. For
374
example, aninstance.method1(10) is equivalent to method1(aninstance,
375
10). In the example if this call, first a temporary BoundMethod
376
instance is created with the following constructor call: temp =
377
BoundMethod(method1, aninstance); then this instance is called as
378
temp(10). After the call, the temporary instance is discarded.
384
<LI>The __init__ method is invoked for the constructor call
385
BoundMethod(method1, aninstance). It simply saves away its
388
<LI>The __call__ method is invoked when the bound method instance is
389
called, as in temp(10). It needs to call method1(aninstance, 10).
390
However, even though self.function is now method1 and self.instance is
391
aninstance, it can't call self.function(self.instance, args) directly,
392
because it should work regardless of the number of arguments passed.
393
(For simplicity, support for keyword arguments has been omitted.)<P>
397
<P>In order to be able to support arbitrary argument lists, the
398
__call__ method first constructs a new argument tuple. Conveniently,
399
because of the notation *args in __call__'s own argument list, the
400
arguments to __call__ (except for self) are placed in the tuple args.
401
To construct the desired argument list, we concatenate a singleton
402
tuple containing the instance with the args tuple: (self.instance,) +
403
args. (Note the trailing comma used to construct the singleton
404
tuple.) In our example, the resulting argument tuple is (aninstance,
407
<P>The intrinsic function apply() takes a function and an argument
408
tuple and calls the function for it. In our example, we are calling
409
apply(method1, (aninstance, 10)) which is equivalent to calling
410
method(aninstance, 10).
412
<P>From here on, things should come together quite easily. The output
413
of the example code is something like this:
416
calling <function method1 at ae8d8> for <Instance instance at 95ab0> with (10,)
417
calling <function method2 at ae900> for <Instance instance at 95ab0> with ()
421
<P>That was about the shortest meaningful example that I could come up
422
with. A real tracing metaclass (for example, <A
423
HREF="#Trace">Trace.py</A> discussed below) needs to be more
424
complicated in two dimensions.
426
<P>First, it needs to support more advanced Python features such as
427
class variables, inheritance, __init__ methods, and keyword arguments.
429
<P>Second, it needs to provide a more flexible way to handle the
430
actual tracing information; perhaps it should be possible to write
431
your own tracing function that gets called, perhaps it should be
432
possible to enable and disable tracing on a per-class or per-instance
433
basis, and perhaps a filter so that only interesting calls are traced;
434
it should also be able to trace the return value of the call (or the
435
exception it raised if an error occurs). Even the Trace.py example
436
doesn't support all these features yet.
442
<H1>Real-life Examples</H1>
444
<P>Have a look at some very preliminary examples that I coded up to
445
teach myself how to write metaclasses:
449
<DT><A HREF="Enum.py">Enum.py</A>
451
<DD>This (ab)uses the class syntax as an elegant way to define
452
enumerated types. The resulting classes are never instantiated --
453
rather, their class attributes are the enumerated values. For
464
will print the string ``Color.red'', while ``Color.red==1'' is true,
465
and ``Color.red + 1'' raise a TypeError exception.
469
<DT><A NAME=Trace></A><A HREF="Trace.py">Trace.py</A>
471
<DD>The resulting classes work much like standard
472
classes, but by setting a special class or instance attribute
473
__trace_output__ to point to a file, all calls to the class's methods
474
are traced. It was a bit of a struggle to get this right. This
475
should probably redone using the generic metaclass below.
479
<DT><A HREF="Meta.py">Meta.py</A>
481
<DD>A generic metaclass. This is an attempt at finding out how much
482
standard class behavior can be mimicked by a metaclass. The
483
preliminary answer appears to be that everything's fine as long as the
484
class (or its clients) don't look at the instance's __class__
485
attribute, nor at the class's __dict__ attribute. The use of
486
__getattr__ internally makes the classic implementation of __getattr__
487
hooks tough; we provide a similar hook _getattr_ instead.
488
(__setattr__ and __delattr__ are not affected.)
489
(XXX Hm. Could detect presence of __getattr__ and rename it.)
493
<DT><A HREF="Eiffel.py">Eiffel.py</A>
495
<DD>Uses the above generic metaclass to implement Eiffel style
496
pre-conditions and post-conditions.
500
<DT><A HREF="Synch.py">Synch.py</A>
502
<DD>Uses the above generic metaclass to implement synchronized
507
<DT><A HREF="Simple.py">Simple.py</A>
509
<DD>The example module used above.
515
<P>A pattern seems to be emerging: almost all these uses of
516
metaclasses (except for Enum, which is probably more cute than useful)
517
mostly work by placing wrappers around method calls. An obvious
518
problem with that is that it's not easy to combine the features of
519
different metaclasses, while this would actually be quite useful: for
520
example, I wouldn't mind getting a trace from the test run of the
521
Synch module, and it would be interesting to add preconditions to it
522
as well. This needs more research. Perhaps a metaclass could be
523
provided that allows stackable wrappers...
529
<H2>Things You Could Do With Metaclasses</H2>
531
<P>There are lots of things you could do with metaclasses. Most of
532
these can also be done with creative use of __getattr__, but
533
metaclasses make it easier to modify the attribute lookup behavior of
534
classes. Here's a partial list.
540
<LI>Enforce different inheritance semantics, e.g. automatically call
541
base class methods when a derived class overrides<P>
543
<LI>Implement class methods (e.g. if the first argument is not named
546
<LI>Implement that each instance is initialized with <b>copies</b> of
547
all class variables<P>
549
<LI>Implement a different way to store instance variables (e.g. in a
550
list kept outside the instance but indexed by the instance's id())<P>
552
<LI>Automatically wrap or trap all or certain methods
558
<LI>for precondition and postcondition checking
560
<LI>for synchronized methods
562
<LI>for automatic value caching
567
<LI>When an attribute is a parameterless function, call it on
568
reference (to mimic it being an instance variable); same on assignment<P>
570
<LI>Instrumentation: see how many times various attributes are used<P>
572
<LI>Different semantics for __setattr__ and __getattr__ (e.g. disable
573
them when they are being used recursively)<P>
575
<LI>Abuse class syntax for other things<P>
577
<LI>Experiment with automatic type checking<P>
579
<LI>Delegation (or acquisition)<P>
581
<LI>Dynamic inheritance patterns<P>
583
<LI>Automatic caching of methods<P>
593
<P>Many thanks to David Ascher and Donald Beaudry for their comments
594
on earlier draft of this paper. Also thanks to Matt Conway and Tommy
595
Burnette for putting a seed for the idea of metaclasses in my
596
mind, nearly three years ago, even though at the time my response was
597
``you can do that with __getattr__ hooks...'' :-)
added
removed
Demo/metaclasses/index.html
