3
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
4
<title>Mako Documentation - The Unicode Chapter</title>
6
<link rel="stylesheet" href="docs.css"></link>
7
<link rel="stylesheet" href="highlight.css"></link>
25
<div id="topanchor"><a name="top"> </a></div>
27
<div id="pagecontrol"><a href="index.html">Multiple Pages</a> | <a href="documentation.html">One Page</a></div>
29
<h1>Mako Documentation</h1>
31
<div class="versionheader">Version: 0.1.9 Last Updated: 09/25/07 16:19:18</div>
46
<div class="prevnext">
47
Previous: <a href="filtering.html">Filtering and Buffering</a>
50
Next: <a href="caching.html">Caching</a>
52
<h3><a href="index.html">Table of Contents</a></h3>
57
<a href="#unicode">The Unicode Chapter</a>
61
<li><A style="" href="unicode.html#unicode_specifying">Specifying the Encoding of a Template File</a></li>
67
<li><A style="" href="unicode.html#unicode_handling">Handling Expressions</a></li>
73
<li><A style="" href="unicode.html#unicode_defining">Defining Output Encoding</a></li>
77
<li><A style="" href="unicode.html#unicode_defining_buffer">Buffer Selection</a></li>
100
<A name="unicode"></a>
102
<div class="section">
105
<h3>The Unicode Chapter</h3>
109
<p>The Python language in the 2.x series supports two ways of representing string objects. One is the <code>string</code> type and the other is the <code>unicode</code> type, both of which extend a type called <code>basestring</code>. A key issue in Python, which is hopefully to be resolved in Python 3000, is that objects of type <code>string</code> (i.e. created from an expression such as <code>"hello world"</code>) contain no information regarding what <strong>encoding</strong> the data is stored in. For this reason they are often referred to as <strong>byte strings</strong>. The origins of this come from Python's background of being developed before the Unicode standard was even available, back when strings were C-style strings and were just that, a series of bytes. Strings that had only values below 128 just happened to be <strong>ascii</strong> strings and were printable on the console, whereas strings with values above 128 would produce all kinds of graphical characters and bells.
111
<p>Contrast the Python <code>string</code> type with the Python <code>unicode</code> type. Objects of this type are created whenever you say something like <code>u"hello world"</code>. In this case, Python represents each character in the string internally using multiple bytes per character (something similar to UTF-16). Whats important is that when using the <code>unicode</code> type to store strings, Python knows the data's encoding; its in its own internal format. Whereas when using the <code>string</code> type, it does not.
113
<p>When Python attempts to treat a byte-string as a string, which means its attempting to compare/parse its characters, to coerce it into another encoding, or to decode it to a unicode object, it has to guess what the encoding is. In this case, it will pretty much always guess the encoding as <code>ascii</code>...and if the bytestring contains bytes above value 128, you'll get an error.
115
<p>There is one operation that Python <em>can</em> do with a non-ascii bytestring, and its a great source of confusion: it can dump the bytestring straight out to a stream or a file, with nary a care what the encoding is. To Python, this is pretty much like dumping any other kind of binary data (like an image) to a stream somewhere. So in a lot of cases, programs that embed all kinds of international characters and encodings into plain byte-strings (i.e. using <code>"hello world"</code> style literals) can fly right through their run, sending reams of strings out to whereever they are going, and the programmer, seeing the same output as was expressed in the input, is now under the illusion that his or her program is Unicode-compliant. In fact, their program has no unicode awareness whatsoever, and similarly has no ability to interact with libraries that <em>are</em> unicode aware.
117
<p>Particularly, some template languages like Cheetah, as well as earlier versions of Myghty, will treat expressions in this manner..they just go right through. Theres nothing "incorrect" about this, but Mako, since it deals with unicode internally, usually requires explicitness when dealing with non-ascii encodings. Additionally, if you ever need to handle unicode strings and other kinds of encoding conversions more intelligently, the usage of raw bytestrings quickly becomes a nightmare, since you are sending the Python interpreter collections of bytes for which it can make no intelligent decisions with regards to encoding.
119
<p>In Mako, all parsed template constructs and output streams are handled internally as Python <code>unicode</code> objects. Its only at the point of <code>render()</code> that this unicode stream is rendered into whatever the desired output encoding is. The implication here is that the template developer must ensure that the encoding of all non-ascii templates is explicit, that all non-ascii-encoded expressions are in one way or another converted to unicode, and that the output stream of the template is handled as a unicode stream being encoded to some encoding.
124
<A name="unicode_specifying"></a>
126
<div class="subsection">
129
<h3>Specifying the Encoding of a Template File</h3>
133
<p>This is the most basic encoding-related setting, and it is equivalent to Python's "magic encoding comment", as described in <a href='http://www.python.org/dev/peps/pep-0263/'>pep-0263</a>. Any template that contains non-ascii characters requires that this comment be present so that Mako can decode to unicode (and also make usage of Python's AST parsing services). Mako's lexer will use this encoding in order to convert the template source into a <code>unicode</code> object before continuing its parsing:
140
<div class="highlight" ><pre><span class="cp">## -*- coding: utf-8 -*-</span>
142
Alors vous imaginez ma surprise, au lever du jour, quand une drôle de petit voix m’a réveillé. Elle disait: « S’il vous plaît… dessine-moi un mouton! »
146
<p>For the picky, the regular expression used is derived from that of the abovementioned pep:
153
<div class="highlight" ><pre><span class="c">#.*coding[:=]\s*([-\w.]+).*\n</span>
157
<p>The lexer will convert to unicode in all cases, so that if any characters exist in the template that are outside of the specified encoding (or the default of <code>ascii</code>), the error will be immediate.
159
<p>As an alternative, the template encoding can be specified programmatically to either <code>Template</code> or <code>TemplateLookup</code> via the <code>input_encoding</code> parameter:
166
<div class="highlight" ><pre><span class="n">t</span> <span class="o">=</span> <span class="n">TemplateLookup</span><span class="p">(</span><span class="n">directories</span><span class="o">=</span><span class="p">[</span><span class="s">'./'</span><span class="p">],</span> <span class="n">input_encoding</span><span class="o">=</span><span class="s">'utf-8'</span><span class="p">)</span>
170
<p>The above will assume all located templates specify <code>utf-8</code> encoding, unless the template itself contains its own magic encoding comment, which takes precedence.
175
<a href="#top">back to section top</a>
181
<A name="unicode_handling"></a>
183
<div class="subsection">
186
<h3>Handling Expressions</h3>
190
<p>The next area that encoding comes into play is in expression constructs. By default, Mako's treatment of an expression like this:
197
<div class="highlight" ><pre><span class="cp">${</span><span class="s">"hello world"</span><span class="cp">}</span>
201
<p>looks something like this:
208
<div class="highlight" ><pre><span class="n">context</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">unicode</span><span class="p">(</span><span class="s">"hello world"</span><span class="p">))</span>
212
<p>That is, <em></em>the output of all expressions is run through the <code>unicode</code> builtin<em></em>. This is the default setting, and can be modified to expect various encodings. The <code>unicode</code> step serves both the purpose of rendering non-string expressions into strings (such as integers or objects which contain <code>__str()__</code> methods), and to ensure that the final output stream is constructed as a unicode object. The main implication of this is that <strong>any raw bytestrings that contain an encoding other than ascii must first be decoded to a Python unicode object</strong>. It means you can't say this:
219
<div class="highlight" ><pre><span class="cp">${</span><span class="s">"voix m’a réveillé."</span><span class="cp">}</span> <span class="cp">## error !</span>
223
<p>You must instead say this:
230
<div class="highlight" ><pre><span class="cp">${</span><span class="n">u</span><span class="s">"voix m’a réveillé."</span><span class="cp">}</span> <span class="cp">## OK !</span>
234
<p>Similarly, if you are reading data from a file, or returning data from some object that is returning a Python bytestring containing a non-ascii encoding, you have to explcitly decode to unicode first, such as:
241
<div class="highlight" ><pre><span class="cp">${</span><span class="n">call_my_object</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">)</span><span class="cp">}</span>
245
<p>If you want a certain encoding applied to <em>all</em> expressions, override the <code>unicode</code> builtin with the <code>decode</code> builtin at the <code>Template</code> or <code>TemplateLookup</code> level:
252
<div class="highlight" ><pre><span class="n">t</span> <span class="o">=</span> <span class="n">Template</span><span class="p">(</span><span class="n">templatetext</span><span class="p">,</span> <span class="n">default_filters</span><span class="o">=</span><span class="p">[</span><span class="s">'decode.utf8'</span><span class="p">])</span>
256
<p>Note that the built-in <code>decode</code> object is slower than the <code>unicode</code> function, since unlike <code>unicode</code> its not a Python builtin, and it also checks the type of the incoming data to determine if string conversion is needed first.
258
<p>The <code>default_filters</code> argument can be used to entirely customize the filtering process of expressions. This argument is described in <a href="filtering.html#filtering_expression_defaultfilters">The default_filters Argument</a>.
263
<a href="#top">back to section top</a>
269
<A name="unicode_defining"></a>
271
<div class="subsection">
274
<h3>Defining Output Encoding</h3>
278
<p>Now that we have a template which produces a pure unicode output stream, all the hard work is done. We can take the output and do anything with it.
280
<p>As stated in the "Usage" chapter, both <code>Template</code> and <code>TemplateLookup</code> accept <code>output_encoding</code> and <code>encoding_errors</code> parameters which can be used to encode the output in any Python supported codec:
287
<div class="highlight" ><pre><span class="k">from</span> <span class="nn">mako.template</span> <span class="k">import</span> <span class="n">Template</span>
288
<span class="k">from</span> <span class="nn">mako.lookup</span> <span class="k">import</span> <span class="n">TemplateLookup</span>
290
<span class="n">mylookup</span> <span class="o">=</span> <span class="n">TemplateLookup</span><span class="p">(</span><span class="n">directories</span><span class="o">=</span><span class="p">[</span><span class="s">'/docs'</span><span class="p">],</span> <span class="n">output_encoding</span><span class="o">=</span><span class="s">'utf-8'</span><span class="p">,</span> <span class="n">encoding_errors</span><span class="o">=</span><span class="s">'replace'</span><span class="p">)</span>
292
<span class="n">mytemplate</span> <span class="o">=</span> <span class="n">mylookup</span><span class="o">.</span><span class="n">get_template</span><span class="p">(</span><span class="s">"foo.txt"</span><span class="p">)</span>
293
<span class="k">print</span> <span class="n">mytemplate</span><span class="o">.</span><span class="n">render</span><span class="p">()</span>
297
<p>And <code>render_unicode()</code> will return the template output as a Python <code>unicode</code> object:
304
<div class="highlight" ><pre><span class="k">print</span> <span class="n">mytemplate</span><span class="o">.</span><span class="n">render_unicode</span><span class="p">()</span>
308
<p>The above method disgards the output encoding keyword argument; you can encode yourself by saying:
315
<div class="highlight" ><pre><span class="k">print</span> <span class="n">mytemplate</span><span class="o">.</span><span class="n">render_unicode</span><span class="p">()</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">,</span> <span class="s">'replace'</span><span class="p">)</span>
322
<A name="unicode_defining_buffer"></a>
324
<div class="subsection">
327
<h3>Buffer Selection</h3>
331
<p>Mako does play some games with the style of buffering used internally, to maximize performance. Since the buffer is by far the most heavily used object in a render operation, its important!
333
<p>When calling <code>render()</code> on a template that does not specify any output encoding (i.e. its <code>ascii</code>), Python's <code>cStringIO</code> module, which cannot handle encoding of non-ascii <code>unicode</code> objects (even though it can send raw bytestrings through), is used for buffering. Otherwise, a custom Mako class called <code>FastEncodingBuffer</code> is used, which essentially is a super dumbed-down version of <code>StringIO</code> that gathers all strings into a list and uses <code>u''.join(elements)</code> to produce the final output - its markedly faster than <code>StringIO</code>.
349
<a href="#top">back to section top</a>
356
<div class="toolbar">
357
<div class="prevnext">
358
Previous: <a href="filtering.html">Filtering and Buffering</a>
361
Next: <a href="caching.html">Caching</a>
363
<h3><a href="index.html">Table of Contents</a></h3>
added
removed
src/Mako-0.1.9/doc/unicode.html
