2
:mod:`codecs` --- Codec registry and base classes
3
=================================================
6
:synopsis: Encode and decode data and streams.
7
.. moduleauthor:: Marc-Andre Lemburg <mal@lemburg.com>
8
.. sectionauthor:: Marc-Andre Lemburg <mal@lemburg.com>
9
.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de>
18
pair: stackable; streams
20
This module defines base classes for standard Python codecs (encoders and
21
decoders) and provides access to the internal Python codec registry which
22
manages the codec and error handling lookup process.
24
It defines the following functions:
27
.. function:: register(search_function)
29
Register a codec search function. Search functions are expected to take one
30
argument, the encoding name in all lower case letters, and return a
31
:class:`CodecInfo` object having the following attributes:
33
* ``name`` The name of the encoding;
35
* ``encode`` The stateless encoding function;
37
* ``decode`` The stateless decoding function;
39
* ``incrementalencoder`` An incremental encoder class or factory function;
41
* ``incrementaldecoder`` An incremental decoder class or factory function;
43
* ``streamwriter`` A stream writer class or factory function;
45
* ``streamreader`` A stream reader class or factory function.
47
The various functions or classes take the following arguments:
49
*encode* and *decode*: These must be functions or methods which have the same
50
interface as the :meth:`encode`/:meth:`decode` methods of Codec instances (see
51
Codec Interface). The functions/methods are expected to work in a stateless
54
*incrementalencoder* and *incrementaldecoder*: These have to be factory
55
functions providing the following interface:
57
``factory(errors='strict')``
59
The factory functions must return objects providing the interfaces defined by
60
the base classes :class:`IncrementalEncoder` and :class:`IncrementalDecoder`,
61
respectively. Incremental codecs can maintain state.
63
*streamreader* and *streamwriter*: These have to be factory functions providing
64
the following interface:
66
``factory(stream, errors='strict')``
68
The factory functions must return objects providing the interfaces defined by
69
the base classes :class:`StreamWriter` and :class:`StreamReader`, respectively.
70
Stream codecs can maintain state.
72
Possible values for errors are ``'strict'`` (raise an exception in case of an
73
encoding error), ``'replace'`` (replace malformed data with a suitable
74
replacement marker, such as ``'?'``), ``'ignore'`` (ignore malformed data and
75
continue without further notice), ``'xmlcharrefreplace'`` (replace with the
76
appropriate XML character reference (for encoding only)) and
77
``'backslashreplace'`` (replace with backslashed escape sequences (for encoding
78
only)) as well as any other error handling name defined via
79
:func:`register_error`.
81
In case a search function cannot find a given encoding, it should return
85
.. function:: lookup(encoding)
87
Looks up the codec info in the Python codec registry and returns a
88
:class:`CodecInfo` object as defined above.
90
Encodings are first looked up in the registry's cache. If not found, the list of
91
registered search functions is scanned. If no :class:`CodecInfo` object is
92
found, a :exc:`LookupError` is raised. Otherwise, the :class:`CodecInfo` object
93
is stored in the cache and returned to the caller.
95
To simplify access to the various codecs, the module provides these additional
96
functions which use :func:`lookup` for the codec lookup:
99
.. function:: getencoder(encoding)
101
Look up the codec for the given encoding and return its encoder function.
103
Raises a :exc:`LookupError` in case the encoding cannot be found.
106
.. function:: getdecoder(encoding)
108
Look up the codec for the given encoding and return its decoder function.
110
Raises a :exc:`LookupError` in case the encoding cannot be found.
113
.. function:: getincrementalencoder(encoding)
115
Look up the codec for the given encoding and return its incremental encoder
116
class or factory function.
118
Raises a :exc:`LookupError` in case the encoding cannot be found or the codec
119
doesn't support an incremental encoder.
122
.. function:: getincrementaldecoder(encoding)
124
Look up the codec for the given encoding and return its incremental decoder
125
class or factory function.
127
Raises a :exc:`LookupError` in case the encoding cannot be found or the codec
128
doesn't support an incremental decoder.
131
.. function:: getreader(encoding)
133
Look up the codec for the given encoding and return its StreamReader class or
136
Raises a :exc:`LookupError` in case the encoding cannot be found.
139
.. function:: getwriter(encoding)
141
Look up the codec for the given encoding and return its StreamWriter class or
144
Raises a :exc:`LookupError` in case the encoding cannot be found.
147
.. function:: register_error(name, error_handler)
149
Register the error handling function *error_handler* under the name *name*.
150
*error_handler* will be called during encoding and decoding in case of an error,
151
when *name* is specified as the errors parameter.
153
For encoding *error_handler* will be called with a :exc:`UnicodeEncodeError`
154
instance, which contains information about the location of the error. The error
155
handler must either raise this or a different exception or return a tuple with a
156
replacement for the unencodable part of the input and a position where encoding
157
should continue. The encoder will encode the replacement and continue encoding
158
the original input at the specified position. Negative position values will be
159
treated as being relative to the end of the input string. If the resulting
160
position is out of bound an :exc:`IndexError` will be raised.
162
Decoding and translating works similar, except :exc:`UnicodeDecodeError` or
163
:exc:`UnicodeTranslateError` will be passed to the handler and that the
164
replacement from the error handler will be put into the output directly.
167
.. function:: lookup_error(name)
169
Return the error handler previously registered under the name *name*.
171
Raises a :exc:`LookupError` in case the handler cannot be found.
174
.. function:: strict_errors(exception)
176
Implements the ``strict`` error handling.
179
.. function:: replace_errors(exception)
181
Implements the ``replace`` error handling.
184
.. function:: ignore_errors(exception)
186
Implements the ``ignore`` error handling.
189
.. function:: xmlcharrefreplace_errors(exception)
191
Implements the ``xmlcharrefreplace`` error handling.
194
.. function:: backslashreplace_errors(exception)
196
Implements the ``backslashreplace`` error handling.
198
To simplify working with encoded files or stream, the module also defines these
202
.. function:: open(filename, mode[, encoding[, errors[, buffering]]])
204
Open an encoded file using the given *mode* and return a wrapped version
205
providing transparent encoding/decoding. The default file mode is ``'r'``
206
meaning to open the file in read mode.
210
The wrapped version's methods will accept and return strings only. Bytes
211
arguments will be rejected.
215
Files are always opened in binary mode, even if no binary mode was
216
specified. This is done to avoid data loss due to encodings using 8-bit
217
values. This means that no automatic conversion of ``b'\n'`` is done
218
on reading and writing.
220
*encoding* specifies the encoding which is to be used for the file.
222
*errors* may be given to define the error handling. It defaults to ``'strict'``
223
which causes a :exc:`ValueError` to be raised in case an encoding error occurs.
225
*buffering* has the same meaning as for the built-in :func:`open` function. It
226
defaults to line buffered.
229
.. function:: EncodedFile(file, input[, output[, errors]])
231
Return a wrapped version of file which provides transparent encoding
234
Bytes written to the wrapped file are interpreted according to the given
235
*input* encoding and then written to the original file as bytes using the
238
If *output* is not given, it defaults to *input*.
240
*errors* may be given to define the error handling. It defaults to ``'strict'``,
241
which causes :exc:`ValueError` to be raised in case an encoding error occurs.
244
.. function:: iterencode(iterable, encoding[, errors])
246
Uses an incremental encoder to iteratively encode the input provided by
247
*iterable*. This function is a :term:`generator`. *errors* (as well as any
248
other keyword argument) is passed through to the incremental encoder.
251
.. function:: iterdecode(iterable, encoding[, errors])
253
Uses an incremental decoder to iteratively decode the input provided by
254
*iterable*. This function is a :term:`generator`. *errors* (as well as any
255
other keyword argument) is passed through to the incremental decoder.
257
The module also provides the following constants which are useful for reading
258
and writing to platform dependent files:
272
These constants define various encodings of the Unicode byte order mark (BOM)
273
used in UTF-16 and UTF-32 data streams to indicate the byte order used in the
274
stream or file and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either
275
:const:`BOM_UTF16_BE` or :const:`BOM_UTF16_LE` depending on the platform's
276
native byte order, :const:`BOM` is an alias for :const:`BOM_UTF16`,
277
:const:`BOM_LE` for :const:`BOM_UTF16_LE` and :const:`BOM_BE` for
278
:const:`BOM_UTF16_BE`. The others represent the BOM in UTF-8 and UTF-32
282
.. _codec-base-classes:
287
The :mod:`codecs` module defines a set of base classes which define the
288
interface and can also be used to easily write your own codecs for use in
291
Each codec has to define four interfaces to make it usable as codec in Python:
292
stateless encoder, stateless decoder, stream reader and stream writer. The
293
stream reader and writers typically reuse the stateless encoder/decoder to
294
implement the file protocols.
296
The :class:`Codec` class defines the interface for stateless encoders/decoders.
298
To simplify and standardize error handling, the :meth:`encode` and
299
:meth:`decode` methods may implement different error handling schemes by
300
providing the *errors* string argument. The following string values are defined
301
and implemented by all standard Python codecs:
303
+-------------------------+-----------------------------------------------+
305
+=========================+===============================================+
306
| ``'strict'`` | Raise :exc:`UnicodeError` (or a subclass); |
307
| | this is the default. |
308
+-------------------------+-----------------------------------------------+
309
| ``'ignore'`` | Ignore the character and continue with the |
311
+-------------------------+-----------------------------------------------+
312
| ``'replace'`` | Replace with a suitable replacement |
313
| | character; Python will use the official |
314
| | U+FFFD REPLACEMENT CHARACTER for the built-in |
315
| | Unicode codecs on decoding and '?' on |
317
+-------------------------+-----------------------------------------------+
318
| ``'xmlcharrefreplace'`` | Replace with the appropriate XML character |
319
| | reference (only for encoding). |
320
+-------------------------+-----------------------------------------------+
321
| ``'backslashreplace'`` | Replace with backslashed escape sequences |
322
| | (only for encoding). |
323
+-------------------------+-----------------------------------------------+
325
The set of allowed values can be extended via :meth:`register_error`.
333
The :class:`Codec` class defines these methods which also define the function
334
interfaces of the stateless encoder and decoder:
337
.. method:: Codec.encode(input[, errors])
339
Encodes the object *input* and returns a tuple (output object, length consumed).
340
Encoding converts a string object to a bytes object using a particular
341
character set encoding (e.g., ``cp1252`` or ``iso-8859-1``).
343
*errors* defines the error handling to apply. It defaults to ``'strict'``
346
The method may not store state in the :class:`Codec` instance. Use
347
:class:`StreamCodec` for codecs which have to keep state in order to make
348
encoding/decoding efficient.
350
The encoder must be able to handle zero length input and return an empty object
351
of the output object type in this situation.
354
.. method:: Codec.decode(input[, errors])
356
Decodes the object *input* and returns a tuple (output object, length
357
consumed). Decoding converts a bytes object encoded using a particular
358
character set encoding to a string object.
360
*input* must be a bytes object or one which provides the read-only character
361
buffer interface -- for example, buffer objects and memory mapped files.
363
*errors* defines the error handling to apply. It defaults to ``'strict'``
366
The method may not store state in the :class:`Codec` instance. Use
367
:class:`StreamCodec` for codecs which have to keep state in order to make
368
encoding/decoding efficient.
370
The decoder must be able to handle zero length input and return an empty object
371
of the output object type in this situation.
373
The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide
374
the basic interface for incremental encoding and decoding. Encoding/decoding the
375
input isn't done with one call to the stateless encoder/decoder function, but
376
with multiple calls to the :meth:`encode`/:meth:`decode` method of the
377
incremental encoder/decoder. The incremental encoder/decoder keeps track of the
378
encoding/decoding process during method calls.
380
The joined output of calls to the :meth:`encode`/:meth:`decode` method is the
381
same as if all the single inputs were joined into one, and this input was
382
encoded/decoded with the stateless encoder/decoder.
385
.. _incremental-encoder-objects:
387
IncrementalEncoder Objects
388
^^^^^^^^^^^^^^^^^^^^^^^^^^
390
The :class:`IncrementalEncoder` class is used for encoding an input in multiple
391
steps. It defines the following methods which every incremental encoder must
392
define in order to be compatible with the Python codec registry.
395
.. class:: IncrementalEncoder([errors])
397
Constructor for an :class:`IncrementalEncoder` instance.
399
All incremental encoders must provide this constructor interface. They are free
400
to add additional keyword arguments, but only the ones defined here are used by
401
the Python codec registry.
403
The :class:`IncrementalEncoder` may implement different error handling schemes
404
by providing the *errors* keyword argument. These parameters are predefined:
406
* ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
408
* ``'ignore'`` Ignore the character and continue with the next.
410
* ``'replace'`` Replace with a suitable replacement character
412
* ``'xmlcharrefreplace'`` Replace with the appropriate XML character reference
414
* ``'backslashreplace'`` Replace with backslashed escape sequences.
416
The *errors* argument will be assigned to an attribute of the same name.
417
Assigning to this attribute makes it possible to switch between different error
418
handling strategies during the lifetime of the :class:`IncrementalEncoder`
421
The set of allowed values for the *errors* argument can be extended with
422
:func:`register_error`.
425
.. method:: encode(object[, final])
427
Encodes *object* (taking the current state of the encoder into account)
428
and returns the resulting encoded object. If this is the last call to
429
:meth:`encode` *final* must be true (the default is false).
434
Reset the encoder to the initial state.
437
.. method:: IncrementalEncoder.getstate()
439
Return the current state of the encoder which must be an integer. The
440
implementation should make sure that ``0`` is the most common state. (States
441
that are more complicated than integers can be converted into an integer by
442
marshaling/pickling the state and encoding the bytes of the resulting string
446
.. method:: IncrementalEncoder.setstate(state)
448
Set the state of the encoder to *state*. *state* must be an encoder state
449
returned by :meth:`getstate`.
452
.. _incremental-decoder-objects:
454
IncrementalDecoder Objects
455
^^^^^^^^^^^^^^^^^^^^^^^^^^
457
The :class:`IncrementalDecoder` class is used for decoding an input in multiple
458
steps. It defines the following methods which every incremental decoder must
459
define in order to be compatible with the Python codec registry.
462
.. class:: IncrementalDecoder([errors])
464
Constructor for an :class:`IncrementalDecoder` instance.
466
All incremental decoders must provide this constructor interface. They are free
467
to add additional keyword arguments, but only the ones defined here are used by
468
the Python codec registry.
470
The :class:`IncrementalDecoder` may implement different error handling schemes
471
by providing the *errors* keyword argument. These parameters are predefined:
473
* ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
475
* ``'ignore'`` Ignore the character and continue with the next.
477
* ``'replace'`` Replace with a suitable replacement character.
479
The *errors* argument will be assigned to an attribute of the same name.
480
Assigning to this attribute makes it possible to switch between different error
481
handling strategies during the lifetime of the :class:`IncrementalDecoder`
484
The set of allowed values for the *errors* argument can be extended with
485
:func:`register_error`.
488
.. method:: decode(object[, final])
490
Decodes *object* (taking the current state of the decoder into account)
491
and returns the resulting decoded object. If this is the last call to
492
:meth:`decode` *final* must be true (the default is false). If *final* is
493
true the decoder must decode the input completely and must flush all
494
buffers. If this isn't possible (e.g. because of incomplete byte sequences
495
at the end of the input) it must initiate error handling just like in the
496
stateless case (which might raise an exception).
501
Reset the decoder to the initial state.
504
.. method:: getstate()
506
Return the current state of the decoder. This must be a tuple with two
507
items, the first must be the buffer containing the still undecoded
508
input. The second must be an integer and can be additional state
509
info. (The implementation should make sure that ``0`` is the most common
510
additional state info.) If this additional state info is ``0`` it must be
511
possible to set the decoder to the state which has no input buffered and
512
``0`` as the additional state info, so that feeding the previously
513
buffered input to the decoder returns it to the previous state without
514
producing any output. (Additional state info that is more complicated than
515
integers can be converted into an integer by marshaling/pickling the info
516
and encoding the bytes of the resulting string into an integer.)
519
.. method:: setstate(state)
521
Set the state of the encoder to *state*. *state* must be a decoder state
522
returned by :meth:`getstate`.
525
The :class:`StreamWriter` and :class:`StreamReader` classes provide generic
526
working interfaces which can be used to implement new encoding submodules very
527
easily. See :mod:`encodings.utf_8` for an example of how this is done.
530
.. _stream-writer-objects:
535
The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the
536
following methods which every stream writer must define in order to be
537
compatible with the Python codec registry.
540
.. class:: StreamWriter(stream[, errors])
542
Constructor for a :class:`StreamWriter` instance.
544
All stream writers must provide this constructor interface. They are free to add
545
additional keyword arguments, but only the ones defined here are used by the
546
Python codec registry.
548
*stream* must be a file-like object open for writing binary data.
550
The :class:`StreamWriter` may implement different error handling schemes by
551
providing the *errors* keyword argument. These parameters are predefined:
553
* ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
555
* ``'ignore'`` Ignore the character and continue with the next.
557
* ``'replace'`` Replace with a suitable replacement character
559
* ``'xmlcharrefreplace'`` Replace with the appropriate XML character reference
561
* ``'backslashreplace'`` Replace with backslashed escape sequences.
563
The *errors* argument will be assigned to an attribute of the same name.
564
Assigning to this attribute makes it possible to switch between different error
565
handling strategies during the lifetime of the :class:`StreamWriter` object.
567
The set of allowed values for the *errors* argument can be extended with
568
:func:`register_error`.
571
.. method:: write(object)
573
Writes the object's contents encoded to the stream.
576
.. method:: writelines(list)
578
Writes the concatenated list of strings to the stream (possibly by reusing
579
the :meth:`write` method).
584
Flushes and resets the codec buffers used for keeping state.
586
Calling this method should ensure that the data on the output is put into
587
a clean state that allows appending of new fresh data without having to
588
rescan the whole stream to recover state.
591
In addition to the above methods, the :class:`StreamWriter` must also inherit
592
all other methods and attributes from the underlying stream.
595
.. _stream-reader-objects:
600
The :class:`StreamReader` class is a subclass of :class:`Codec` and defines the
601
following methods which every stream reader must define in order to be
602
compatible with the Python codec registry.
605
.. class:: StreamReader(stream[, errors])
607
Constructor for a :class:`StreamReader` instance.
609
All stream readers must provide this constructor interface. They are free to add
610
additional keyword arguments, but only the ones defined here are used by the
611
Python codec registry.
613
*stream* must be a file-like object open for reading (binary) data.
615
The :class:`StreamReader` may implement different error handling schemes by
616
providing the *errors* keyword argument. These parameters are defined:
618
* ``'strict'`` Raise :exc:`ValueError` (or a subclass); this is the default.
620
* ``'ignore'`` Ignore the character and continue with the next.
622
* ``'replace'`` Replace with a suitable replacement character.
624
The *errors* argument will be assigned to an attribute of the same name.
625
Assigning to this attribute makes it possible to switch between different error
626
handling strategies during the lifetime of the :class:`StreamReader` object.
628
The set of allowed values for the *errors* argument can be extended with
629
:func:`register_error`.
632
.. method:: read([size[, chars, [firstline]]])
634
Decodes data from the stream and returns the resulting object.
636
*chars* indicates the number of characters to read from the
637
stream. :func:`read` will never return more than *chars* characters, but
638
it might return less, if there are not enough characters available.
640
*size* indicates the approximate maximum number of bytes to read from the
641
stream for decoding purposes. The decoder can modify this setting as
642
appropriate. The default value -1 indicates to read and decode as much as
643
possible. *size* is intended to prevent having to decode huge files in
646
*firstline* indicates that it would be sufficient to only return the first
647
line, if there are decoding errors on later lines.
649
The method should use a greedy read strategy meaning that it should read
650
as much data as is allowed within the definition of the encoding and the
651
given size, e.g. if optional encoding endings or state markers are
652
available on the stream, these should be read too.
655
.. method:: readline([size[, keepends]])
657
Read one line from the input stream and return the decoded data.
659
*size*, if given, is passed as size argument to the stream's
660
:meth:`readline` method.
662
If *keepends* is false line-endings will be stripped from the lines
666
.. method:: readlines([sizehint[, keepends]])
668
Read all lines available on the input stream and return them as a list of
671
Line-endings are implemented using the codec's decoder method and are
672
included in the list entries if *keepends* is true.
674
*sizehint*, if given, is passed as the *size* argument to the stream's
680
Resets the codec buffers used for keeping state.
682
Note that no stream repositioning should take place. This method is
683
primarily intended to be able to recover from decoding errors.
686
In addition to the above methods, the :class:`StreamReader` must also inherit
687
all other methods and attributes from the underlying stream.
689
The next two base classes are included for convenience. They are not needed by
690
the codec registry, but may provide useful in practice.
693
.. _stream-reader-writer:
695
StreamReaderWriter Objects
696
^^^^^^^^^^^^^^^^^^^^^^^^^^
698
The :class:`StreamReaderWriter` allows wrapping streams which work in both read
701
The design is such that one can use the factory functions returned by the
702
:func:`lookup` function to construct the instance.
705
.. class:: StreamReaderWriter(stream, Reader, Writer, errors)
707
Creates a :class:`StreamReaderWriter` instance. *stream* must be a file-like
708
object. *Reader* and *Writer* must be factory functions or classes providing the
709
:class:`StreamReader` and :class:`StreamWriter` interface resp. Error handling
710
is done in the same way as defined for the stream readers and writers.
712
:class:`StreamReaderWriter` instances define the combined interfaces of
713
:class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other
714
methods and attributes from the underlying stream.
717
.. _stream-recoder-objects:
719
StreamRecoder Objects
720
^^^^^^^^^^^^^^^^^^^^^
722
The :class:`StreamRecoder` provide a frontend - backend view of encoding data
723
which is sometimes useful when dealing with different encoding environments.
725
The design is such that one can use the factory functions returned by the
726
:func:`lookup` function to construct the instance.
729
.. class:: StreamRecoder(stream, encode, decode, Reader, Writer, errors)
731
Creates a :class:`StreamRecoder` instance which implements a two-way conversion:
732
*encode* and *decode* work on the frontend (the input to :meth:`read` and output
733
of :meth:`write`) while *Reader* and *Writer* work on the backend (reading and
734
writing to the stream).
736
You can use these objects to do transparent direct recodings from e.g. Latin-1
739
*stream* must be a file-like object.
741
*encode*, *decode* must adhere to the :class:`Codec` interface. *Reader*,
742
*Writer* must be factory functions or classes providing objects of the
743
:class:`StreamReader` and :class:`StreamWriter` interface respectively.
745
*encode* and *decode* are needed for the frontend translation, *Reader* and
746
*Writer* for the backend translation.
748
Error handling is done in the same way as defined for the stream readers and
752
:class:`StreamRecoder` instances define the combined interfaces of
753
:class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other
754
methods and attributes from the underlying stream.
757
.. _encodings-overview:
759
Encodings and Unicode
760
---------------------
762
Strings are stored internally as sequences of codepoints (to be precise
763
as :ctype:`Py_UNICODE` arrays). Depending on the way Python is compiled (either
764
via :option:`--without-wide-unicode` or :option:`--with-wide-unicode`, with the
765
former being the default) :ctype:`Py_UNICODE` is either a 16-bit or 32-bit data
766
type. Once a string object is used outside of CPU and memory, CPU endianness
767
and how these arrays are stored as bytes become an issue. Transforming a
768
string object into a sequence of bytes is called encoding and recreating the
769
string object from the sequence of bytes is known as decoding. There are many
770
different methods for how this transformation can be done (these methods are
771
also called encodings). The simplest method is to map the codepoints 0-255 to
772
the bytes ``0x0``-``0xff``. This means that a string object that contains
773
codepoints above ``U+00FF`` can't be encoded with this method (which is called
774
``'latin-1'`` or ``'iso-8859-1'``). :func:`str.encode` will raise a
775
:exc:`UnicodeEncodeError` that looks like this: ``UnicodeEncodeError: 'latin-1'
776
codec can't encode character '\u1234' in position 3: ordinal not in
779
There's another group of encodings (the so called charmap encodings) that choose
780
a different subset of all Unicode code points and how these codepoints are
781
mapped to the bytes ``0x0``-``0xff``. To see how this is done simply open
782
e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on
783
Windows). There's a string constant with 256 characters that shows you which
784
character is mapped to which byte value.
786
All of these encodings can only encode 256 of the 65536 (or 1114111) codepoints
787
defined in Unicode. A simple and straightforward way that can store each Unicode
788
code point, is to store each codepoint as two consecutive bytes. There are two
789
possibilities: Store the bytes in big endian or in little endian order. These
790
two encodings are called UTF-16-BE and UTF-16-LE respectively. Their
791
disadvantage is that if e.g. you use UTF-16-BE on a little endian machine you
792
will always have to swap bytes on encoding and decoding. UTF-16 avoids this
793
problem: Bytes will always be in natural endianness. When these bytes are read
794
by a CPU with a different endianness, then bytes have to be swapped though. To
795
be able to detect the endianness of a UTF-16 byte sequence, there's the so
796
called BOM (the "Byte Order Mark"). This is the Unicode character ``U+FEFF``.
797
This character will be prepended to every UTF-16 byte sequence. The byte swapped
798
version of this character (``0xFFFE``) is an illegal character that may not
799
appear in a Unicode text. So when the first character in an UTF-16 byte sequence
800
appears to be a ``U+FFFE`` the bytes have to be swapped on decoding.
801
Unfortunately upto Unicode 4.0 the character ``U+FEFF`` had a second purpose as
802
a ``ZERO WIDTH NO-BREAK SPACE``: A character that has no width and doesn't allow
803
a word to be split. It can e.g. be used to give hints to a ligature algorithm.
804
With Unicode 4.0 using ``U+FEFF`` as a ``ZERO WIDTH NO-BREAK SPACE`` has been
805
deprecated (with ``U+2060`` (``WORD JOINER``) assuming this role). Nevertheless
806
Unicode software still must be able to handle ``U+FEFF`` in both roles: As a BOM
807
it's a device to determine the storage layout of the encoded bytes, and vanishes
808
once the byte sequence has been decoded into a string; as a ``ZERO WIDTH
809
NO-BREAK SPACE`` it's a normal character that will be decoded like any other.
811
There's another encoding that is able to encoding the full range of Unicode
812
characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues
813
with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two
814
parts: Marker bits (the most significant bits) and payload bits. The marker bits
815
are a sequence of zero to six 1 bits followed by a 0 bit. Unicode characters are
816
encoded like this (with x being payload bits, which when concatenated give the
819
+-----------------------------------+----------------------------------------------+
821
+===================================+==============================================+
822
| ``U-00000000`` ... ``U-0000007F`` | 0xxxxxxx |
823
+-----------------------------------+----------------------------------------------+
824
| ``U-00000080`` ... ``U-000007FF`` | 110xxxxx 10xxxxxx |
825
+-----------------------------------+----------------------------------------------+
826
| ``U-00000800`` ... ``U-0000FFFF`` | 1110xxxx 10xxxxxx 10xxxxxx |
827
+-----------------------------------+----------------------------------------------+
828
| ``U-00010000`` ... ``U-001FFFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx |
829
+-----------------------------------+----------------------------------------------+
830
| ``U-00200000`` ... ``U-03FFFFFF`` | 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
831
+-----------------------------------+----------------------------------------------+
832
| ``U-04000000`` ... ``U-7FFFFFFF`` | 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx |
834
+-----------------------------------+----------------------------------------------+
836
The least significant bit of the Unicode character is the rightmost x bit.
838
As UTF-8 is an 8-bit encoding no BOM is required and any ``U+FEFF`` character in
839
the decoded string (even if it's the first character) is treated as a ``ZERO
840
WIDTH NO-BREAK SPACE``.
842
Without external information it's impossible to reliably determine which
843
encoding was used for encoding a string. Each charmap encoding can
844
decode any random byte sequence. However that's not possible with UTF-8, as
845
UTF-8 byte sequences have a structure that doesn't allow arbitrary byte
846
sequences. To increase the reliability with which a UTF-8 encoding can be
847
detected, Microsoft invented a variant of UTF-8 (that Python 2.5 calls
848
``"utf-8-sig"``) for its Notepad program: Before any of the Unicode characters
849
is written to the file, a UTF-8 encoded BOM (which looks like this as a byte
850
sequence: ``0xef``, ``0xbb``, ``0xbf``) is written. As it's rather improbable
851
that any charmap encoded file starts with these byte values (which would e.g.
854
| LATIN SMALL LETTER I WITH DIAERESIS
855
| RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
856
| INVERTED QUESTION MARK
858
in iso-8859-1), this increases the probability that a utf-8-sig encoding can be
859
correctly guessed from the byte sequence. So here the BOM is not used to be able
860
to determine the byte order used for generating the byte sequence, but as a
861
signature that helps in guessing the encoding. On encoding the utf-8-sig codec
862
will write ``0xef``, ``0xbb``, ``0xbf`` as the first three bytes to the file. On
863
decoding utf-8-sig will skip those three bytes if they appear as the first three
867
.. _standard-encodings:
872
Python comes with a number of codecs built-in, either implemented as C functions
873
or with dictionaries as mapping tables. The following table lists the codecs by
874
name, together with a few common aliases, and the languages for which the
875
encoding is likely used. Neither the list of aliases nor the list of languages
876
is meant to be exhaustive. Notice that spelling alternatives that only differ in
877
case or use a hyphen instead of an underscore are also valid aliases.
879
Many of the character sets support the same languages. They vary in individual
880
characters (e.g. whether the EURO SIGN is supported or not), and in the
881
assignment of characters to code positions. For the European languages in
882
particular, the following variants typically exist:
884
* an ISO 8859 codeset
886
* a Microsoft Windows code page, which is typically derived from a 8859 codeset,
887
but replaces control characters with additional graphic characters
889
* an IBM EBCDIC code page
891
* an IBM PC code page, which is ASCII compatible
893
+-----------------+--------------------------------+--------------------------------+
894
| Codec | Aliases | Languages |
895
+=================+================================+================================+
896
| ascii | 646, us-ascii | English |
897
+-----------------+--------------------------------+--------------------------------+
898
| big5 | big5-tw, csbig5 | Traditional Chinese |
899
+-----------------+--------------------------------+--------------------------------+
900
| big5hkscs | big5-hkscs, hkscs | Traditional Chinese |
901
+-----------------+--------------------------------+--------------------------------+
902
| cp037 | IBM037, IBM039 | English |
903
+-----------------+--------------------------------+--------------------------------+
904
| cp424 | EBCDIC-CP-HE, IBM424 | Hebrew |
905
+-----------------+--------------------------------+--------------------------------+
906
| cp437 | 437, IBM437 | English |
907
+-----------------+--------------------------------+--------------------------------+
908
| cp500 | EBCDIC-CP-BE, EBCDIC-CP-CH, | Western Europe |
910
+-----------------+--------------------------------+--------------------------------+
912
+-----------------+--------------------------------+--------------------------------+
913
| cp775 | IBM775 | Baltic languages |
914
+-----------------+--------------------------------+--------------------------------+
915
| cp850 | 850, IBM850 | Western Europe |
916
+-----------------+--------------------------------+--------------------------------+
917
| cp852 | 852, IBM852 | Central and Eastern Europe |
918
+-----------------+--------------------------------+--------------------------------+
919
| cp855 | 855, IBM855 | Bulgarian, Byelorussian, |
920
| | | Macedonian, Russian, Serbian |
921
+-----------------+--------------------------------+--------------------------------+
923
+-----------------+--------------------------------+--------------------------------+
924
| cp857 | 857, IBM857 | Turkish |
925
+-----------------+--------------------------------+--------------------------------+
926
| cp860 | 860, IBM860 | Portuguese |
927
+-----------------+--------------------------------+--------------------------------+
928
| cp861 | 861, CP-IS, IBM861 | Icelandic |
929
+-----------------+--------------------------------+--------------------------------+
930
| cp862 | 862, IBM862 | Hebrew |
931
+-----------------+--------------------------------+--------------------------------+
932
| cp863 | 863, IBM863 | Canadian |
933
+-----------------+--------------------------------+--------------------------------+
934
| cp864 | IBM864 | Arabic |
935
+-----------------+--------------------------------+--------------------------------+
936
| cp865 | 865, IBM865 | Danish, Norwegian |
937
+-----------------+--------------------------------+--------------------------------+
938
| cp866 | 866, IBM866 | Russian |
939
+-----------------+--------------------------------+--------------------------------+
940
| cp869 | 869, CP-GR, IBM869 | Greek |
941
+-----------------+--------------------------------+--------------------------------+
943
+-----------------+--------------------------------+--------------------------------+
945
+-----------------+--------------------------------+--------------------------------+
946
| cp932 | 932, ms932, mskanji, ms-kanji | Japanese |
947
+-----------------+--------------------------------+--------------------------------+
948
| cp949 | 949, ms949, uhc | Korean |
949
+-----------------+--------------------------------+--------------------------------+
950
| cp950 | 950, ms950 | Traditional Chinese |
951
+-----------------+--------------------------------+--------------------------------+
953
+-----------------+--------------------------------+--------------------------------+
954
| cp1026 | ibm1026 | Turkish |
955
+-----------------+--------------------------------+--------------------------------+
956
| cp1140 | ibm1140 | Western Europe |
957
+-----------------+--------------------------------+--------------------------------+
958
| cp1250 | windows-1250 | Central and Eastern Europe |
959
+-----------------+--------------------------------+--------------------------------+
960
| cp1251 | windows-1251 | Bulgarian, Byelorussian, |
961
| | | Macedonian, Russian, Serbian |
962
+-----------------+--------------------------------+--------------------------------+
963
| cp1252 | windows-1252 | Western Europe |
964
+-----------------+--------------------------------+--------------------------------+
965
| cp1253 | windows-1253 | Greek |
966
+-----------------+--------------------------------+--------------------------------+
967
| cp1254 | windows-1254 | Turkish |
968
+-----------------+--------------------------------+--------------------------------+
969
| cp1255 | windows-1255 | Hebrew |
970
+-----------------+--------------------------------+--------------------------------+
971
| cp1256 | windows1256 | Arabic |
972
+-----------------+--------------------------------+--------------------------------+
973
| cp1257 | windows-1257 | Baltic languages |
974
+-----------------+--------------------------------+--------------------------------+
975
| cp1258 | windows-1258 | Vietnamese |
976
+-----------------+--------------------------------+--------------------------------+
977
| euc_jp | eucjp, ujis, u-jis | Japanese |
978
+-----------------+--------------------------------+--------------------------------+
979
| euc_jis_2004 | jisx0213, eucjis2004 | Japanese |
980
+-----------------+--------------------------------+--------------------------------+
981
| euc_jisx0213 | eucjisx0213 | Japanese |
982
+-----------------+--------------------------------+--------------------------------+
983
| euc_kr | euckr, korean, ksc5601, | Korean |
984
| | ks_c-5601, ks_c-5601-1987, | |
985
| | ksx1001, ks_x-1001 | |
986
+-----------------+--------------------------------+--------------------------------+
987
| gb2312 | chinese, csiso58gb231280, euc- | Simplified Chinese |
988
| | cn, euccn, eucgb2312-cn, | |
989
| | gb2312-1980, gb2312-80, iso- | |
991
+-----------------+--------------------------------+--------------------------------+
992
| gbk | 936, cp936, ms936 | Unified Chinese |
993
+-----------------+--------------------------------+--------------------------------+
994
| gb18030 | gb18030-2000 | Unified Chinese |
995
+-----------------+--------------------------------+--------------------------------+
996
| hz | hzgb, hz-gb, hz-gb-2312 | Simplified Chinese |
997
+-----------------+--------------------------------+--------------------------------+
998
| iso2022_jp | csiso2022jp, iso2022jp, | Japanese |
1000
+-----------------+--------------------------------+--------------------------------+
1001
| iso2022_jp_1 | iso2022jp-1, iso-2022-jp-1 | Japanese |
1002
+-----------------+--------------------------------+--------------------------------+
1003
| iso2022_jp_2 | iso2022jp-2, iso-2022-jp-2 | Japanese, Korean, Simplified |
1004
| | | Chinese, Western Europe, Greek |
1005
+-----------------+--------------------------------+--------------------------------+
1006
| iso2022_jp_2004 | iso2022jp-2004, | Japanese |
1007
| | iso-2022-jp-2004 | |
1008
+-----------------+--------------------------------+--------------------------------+
1009
| iso2022_jp_3 | iso2022jp-3, iso-2022-jp-3 | Japanese |
1010
+-----------------+--------------------------------+--------------------------------+
1011
| iso2022_jp_ext | iso2022jp-ext, iso-2022-jp-ext | Japanese |
1012
+-----------------+--------------------------------+--------------------------------+
1013
| iso2022_kr | csiso2022kr, iso2022kr, | Korean |
1015
+-----------------+--------------------------------+--------------------------------+
1016
| latin_1 | iso-8859-1, iso8859-1, 8859, | West Europe |
1017
| | cp819, latin, latin1, L1 | |
1018
+-----------------+--------------------------------+--------------------------------+
1019
| iso8859_2 | iso-8859-2, latin2, L2 | Central and Eastern Europe |
1020
+-----------------+--------------------------------+--------------------------------+
1021
| iso8859_3 | iso-8859-3, latin3, L3 | Esperanto, Maltese |
1022
+-----------------+--------------------------------+--------------------------------+
1023
| iso8859_4 | iso-8859-4, latin4, L4 | Baltic languages |
1024
+-----------------+--------------------------------+--------------------------------+
1025
| iso8859_5 | iso-8859-5, cyrillic | Bulgarian, Byelorussian, |
1026
| | | Macedonian, Russian, Serbian |
1027
+-----------------+--------------------------------+--------------------------------+
1028
| iso8859_6 | iso-8859-6, arabic | Arabic |
1029
+-----------------+--------------------------------+--------------------------------+
1030
| iso8859_7 | iso-8859-7, greek, greek8 | Greek |
1031
+-----------------+--------------------------------+--------------------------------+
1032
| iso8859_8 | iso-8859-8, hebrew | Hebrew |
1033
+-----------------+--------------------------------+--------------------------------+
1034
| iso8859_9 | iso-8859-9, latin5, L5 | Turkish |
1035
+-----------------+--------------------------------+--------------------------------+
1036
| iso8859_10 | iso-8859-10, latin6, L6 | Nordic languages |
1037
+-----------------+--------------------------------+--------------------------------+
1038
| iso8859_13 | iso-8859-13 | Baltic languages |
1039
+-----------------+--------------------------------+--------------------------------+
1040
| iso8859_14 | iso-8859-14, latin8, L8 | Celtic languages |
1041
+-----------------+--------------------------------+--------------------------------+
1042
| iso8859_15 | iso-8859-15 | Western Europe |
1043
+-----------------+--------------------------------+--------------------------------+
1044
| johab | cp1361, ms1361 | Korean |
1045
+-----------------+--------------------------------+--------------------------------+
1046
| koi8_r | | Russian |
1047
+-----------------+--------------------------------+--------------------------------+
1048
| koi8_u | | Ukrainian |
1049
+-----------------+--------------------------------+--------------------------------+
1050
| mac_cyrillic | maccyrillic | Bulgarian, Byelorussian, |
1051
| | | Macedonian, Russian, Serbian |
1052
+-----------------+--------------------------------+--------------------------------+
1053
| mac_greek | macgreek | Greek |
1054
+-----------------+--------------------------------+--------------------------------+
1055
| mac_iceland | maciceland | Icelandic |
1056
+-----------------+--------------------------------+--------------------------------+
1057
| mac_latin2 | maclatin2, maccentraleurope | Central and Eastern Europe |
1058
+-----------------+--------------------------------+--------------------------------+
1059
| mac_roman | macroman | Western Europe |
1060
+-----------------+--------------------------------+--------------------------------+
1061
| mac_turkish | macturkish | Turkish |
1062
+-----------------+--------------------------------+--------------------------------+
1063
| ptcp154 | csptcp154, pt154, cp154, | Kazakh |
1064
| | cyrillic-asian | |
1065
+-----------------+--------------------------------+--------------------------------+
1066
| shift_jis | csshiftjis, shiftjis, sjis, | Japanese |
1068
+-----------------+--------------------------------+--------------------------------+
1069
| shift_jis_2004 | shiftjis2004, sjis_2004, | Japanese |
1071
+-----------------+--------------------------------+--------------------------------+
1072
| shift_jisx0213 | shiftjisx0213, sjisx0213, | Japanese |
1074
+-----------------+--------------------------------+--------------------------------+
1075
| utf_32 | U32, utf32 | all languages |
1076
+-----------------+--------------------------------+--------------------------------+
1077
| utf_32_be | UTF-32BE | all languages |
1078
+-----------------+--------------------------------+--------------------------------+
1079
| utf_32_le | UTF-32LE | all languages |
1080
+-----------------+--------------------------------+--------------------------------+
1081
| utf_16 | U16, utf16 | all languages |
1082
+-----------------+--------------------------------+--------------------------------+
1083
| utf_16_be | UTF-16BE | all languages (BMP only) |
1084
+-----------------+--------------------------------+--------------------------------+
1085
| utf_16_le | UTF-16LE | all languages (BMP only) |
1086
+-----------------+--------------------------------+--------------------------------+
1087
| utf_7 | U7, unicode-1-1-utf-7 | all languages |
1088
+-----------------+--------------------------------+--------------------------------+
1089
| utf_8 | U8, UTF, utf8 | all languages |
1090
+-----------------+--------------------------------+--------------------------------+
1091
| utf_8_sig | | all languages |
1092
+-----------------+--------------------------------+--------------------------------+
1094
.. XXX fix here, should be in above table
1096
+--------------------+---------+---------------------------+
1097
| Codec | Aliases | Purpose |
1098
+====================+=========+===========================+
1099
| idna | | Implements :rfc:`3490`, |
1101
| | | :mod:`encodings.idna` |
1102
+--------------------+---------+---------------------------+
1103
| mbcs | dbcs | Windows only: Encode |
1104
| | | operand according to the |
1105
| | | ANSI codepage (CP_ACP) |
1106
+--------------------+---------+---------------------------+
1107
| palmos | | Encoding of PalmOS 3.5 |
1108
+--------------------+---------+---------------------------+
1109
| punycode | | Implements :rfc:`3492` |
1110
+--------------------+---------+---------------------------+
1111
| raw_unicode_escape | | Produce a string that is |
1112
| | | suitable as raw Unicode |
1113
| | | literal in Python source |
1115
+--------------------+---------+---------------------------+
1116
| undefined | | Raise an exception for |
1117
| | | all conversions. Can be |
1118
| | | used as the system |
1119
| | | encoding if no automatic |
1120
| | | coercion between byte and |
1121
| | | Unicode strings is |
1123
+--------------------+---------+---------------------------+
1124
| unicode_escape | | Produce a string that is |
1125
| | | suitable as Unicode |
1126
| | | literal in Python source |
1128
+--------------------+---------+---------------------------+
1129
| unicode_internal | | Return the internal |
1130
| | | representation of the |
1132
+--------------------+---------+---------------------------+
1135
:mod:`encodings.idna` --- Internationalized Domain Names in Applications
1136
------------------------------------------------------------------------
1138
.. module:: encodings.idna
1139
:synopsis: Internationalized Domain Names implementation
1140
.. moduleauthor:: Martin v. Löwis
1142
This module implements :rfc:`3490` (Internationalized Domain Names in
1143
Applications) and :rfc:`3492` (Nameprep: A Stringprep Profile for
1144
Internationalized Domain Names (IDN)). It builds upon the ``punycode`` encoding
1145
and :mod:`stringprep`.
1147
These RFCs together define a protocol to support non-ASCII characters in domain
1148
names. A domain name containing non-ASCII characters (such as
1149
``www.Alliancefrançaise.nu``) is converted into an ASCII-compatible encoding
1150
(ACE, such as ``www.xn--alliancefranaise-npb.nu``). The ACE form of the domain
1151
name is then used in all places where arbitrary characters are not allowed by
1152
the protocol, such as DNS queries, HTTP :mailheader:`Host` fields, and so
1153
on. This conversion is carried out in the application; if possible invisible to
1154
the user: The application should transparently convert Unicode domain labels to
1155
IDNA on the wire, and convert back ACE labels to Unicode before presenting them
1158
Python supports this conversion in several ways: The ``idna`` codec allows to
1159
convert between Unicode and the ACE. Furthermore, the :mod:`socket` module
1160
transparently converts Unicode host names to ACE, so that applications need not
1161
be concerned about converting host names themselves when they pass them to the
1162
socket module. On top of that, modules that have host names as function
1163
parameters, such as :mod:`http.client` and :mod:`ftplib`, accept Unicode host
1164
names (:mod:`http.client` then also transparently sends an IDNA hostname in the
1165
:mailheader:`Host` field if it sends that field at all).
1167
When receiving host names from the wire (such as in reverse name lookup), no
1168
automatic conversion to Unicode is performed: Applications wishing to present
1169
such host names to the user should decode them to Unicode.
1171
The module :mod:`encodings.idna` also implements the nameprep procedure, which
1172
performs certain normalizations on host names, to achieve case-insensitivity of
1173
international domain names, and to unify similar characters. The nameprep
1174
functions can be used directly if desired.
1177
.. function:: nameprep(label)
1179
Return the nameprepped version of *label*. The implementation currently assumes
1180
query strings, so ``AllowUnassigned`` is true.
1183
.. function:: ToASCII(label)
1185
Convert a label to ASCII, as specified in :rfc:`3490`. ``UseSTD3ASCIIRules`` is
1186
assumed to be false.
1189
.. function:: ToUnicode(label)
1191
Convert a label to Unicode, as specified in :rfc:`3490`.
1194
:mod:`encodings.utf_8_sig` --- UTF-8 codec with BOM signature
1195
-------------------------------------------------------------
1197
.. module:: encodings.utf_8_sig
1198
:synopsis: UTF-8 codec with BOM signature
1199
.. moduleauthor:: Walter Dörwald
1201
This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded
1202
BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this
1203
is only done once (on the first write to the byte stream). For decoding an
1204
optional UTF-8 encoded BOM at the start of the data will be skipped.
added
removed
Doc/library/codecs.rst
