1
:mod:`shlex` --- Simple lexical analysis
2
========================================
5
:synopsis: Simple lexical analysis for Unix shell-like languages.
6
.. moduleauthor:: Eric S. Raymond <esr@snark.thyrsus.com>
7
.. moduleauthor:: Gustavo Niemeyer <niemeyer@conectiva.com>
8
.. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com>
9
.. sectionauthor:: Gustavo Niemeyer <niemeyer@conectiva.com>
11
**Source code:** :source:`Lib/shlex.py`
15
The :class:`~shlex.shlex` class makes it easy to write lexical analyzers for
16
simple syntaxes resembling that of the Unix shell. This will often be useful
17
for writing minilanguages, (for example, in run control files for Python
18
applications) or for parsing quoted strings.
20
The :mod:`shlex` module defines the following functions:
23
.. function:: split(s, comments=False, posix=True)
25
Split the string *s* using shell-like syntax. If *comments* is :const:`False`
26
(the default), the parsing of comments in the given string will be disabled
27
(setting the :attr:`~shlex.commenters` attribute of the
28
:class:`~shlex.shlex` instance to the empty string). This function operates
29
in POSIX mode by default, but uses non-POSIX mode if the *posix* argument is
34
Since the :func:`split` function instantiates a :class:`~shlex.shlex`
35
instance, passing ``None`` for *s* will read the string to split from
39
.. function:: quote(s)
41
Return a shell-escaped version of the string *s*. The returned value is a
42
string that can safely be used as one token in a shell command line, for
43
cases where you cannot use a list.
45
This idiom would be unsafe::
47
>>> filename = 'somefile; rm -rf ~'
48
>>> command = 'ls -l {}'.format(filename)
49
>>> print(command) # executed by a shell: boom!
50
ls -l somefile; rm -rf ~
52
:func:`quote` lets you plug the security hole::
54
>>> command = 'ls -l {}'.format(quote(filename))
56
ls -l 'somefile; rm -rf ~'
57
>>> remote_command = 'ssh home {}'.format(quote(command))
58
>>> print(remote_command)
59
ssh home 'ls -l '"'"'somefile; rm -rf ~'"'"''
61
The quoting is compatible with UNIX shells and with :func:`split`:
63
>>> remote_command = split(remote_command)
65
['ssh', 'home', "ls -l 'somefile; rm -rf ~'"]
66
>>> command = split(remote_command[-1])
68
['ls', '-l', 'somefile; rm -rf ~']
72
The :mod:`shlex` module defines the following class:
75
.. class:: shlex(instream=None, infile=None, posix=False)
77
A :class:`~shlex.shlex` instance or subclass instance is a lexical analyzer
78
object. The initialization argument, if present, specifies where to read
79
characters from. It must be a file-/stream-like object with
80
:meth:`~io.TextIOBase.read` and :meth:`~io.TextIOBase.readline` methods, or
81
a string. If no argument is given, input will be taken from ``sys.stdin``.
82
The second optional argument is a filename string, which sets the initial
83
value of the :attr:`~shlex.infile` attribute. If the *instream*
84
argument is omitted or equal to ``sys.stdin``, this second argument
85
defaults to "stdin". The *posix* argument defines the operational mode:
86
when *posix* is not true (default), the :class:`~shlex.shlex` instance will
87
operate in compatibility mode. When operating in POSIX mode,
88
:class:`~shlex.shlex` will try to be as close as possible to the POSIX shell
94
Module :mod:`configparser`
95
Parser for configuration files similar to the Windows :file:`.ini` files.
103
A :class:`~shlex.shlex` instance has the following methods:
106
.. method:: shlex.get_token()
108
Return a token. If tokens have been stacked using :meth:`push_token`, pop a
109
token off the stack. Otherwise, read one from the input stream. If reading
110
encounters an immediate end-of-file, :attr:`eof` is returned (the empty
111
string (``''``) in non-POSIX mode, and ``None`` in POSIX mode).
114
.. method:: shlex.push_token(str)
116
Push the argument onto the token stack.
119
.. method:: shlex.read_token()
121
Read a raw token. Ignore the pushback stack, and do not interpret source
122
requests. (This is not ordinarily a useful entry point, and is documented here
123
only for the sake of completeness.)
126
.. method:: shlex.sourcehook(filename)
128
When :class:`~shlex.shlex` detects a source request (see :attr:`source`
129
below) this method is given the following token as argument, and expected
130
to return a tuple consisting of a filename and an open file-like object.
132
Normally, this method first strips any quotes off the argument. If the result
133
is an absolute pathname, or there was no previous source request in effect, or
134
the previous source was a stream (such as ``sys.stdin``), the result is left
135
alone. Otherwise, if the result is a relative pathname, the directory part of
136
the name of the file immediately before it on the source inclusion stack is
137
prepended (this behavior is like the way the C preprocessor handles ``#include
140
The result of the manipulations is treated as a filename, and returned as the
141
first component of the tuple, with :func:`open` called on it to yield the second
142
component. (Note: this is the reverse of the order of arguments in instance
145
This hook is exposed so that you can use it to implement directory search paths,
146
addition of file extensions, and other namespace hacks. There is no
147
corresponding 'close' hook, but a shlex instance will call the
148
:meth:`~io.IOBase.close` method of the sourced input stream when it returns
151
For more explicit control of source stacking, use the :meth:`push_source` and
152
:meth:`pop_source` methods.
155
.. method:: shlex.push_source(newstream, newfile=None)
157
Push an input source stream onto the input stack. If the filename argument is
158
specified it will later be available for use in error messages. This is the
159
same method used internally by the :meth:`sourcehook` method.
162
.. method:: shlex.pop_source()
164
Pop the last-pushed input source from the input stack. This is the same method
165
used internally when the lexer reaches EOF on a stacked input stream.
168
.. method:: shlex.error_leader(infile=None, lineno=None)
170
This method generates an error message leader in the format of a Unix C compiler
171
error label; the format is ``'"%s", line %d: '``, where the ``%s`` is replaced
172
with the name of the current source file and the ``%d`` with the current input
173
line number (the optional arguments can be used to override these).
175
This convenience is provided to encourage :mod:`shlex` users to generate error
176
messages in the standard, parseable format understood by Emacs and other Unix
179
Instances of :class:`~shlex.shlex` subclasses have some public instance
180
variables which either control lexical analysis or can be used for debugging:
183
.. attribute:: shlex.commenters
185
The string of characters that are recognized as comment beginners. All
186
characters from the comment beginner to end of line are ignored. Includes just
190
.. attribute:: shlex.wordchars
192
The string of characters that will accumulate into multi-character tokens. By
193
default, includes all ASCII alphanumerics and underscore.
196
.. attribute:: shlex.whitespace
198
Characters that will be considered whitespace and skipped. Whitespace bounds
199
tokens. By default, includes space, tab, linefeed and carriage-return.
202
.. attribute:: shlex.escape
204
Characters that will be considered as escape. This will be only used in POSIX
205
mode, and includes just ``'\'`` by default.
208
.. attribute:: shlex.quotes
210
Characters that will be considered string quotes. The token accumulates until
211
the same quote is encountered again (thus, different quote types protect each
212
other as in the shell.) By default, includes ASCII single and double quotes.
215
.. attribute:: shlex.escapedquotes
217
Characters in :attr:`quotes` that will interpret escape characters defined in
218
:attr:`escape`. This is only used in POSIX mode, and includes just ``'"'`` by
222
.. attribute:: shlex.whitespace_split
224
If ``True``, tokens will only be split in whitespaces. This is useful, for
225
example, for parsing command lines with :class:`~shlex.shlex`, getting
226
tokens in a similar way to shell arguments.
229
.. attribute:: shlex.infile
231
The name of the current input file, as initially set at class instantiation time
232
or stacked by later source requests. It may be useful to examine this when
233
constructing error messages.
236
.. attribute:: shlex.instream
238
The input stream from which this :class:`~shlex.shlex` instance is reading
242
.. attribute:: shlex.source
244
This attribute is ``None`` by default. If you assign a string to it, that
245
string will be recognized as a lexical-level inclusion request similar to the
246
``source`` keyword in various shells. That is, the immediately following token
247
will opened as a filename and input taken from that stream until EOF, at which
248
point the :meth:`~io.IOBase.close` method of that stream will be called and
249
the input source will again become the original input stream. Source
250
requests may be stacked any number of levels deep.
253
.. attribute:: shlex.debug
255
If this attribute is numeric and ``1`` or more, a :class:`~shlex.shlex`
256
instance will print verbose progress output on its behavior. If you need
257
to use this, you can read the module source code to learn the details.
260
.. attribute:: shlex.lineno
262
Source line number (count of newlines seen so far plus one).
265
.. attribute:: shlex.token
267
The token buffer. It may be useful to examine this when catching exceptions.
270
.. attribute:: shlex.eof
272
Token used to determine end of file. This will be set to the empty string
273
(``''``), in non-POSIX mode, and to ``None`` in POSIX mode.
276
.. _shlex-parsing-rules:
281
When operating in non-POSIX mode, :class:`~shlex.shlex` will try to obey to the
284
* Quote characters are not recognized within words (``Do"Not"Separate`` is
285
parsed as the single word ``Do"Not"Separate``);
287
* Escape characters are not recognized;
289
* Enclosing characters in quotes preserve the literal value of all characters
292
* Closing quotes separate words (``"Do"Separate`` is parsed as ``"Do"`` and
295
* If :attr:`~shlex.whitespace_split` is ``False``, any character not
296
declared to be a word character, whitespace, or a quote will be returned as
297
a single-character token. If it is ``True``, :class:`~shlex.shlex` will only
298
split words in whitespaces;
300
* EOF is signaled with an empty string (``''``);
302
* It's not possible to parse empty strings, even if quoted.
304
When operating in POSIX mode, :class:`~shlex.shlex` will try to obey to the
305
following parsing rules.
307
* Quotes are stripped out, and do not separate words (``"Do"Not"Separate"`` is
308
parsed as the single word ``DoNotSeparate``);
310
* Non-quoted escape characters (e.g. ``'\'``) preserve the literal value of the
311
next character that follows;
313
* Enclosing characters in quotes which are not part of
314
:attr:`~shlex.escapedquotes` (e.g. ``"'"``) preserve the literal value
315
of all characters within the quotes;
317
* Enclosing characters in quotes which are part of
318
:attr:`~shlex.escapedquotes` (e.g. ``'"'``) preserves the literal value
319
of all characters within the quotes, with the exception of the characters
320
mentioned in :attr:`~shlex.escape`. The escape characters retain its
321
special meaning only when followed by the quote in use, or the escape
322
character itself. Otherwise the escape character will be considered a
325
* EOF is signaled with a :const:`None` value;
327
* Quoted empty strings (``''``) are allowed.
added
removed
Doc/library/shlex.rst
