2
Title: Docstring Processing System Framework
3
Version: $Revision: 3398 $
4
Last-Modified: $Date: 2005-05-28 03:22:12 +0200 (Sat, 28 May 2005) $
5
Author: David Goodger <goodger@users.sourceforge.net>
6
Discussions-To: <doc-sig@python.org>
9
Content-Type: text/x-rst
11
Post-History: 13-Jun-2001
17
Python lends itself to inline documentation. With its built-in
18
docstring syntax, a limited form of `Literate Programming`_ is easy to
19
do in Python. However, there are no satisfactory standard tools for
20
extracting and processing Python docstrings. The lack of a standard
21
toolset is a significant gap in Python's infrastructure; this PEP aims
24
The issues surrounding docstring processing have been contentious and
25
difficult to resolve. This PEP proposes a generic Docstring
26
Processing System (DPS) framework, which separates out the components
27
(program and conceptual), enabling the resolution of individual issues
28
either through consensus (one solution) or through divergence (many).
29
It promotes standard interfaces which will allow a variety of plug-in
30
components (input context readers, markup parsers, and output format
33
The concepts of a DPS framework are presented independently of
34
implementation details.
37
Road Map to the Docstring PEPs
38
==============================
40
There are many aspects to docstring processing. The "Docstring PEPs"
41
have broken up the issues in order to deal with each of them in
42
isolation, or as close as possible. The individual aspects and
43
associated PEPs are as follows:
45
* Docstring syntax. PEP 287, "reStructuredText Docstring Format"
46
[#PEP-287]_, proposes a syntax for Python docstrings, PEPs, and
49
* Docstring semantics consist of at least two aspects:
51
- Conventions: the high-level structure of docstrings. Dealt with
52
in PEP 257, "Docstring Conventions" [#PEP-257]_.
54
- Methodology: rules for the informational content of docstrings.
57
* Processing mechanisms. This PEP (PEP 256) outlines the high-level
58
issues and specification of an abstract docstring processing system
59
(DPS). PEP 258, "Docutils Design Specification" [#PEP-258]_, is an
60
overview of the design and implementation of one DPS under
63
* Output styles: developers want the documentation generated from
64
their source code to look good, and there are many different ideas
65
about what that means. PEP 258 touches on "Stylist Transforms".
66
This aspect of docstring processing has yet to be fully explored.
68
By separating out the issues, we can form consensus more easily
69
(smaller fights ;-), and accept divergence more readily.
75
There are standard inline documentation systems for some other
76
languages. For example, Perl has POD_ ("Plain Old Documentation") and
77
Java has Javadoc_, but neither of these mesh with the Pythonic way.
78
POD syntax is very explicit, but takes after Perl in terms of
79
readability. Javadoc is HTML-centric; except for "``@field``" tags,
80
raw HTML is used for markup. There are also general tools such as
81
Autoduck_ and Web_ (Tangle & Weave), useful for multiple languages.
83
There have been many attempts to write auto-documentation systems
84
for Python (not an exhaustive list):
86
- Marc-Andre Lemburg's doc.py_
88
- Daniel Larsson's pythondoc_ & gendoc_
90
- Doug Hellmann's HappyDoc_
92
- Laurence Tratt's Crystal (no longer available on the web)
94
- Ka-Ping Yee's pydoc_ (pydoc.py is now part of the Python standard
97
- Tony Ibbs' docutils_ (Tony has donated this name to the `Docutils
100
- Edward Loper's STminus_ formalization and related efforts
102
These systems, each with different goals, have had varying degrees of
103
success. A problem with many of the above systems was over-ambition
104
combined with inflexibility. They provided a self-contained set of
105
components: a docstring extraction system, a markup parser, an
106
internal processing system and one or more output format writers with
107
a fixed style. Inevitably, one or more aspects of each system had
108
serious shortcomings, and they were not easily extended or modified,
109
preventing them from being adopted as standard tools.
111
It has become clear (to this author, at least) that the "all or
112
nothing" approach cannot succeed, since no monolithic self-contained
113
system could possibly be agreed upon by all interested parties. A
114
modular component approach designed for extension, where components
115
may be multiply implemented, may be the only chance for success.
116
Standard inter-component APIs will make the DPS components
117
comprehensible without requiring detailed knowledge of the whole,
118
lowering the barrier for contributions, and ultimately resulting in a
119
rich and varied system.
121
Each of the components of a docstring processing system should be
122
developed independently. A "best of breed" system should be chosen,
123
either merged from existing systems, and/or developed anew. This
124
system should be included in Python's standard library.
127
PyDoc & Other Existing Systems
128
------------------------------
130
PyDoc became part of the Python standard library as of release 2.1.
131
It extracts and displays docstrings from within the Python interactive
132
interpreter, from the shell command line, and from a GUI window into a
133
web browser (HTML). Although a very useful tool, PyDoc has several
134
deficiencies, including:
136
- In the case of the GUI/HTML, except for some heuristic hyperlinking
137
of identifier names, no formatting of the docstrings is done. They
138
are presented within ``<p><small><tt>`` tags to avoid unwanted line
139
wrapping. Unfortunately, the result is not attractive.
141
- PyDoc extracts docstrings and structural information (class
142
identifiers, method signatures, etc.) from imported module objects.
143
There are security issues involved with importing untrusted code.
144
Also, information from the source is lost when importing, such as
145
comments, "additional docstrings" (string literals in non-docstring
146
contexts; see PEP 258 [#PEP-258]_), and the order of definitions.
148
The functionality proposed in this PEP could be added to or used by
149
PyDoc when serving HTML pages. The proposed docstring processing
150
system's functionality is much more than PyDoc needs in its current
151
form. Either an independent tool will be developed (which PyDoc may
152
or may not use), or PyDoc could be expanded to encompass this
153
functionality and *become* the docstring processing system (or one
154
such system). That decision is beyond the scope of this PEP.
156
Similarly for other existing docstring processing systems, their
157
authors may or may not choose compatibility with this framework.
158
However, if this framework is accepted and adopted as the Python
159
standard, compatibility will become an important consideration in
160
these systems' future.
166
The docstring processing system framework is broken up as follows:
168
1. Docstring conventions. Documents issues such as:
170
- What should be documented where.
172
- First line is a one-line synopsis.
174
PEP 257 [#PEP-257]_ documents some of these issues.
176
2. Docstring processing system design specification. Documents
179
- High-level spec: what a DPS does.
181
- Command-line interface for executable script.
185
- Docstring extraction rules.
187
- Readers, which encapsulate the input context.
191
- Document tree: the intermediate internal data structure. The
192
output of the Parser and Reader, and the input to the Writer all
193
share the same data structure.
195
- Transforms, which modify the document tree.
197
- Writers for output formats.
199
- Distributors, which handle output management (one file, many
200
files, or objects in memory).
202
These issues are applicable to any docstring processing system
203
implementation. PEP 258 [#PEP-258]_ documents these issues.
205
3. Docstring processing system implementation.
207
4. Input markup specifications: docstring syntax. PEP 287 [#PEP-287]_
208
proposes a standard syntax.
210
5. Input parser implementations.
212
6. Input context readers ("modes": Python source code, PEP, standalone
213
text file, email, etc.) and implementations.
215
7. Stylists: certain input context readers may have associated
216
stylists which allow for a variety of output document styles.
218
8. Output formats (HTML, XML, TeX, DocBook, info, etc.) and writer
221
Components 1, 2/3/5, and 4 are the subject of individual companion
222
PEPs. If there is another implementation of the framework or
223
syntax/parser, additional PEPs may be required. Multiple
224
implementations of each of components 6 and 7 will be required; the
225
PEP mechanism may be overkill for these components.
231
A SourceForge project has been set up for this work at
232
http://docutils.sourceforge.net/.
235
References and Footnotes
236
========================
238
.. [#PEP-287] PEP 287, reStructuredText Docstring Format, Goodger
239
(http://www.python.org/peps/pep-0287.html)
241
.. [#PEP-257] PEP 257, Docstring Conventions, Goodger, Van Rossum
242
(http://www.python.org/peps/pep-0257.html)
244
.. [#PEP-258] PEP 258, Docutils Design Specification, Goodger
245
(http://www.python.org/peps/pep-0258.html)
247
.. _Literate Programming: http://www.literateprogramming.com/
249
.. _POD: http://perldoc.perl.org/perlpod.html
251
.. _Javadoc: http://java.sun.com/j2se/javadoc/
254
http://www.helpmaster.com/hlp-developmentaids-autoduck.htm
256
.. _Web: http://www-cs-faculty.stanford.edu/~knuth/cweb.html
259
http://www.egenix.com/files/python/SoftwareDescriptions.html#doc.py
262
.. _gendoc: http://starship.python.net/crew/danilo/pythondoc/
264
.. _HappyDoc: http://happydoc.sourceforge.net/
266
.. _pydoc: http://www.python.org/doc/current/lib/module-pydoc.html
268
.. _docutils: http://www.tibsnjoan.co.uk/docutils.html
270
.. _Docutils project: http://docutils.sourceforge.net/
272
.. _STMinus: http://www.cis.upenn.edu/~edloper/pydoc/
274
.. _Python Doc-SIG: http://www.python.org/sigs/doc-sig/
280
This document has been placed in the public domain.
286
This document borrows ideas from the archives of the `Python
287
Doc-SIG`_. Thanks to all members past & present.
294
indent-tabs-mode: nil
295
sentence-end-double-space: t
added
removed
docs/peps/pep-0256.txt
