2
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
3
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
5
<html xmlns="http://www.w3.org/1999/xhtml">
8
<title>The Lore Document Generation Framework</title>
13
<h1>The Lore Document Generation Framework</h1>
17
<a href="mailto:moshez@twistedmatrix.com">moshez@twistedmatrix.com</a></li>
19
<a href="mailto:spiv@twistedmatrix.com">spiv@twistedmatrix.com</a></li>
24
<p>Lore is a documentation generation system which uses a limited subset
25
of XHTML, together with some class attributes, as its source format. This
26
allows for lower barrier of entry than many other similar systems, since HTML
27
authoring tools are plentiful
28
as is knowledge of HTML writing. As an added advantage, the source format
29
is viewable directly, so that even if Lore is not available the documentation
30
is useful. It currently outputs LaTeX and HTML, which allows for most
33
<p>Lore is currently in use by the Twisted project to generate its
34
documentation for versions 1.0.1 and above.</p>
38
<p>At the beginning of Twisted's life cycle, as with any self-respecting
39
free software project, it came completely devoid of documentation. As
40
Twisted progressed in maturity, the Twisted development team realized
41
that documentation is necessary.</p>
43
<p>Since at that time the Twisted development
44
team did not want the overhead of integrating
45
a full-scale document generation framework into its build infrastructure,
46
documents were written for the least common denominator -- plain HTML.
47
When the Twisted team wanted the documentation to be
48
featured on the web site, it was desirable to have them integrated with
49
the web site's look and feel. Thus, <code class="shell">generate-domdocs</code>
50
was born as a simple XML-based command line hack which improved the look of the
51
documents so they would share the look and feel of the other pages in the web
52
site, including a standard header and footer. As
53
<code class="shell">generate-domdocs</code>
54
slowly grew more and more features, it gradually became too large to maintain.
55
The authors, members of the Twisted development team, decided that in order to
56
make it more maintainable, it should be refactored into a
57
library and by the way also add alternate output formats. Some of the documents
58
which were reluctant to be transformed into alternate formats were fixed,
59
and guidelines for making compatible documents were drafted. Those documents,
60
together with the conversion code, are the Lore documentation generation
65
<p>Lore is documentation generation system which is a part of the
66
<a href="http://twistedmatrix.com">Twisted</a> framework. It uses
67
the Twisted XML parsing framework
68
(<code class="API" base="twisted.web">microdom</code>) to parse compliant XHTML
69
and generate the various output formats from it.</p>
71
<p>Lore consists of a Python package, <code class="API">twisted.lore</code>,
72
and a command-line program: <code class="shell">lore</code>, which
73
generates HTML output (which is more presentation-oriented than the source
74
format), LaTeX or runs an linter, depending on command-line arguments.</p>
76
<p>In the case where the default output of Lore is not exactly suited to a
78
it is possible to subclass the output generators and customize their behavior.
79
This could be done for many purposes, from straight-forward additions like
80
adding a new <code>span</code> or <code>div</code> class to advanced tweaking
81
such as changing the way Lore does image conversion on LaTeX output.</p>
83
<p>Lore uses reflection intensively to make adding new features as simple
84
as adding a new method, without the need for awkward registration schemes.
85
Thus, adding another check to the linter or letting
86
Lore handle the <code>link</code> element in some way require only the addition
91
<p>Lore was written when the Twisted team felt it needed to write documentation
92
and looked for a documentation format. Looking through alternatives, the
93
best one seemed to be the Python way, using LaTeX format and
94
<code class="shell">latex2html</code>. However, the Python way has its share
95
of problems, not the least of which is <code class="shell">latex2html</code>
96
being a long and crufty Perl program whose Perl APIs, which are the
97
only way to add support for custom markup, change every version.</p>
99
<p>Since documentation writing is important, a documentation system with
100
minimal impact on the writer would be desirable. While LaTeX certainly has
101
very little impact in terms of markup overhead, it has a very big impact
102
both in terms of installed base (installing LaTeX on UNIX systems or
103
Windows is non-trivial at best) and in terms of familiarity.</p>
105
<p>HTML has the benefit of being directly readable on every post-1995
106
computer, so the installed base is as big as could be hoped for. It also has
107
the benefit of being easily parsed, at least in its new XHTML guise.</p>
109
<p>The goals of Lore were taken to be:</p>
112
<li>Source files directly readable.</li>
113
<li>At least output to modern (CSS-based) HTML.</li>
114
<li>Easily parsed by third-parties.</li>
117
<h2>Source Format</h2>
121
<p>Lore's source format is a subset of XHTML; all Lore source documents are
122
valid XHTML documents. The XHTML tags that Lore allows are:
123
<code>html</code>, <code>title</code>, <code>head</code>, <code>body</code>,
124
<code>h1</code>, <code>h2</code>, <code>h3</code>, <code>ol</code>,
125
<code>ul</code>, <code>dl</code>, <code>li</code>, <code>dt</code>,
126
<code>dd</code>, <code>p</code>, <code>code</code>, <code>img</code>,
127
<code>blockquote</code>, <code>a</code>, <code>cite</code>, <code>div</code>,
128
<code>span</code>, <code>strong</code>, <code>em</code>, <code>pre</code>,
129
<code>q</code>, <code>table</code>, <code>tr</code>, <code>td</code>,
130
<code>th</code> and <code>style</code>.
133
<p>We would like to stress the omission of the <code>font</code> tag (which is
134
deprecated in HTML 4.01 anyway). Instead of using <code>font</code>, Lore
135
mandates the use of stylesheets
136
and the <code>class</code> attribute, and in particular Lore defines several
137
classes, such as <code>footnote</code>, <code>API</code>,
138
<code>py-listing</code>. The use of classes on <code>div</code> and
139
<code>span</code> elements effectively allows XHTML to be arbitrarily
140
extensible without needing to define custom tags.</p>
142
<p>Further discouraging explicit style decision, Lore deprecates the
143
<code>style</code> attribute which allowing HTML (and XHTML) authors to embed
144
pieces of the stylesheet in the document. Though Lore properly processes
145
such documents, they are against the specification of Lore -- and
146
the Lore lint-like problem finder will complain.</p>
148
<h3>Advantages and Disadvantages</h3>
150
<p>Requiring XHTML rather than just HTML greatly simplifies the code to
151
manipulate Lore source, because we can use standard XML libraries. For
152
documentation authors, the difference is negligible -- and any mistakes made in
153
balancing tags can be easily found using the linter.
154
Since tag balancing problems, in many cases, cause a discrepancy between
155
author intention and the result, it is better to balance the tags anyway.</p>
157
<p>Like LaTeX, Lore encourages authors to focus on content, letting the
158
presentation take care of itself. This is an inherently restrictive approach,
159
but results in much more consistent and higher-quality output.</p>
161
<p>The Lore source format is quite usable (if somewhat plain) as an end-format.
162
Any web browser can read it, and it does not require special stylesheet support,
163
JavaScript or any other modern HTML additions. It is also, as intended,
164
straightforward to create and edit documents in this format.</p>
166
<p>However, reading the source format directly has some major limitations,
167
which are inherent in the combination of the facilities which render HTML
168
and the requirement that the format will be easily writable, and easy to
169
modify, using any standard text editor.
170
The limitations include:</p>
173
<li>There is no table of contents.</li>
174
<li>Footnotes interrupt the flow of text (although stylesheet tricks can
175
alleviate this to an extent).</li>
176
<li>Python source is not syntax highlighted.</li>
177
<li>File inclusions are implemented as hyper-links.</li>
180
<h2>Output Formats</h2>
182
<p>The two most important formats, for the end-user, are the computer screen and
183
pages of print outs. Any other format should be first and foremost be thought
184
of as a prelude to these final formats.</p>
186
<p>The easiest computer-screen oriented format is HTML. However, the HTML
187
which is most comfortable and useful to the end-user is not necessarily
188
easy to write and modify.
189
For example, it is painful to manually write a table of contents, and even more
190
painful to keep it updated as sections are added, removed or changed. However,
191
when reading a long document having a table of contents, with hyperlinks
192
into the sections, is a boon.
193
Thus, even though both Lore's source and one output format are HTML, an
194
HTML to HTML conversion is still necessary, paradoxical though it may sound.</p>
196
<p>For printable output, the most widely supported formats are PostScript
197
and Portable Document Format. On UNIX systems PostScript is often preferred,
198
since there are many tools for manipulating it and printing it (and PostScript
199
printers are more common in the UNIX world). On Windows and Apple computers,
200
Portable Document Format (PDF) is preferred because of the ease of installation
201
of the necessary tools. Mac OS X, though being technically a UNIX, supports
204
<p>Directly generating PostScript or PDF, however, is hard. Since these formats
205
are very low-level, the application generating them must do the hard work
206
of calculating line breaks, guessing hyphenation points and deciding on fonts.
207
Since these tasks are already implemented by LaTeX, Lore just generates LaTeX
208
code and lets the user run LaTeX to generate PostScript and
209
<code class="shell">ps2pdf</code> to generate PDF. Granted, this still causes
210
the problems with the difficulties of installing LaTeX. It is
211
possible to implement direct Lore to PDF converter, though this hasn't been
212
done yet, by using <code>pdflib</code>.</p>
216
<p>The HTML to HTML converter works by running a series of transformations on
217
the Document Object Model (DOM) tree of the parsed document, and then
218
writing it out. The most important transformation is that of throwing
219
away anything outside the <code>body</code> element, and putting the
220
<code>body</code> element inside a template file. This allows large
221
parts of the common layout code to be customized without modifying or writing
224
<p>Each step is implemented as a separate function, to allow Lore-using
225
Python programmers to customize which tree transformations to do in their
226
own code, without forcing them to rewrite functionality in Lore. In addition,
227
other output generators might perform a subset of these transformations
228
on the input tree before processing it -- and indeed, this is being used
229
even in Lore itself.</p>
231
<p>One of the steps taken is caused by a need which is common in large
232
Python frameworks: many of the class or module names are deeply nested,
233
but are commonly referred to by just their last one or two components
234
in writing. However, the user would like to know the full name of the
235
class or module name, and where to look up the API documentation -- but
236
without having the complete name thrust upon him during the flow of text
237
each time the module is mentioned.</p>
239
<p>Lore makes sure that each class or module name which is mentioned will
240
appear at least once using its full name, and afterwards use a common
241
short name, regardless of how the author wrote it up. This frees authors
242
from needing to observe, manually, this useful rule in their documents.</p>
244
<p>The HTML Lore outputs aims to be the poster boy of graceful degradation.
245
Thus, for example, while footnotes always appear as hyper-links to the footnote
246
text, browsers which respect the <code>title</code> attribute (which is usually
247
rendered as a tooltip) will also show the beginning of the footnote while
248
hovering above the hyper-link.</p>
250
<p>Lore avoids using the <q>font</q> or <q>color</q> tags and attributes,
251
preferring to use HTML classes and using a stylesheet to specify graphical
252
design decisions. This allows the Lore user to customize the presentation of
253
the output without touching Python code. Since most often the stylesheet
254
link is found in the <code>head</code> element, this is determined by
255
the by the template.</p>
257
<p>Lore uses the same approach even for syntax-highlighting Python code,
258
generating such elements as
259
<code><span class="keyword">if</span></code>.</p>
263
<p>The LaTeX home page describes LaTeX as a <q>high-quality typesetting system,
264
with features designed for the production of technical and scientific
265
documentation.</q> LaTeX is very popular for generating printable content,
266
building on Donald Knuth's TeX system to generate nearly optimal output
267
by putting together much of the typesetting industry's experience in the
268
form of a program and adding sophisticated algorithms for line-breaking and
271
<p>It is very common for document generation systems to avoid generating
272
printable output themselves, instead letting LaTeX do the hard work, and
273
Lore is no exception.</p>
275
<p>Lore can output LaTeX in two modes: article mode, in which it generates
276
a complete article ready to be be processed, and a section mode in which
277
it generates a LaTeX file whose top-level element is a section. Such a file
278
is usually included in some other LaTeX file via the include mechanism.
279
Twisted itself uses mainly the section mode, and includes everything in the
280
file <code class="shell">book.tex</code>, which is later processed to generate
281
the Twisted book.</p>
283
<p>While, conceivably, other modes could be done (a chapter mode or a subsection
284
mode) there has not been any demand for those. In the case of demand, supplying
285
these would be very few lines of Python code (less than 10), which can even
286
be done by subclassing existing classes and avoiding the modification of Lore
291
<p>Docbook output is currently experimental. Its chief use to Lore would
292
be in generating Texinfo, which is the source for the GNU info documentation
297
<p>Very early in the Lore development life-cycle it was found that a good
298
Lint-like tool is necessary to find errors without necessitating a full
299
compilation to all formats and sometimes even browsing the results. Because
300
Lore was written to accommodate a large set of already existing documents
301
(which were not previously checked for potential problems), such a tool
302
was very useful so that finding a problem in one document would not mean
303
this problem needs to be manually searched, and corrected, in all the other
306
<p>Lore's linter tries to find problems in documents
307
that would either stop the conversion to other formats by Lore completely
308
(for example, by being not well-formed XML), or that would make it less useful
309
(for example, by warning about tags or classes that are not supported by
312
<p>The linter even detects more exotic problems,
315
<li><code>pre</code> elements containing lines over 80 characters. Long lines
316
can be ugly to render in some output formats, and even impossible to
317
render in others.</li>
318
<li>Explicit use of the <code>"</code> character in a non-pre or non-code
319
environment. This makes a big difference for high-quality typographical
320
output targets like LaTeX, which
321
have distinct left- and right-quote characters.</li>
322
<li>Python code that isn't syntactically valid, with a bit of magic to account
328
This check caught a surprisingly large number of errors in the Twisted
330
<li><code>h1</code> contents being equal to <code>title</code> contents.
331
HTML is somewhat unique in that it has two places to specify the logical
332
idea of <q>title</q>. Since other output formats do not support that,
333
in Lore papers, the contents of both must be the same.</li>
336
<p>Since many of the incremental improvements done to Lore found a problem
337
in the existing documentation files, the linter has been
338
an important part of the Lore development effort. One may even argue that
339
part of the reason other documentation generation systems produce suboptimal
340
output for their <q>non-native</q> application is the lack of a linting
343
<p>Finally, if the linter gives a false positive, that is
344
it emits a warning for something that isn't a problem in a particular situation,
345
the user can add an <code>hlint="off"</code> attribute to the offending tag, and
346
the linter will ignore it. This is necessary only very rarely.</p>
348
<p>The chief design decision made in the linter, after
349
painful experience when running <code class="shell">tidy</code>, is that
350
<em>it must never change the document</em>. Thus, while the linter
351
will be as pedantic as possible finding
352
errors, it never changes the contents. This is particularly important
353
when dealing with version control systems, where spurious changes can
354
render <code class="shell">diff</code> listings useless.</p>
358
<h3>Python Syntax Highlighting</h3>
360
<p>All existing syntax highlighters for Python used pre-<code>tokenize</code>
361
techniques to analyse the Python code. As a result, they were cumbersome
362
and non-standard. The Lore developers decided that writing a Python
363
HTML syntax-highlighter would be easier than modifying one of the existing
364
ones. A syntax-highlighter was built on top of a null-tokenizer: that is,
365
a tokenizer which emits the <em>exact same</em> characters as the input.
366
This allowed easy debugging of the parsing code.</p>
368
<p>The only non-trivial code in the syntax highlighter is when dealing
369
with whitespace which is not significant syntactically, since the tokenizer
370
does not report it. However, since the tokenizer does report row and column,
371
when the code sees a discrepancy between where the previous token ended
372
and the current token starts, it adds whitespace to make up for
375
<p>When writing out the HTML, the only difference between that and the
376
null-tokenizer is the wrapping of each token by a <code>span</code>
377
tag with the appropriate class and escaping.</p>
379
<p>Note that the basic Python tokenizer does not distinguish between the
380
various roles of the production <q>NAME</q> (that is, a string of alphanumeric
382
underscore characters starting with an underscore or a letter) in Python.
383
The tokenizer Lore uses adds that information by having a simple state machine:
384
if the word is a keyword, there is nothing to be determined; otherwise, it
385
depends on the last detected name -- <code>class</code> or
386
<code>def</code> mean it is a function or class names, and after a
387
<code>class</code>/<code>def</code> and until a <code>:</code>, everything
388
is a <q>parameter</q> or a superclass.</p>
390
<p>The Python syntax highlighter Lore uses can be found in the
391
<code class="API">twisted.python.htmlizer</code>.</p>
393
<h3>File Inclusion</h3>
395
<p>Often, when writing detailed documents, the author wishes to test his
396
examples or even use examples from a working project. Pasting such examples
397
directly into the HTML has both the usual problems of pasting code -- the
398
version in the document will not benefit from bug fixes or enhancement to
399
the original version -- and the problem that the HTML needs proper escaping,
400
which is a tedious and error-prone procedure if done manually.
401
Both problems are solved by Lore's <q>listing</q> mechanism. The
402
<q>listing</q> mechanism converts HTML such as</p>
405
<a href="foo.py" class="py-listing>foo.py</a>
408
<p>into inclusion of the <code class="shell">foo.py</code> file. It will always
409
be properly escaped for whatever output format. It will
410
also be syntax-highlighted, just as if it had been included verbatim.</p>
412
<p>A similar class, <code>html-listing</code> is available for inclusion
415
<h3>API Reference Links</h3>
417
<p>Twisted's documentation frequently references API documentation. In Lore,
418
the name of an API such as
419
<code class="API">twisted.internet.defer.Deferred</code> is marked up as</p>
422
<code class="API" base="twisted.internet.defer">Deferred</code>
425
<p>This will unambiguously link to
426
<code class="API">twisted.internet.defer.Deferred</code>, even though it is
428
<q><code class="API" base="twisted.internet.defer">Deferred</code></q>. Lore
429
produces API links that work with
430
<a href="http://epydoc.sourceforge.net">epydoc</a>,
431
but could easily be adapted for another API documentation generator; in fact,
432
Lore originally worked with happydoc.
433
In addition, in the HTML output, Lore will add a <code>title</code>
434
attribute to the API reference, containing the full name of the link.</p>
436
<h3>Cross references</h3>
438
<p>A collection of documents will typically refer to each other, for instance to
439
avoid re-explaining some central concept. In HTML, cross-referencing
440
is implemented as linking:</p>
443
See <a href="defer.html">Deferring Execution</a>.
446
<p>As a collection of HTML documents, this works with no changes. Other output
447
formats do linking in other ways. When Lore is used to convert a collection of
448
source HTML files into a single LaTeX book, each file is its own section, and
449
the links are automatically converted into cross-references. Thus the example
450
above might be rendered as <q>See Deferring Execution (page 163).</q></p>
452
<p>Lore also recognizes <em>fragment identifiers</em> in links, so that a link
453
to <code>glossary.html#psu</code> will be cross-referenced to that part of the
454
glossary named <q>psu</q>, not just the whole glossary. This ensures that the
455
page the reader is referred to is the correct one.</p>
459
<p>Man pages are a fact of life on UNIX, and every self-respecting command
460
line program is expected to come with one. The man format, implemented as
461
troff macros, is somewhat arcane. Since, when Lore was written, we already
462
had written man pages, the decision was to convert them to HTML rather than
463
try to rewrite them in HTML and design a man output format.</p>
465
<p>A limited parser for man pages is available in the
466
<code class="API">twisted.lore.man2lore</code> module. It is not yet
467
exposed via any public command line program.</p>
469
<p>Earlier attempts, using <code class="shell">groff -Thtml</code> to
470
generate HTML and then post-process it into Lore-compatible HTML
471
were crufty and unmaintainable. It seems the man format shares some
472
of LaTeX's problem: being written as a macro package over a powerful
473
processor, it is too flexible for its own good. Fortunately, the subset
474
normally used in man pages is quite small, so heuristically parsing man pages is
475
much easier than the same task with LaTeX.</p>
481
<p>HTML, when invented by Tim Berners-Lee, was meant to be a simple language
482
for writing and sharing documents. With the explosion of the web, HTML has
483
grown to a confusing jumble of logical and presentation features, with more
484
layers, such as CSS, dumped on top of it. As a result, a modern browser is
485
a complicated beast. That given, it is perhaps understandable that today's
486
browsers do a sub-standard job at printing. Thus, while being extremely
487
well suited to the world wide web, HTML is significantly lacking, at least
488
in today's application market, when it comes to paper output. It might
489
be possible to write an application to properly convert HTML with CSS to
490
PostScript or PDF -- however, it would probably be much more complicated
491
than Lore. Moreover, the portability of such an application would
492
be worse of the portability of Lore itself, which currently only depends
493
on Python 2.1 or higher and the Twisted framework.</p>
495
<p>Limiting HTML to a small subset of features enables Lore to be small
496
and readable while remaining useful. By including the <code>class</code>
497
attribute among those features, Lore is also extensible.</p>
501
<p>When it comes to paper output, LaTeX cannot be out done except by a skilled
502
typesetter designing and implementing. However, the architecture of LaTeX
504
significant problems when trying to view LaTeX online. LaTeX is written
505
as a macro layer above TeX rather than a preprocessor. Thus, all of TeX's
506
power is available, and sometimes used, in LaTeX. TeX is non-trivial to
507
parse and format by anyone short of Donald Knuth -- it contains such commands
508
as to change the tokenizer by modifying which characters are considered
509
word characters or even which character is the command character.
510
In fact, the authors are not aware of any application which handles the
511
full power of TeX without being based on the original TeX code.</p>
513
<p>All this makes LaTeX extremely difficult to parse, and even partial attempts
514
to parse LaTeX are big and cumbersome -- for example,
515
<code class="shell">latex2html</code>. It is thus difficult to convert
516
LaTeX to something appropriate to online viewing.</p>
520
<p>LyX's internal source format is not well documented, and the only supported
521
way to write it is using the LyX GUI. Thus it is inherently limiting to
522
documentation authors. In addition, it is not trivial to write LyX preprocessors
523
to save documentation authors tedious work.</p>
527
<p>Docbook is a big standard, with non-trivial to install tool-set. Writing
528
Docbook is different than most other document generation formats, so it
529
takes significant training to write. In addition, using Docbook for
530
a specific project usually requires writing custom DSSSL stylesheets
531
in a scheme-like language, and additional XML DTD snippets. Writing
532
these was quite possibly comparable to writing Lore, and Lore has the advantage
533
of being written in Python.</p>
537
<p>Texinfo imposes a significant effort on authors. Many things need to
538
be written twice, and the error messages leave a lot to be desired.
539
After starting to work on the Lore texinfo output format the authors
540
are grateful they have never had to write Texinfo by hand.</p>
544
<h3>Visitor Pattern</h3>
546
<p>When generating LaTeX, Lore does it via a visitor pattern while visiting
547
the nodes. A node which does not have a specific visitor is visited by
548
first writing the <code>start_</code> attribute, then visiting its
549
children and then writing the <code>end_</code> attribute. If the attributes
550
do not exist, they are treated as though they were empty strings.</p>
552
<p>That code allows most of the HTML elements to LaTeX converters to have no
553
code -- only a pair of strings -- while the elements converters which need
554
more sophisticated programming can do it via defining a method, which can
555
still call the default processor if it needs this functionality.</p>
557
<p>This pattern is also friendly to subclassing: all a subclass needs to
558
do in order to change how an element is handled is to define either a pair
559
of class attributes or a method.</p>
561
<h3>Liberal Use of Reflection</h3>
563
<p>In the above example of the visitor pattern, registration of the methods
564
and attributes is avoided thanks to using the crudest form of reflection
565
in Python -- the <code>getattr()</code> function.</p>
567
<p>In the Lint support tool, more sophisticated reflection is needed when
568
it needs to find all methods whose name begins with <code>check_</code>.
569
This is done via the Twisted reflection code, built on top of the native
570
Python facilities, in the module
571
<code class="API">twisted.python.reflect</code>.</p>
573
<h3>Recursively Searching For Elements</h3>
575
<p>In the HTML output code, the most common operation is that of getting
576
a list of elements which satisfy some property. This is done by one
577
primary work-horse function:
578
<code class="API">twisted.web.domhelpers.findNodes</code>. This function
579
accepts a DOM tree and a function, and returns a list of all elements
580
for which this function returns true. Using this, and the fact that Python makes
581
it easy to combine functions into boolean combinations, makes analysis
582
and modification and of the DOM tree a breeze.</p>
584
<h2>Lessons Learned</h2>
586
<h3>Problems With Some Output Formats</h3>
588
<p>Probably the trickiest thing about non-HTML output formats is escaping.
589
The problem comes from two annoying problems which are not really hard
590
to solve, but do represent annoyances in the code:</p>
593
<li>Different characters are escaped differently (for example, <code>\</code>
594
is escaped, in TeX, as <code>$\backslash$</code> while most other
595
characters are escaped as <code>\<char></code>.</li>
596
<li>Escaping depends on context -- special characters should not be escaped
597
at all inside <code>pre</code>, <code></></code> should not be
598
escaped inside <code>code</code> and should be escaped as
599
<code>$<$/$>$</code> outside it.</li>
602
<p>In Docbook, the sections are nested, so there is only need for
603
a <code>title</code> element. However, in HTML only the headers care
604
at which level they are. This requires the Docbook converter to keep
605
the last header level and when it reaches a new header, to close and open
606
enough sections so the header will get to the correct level. While Docbook's
607
way may be more <q>correct</q>, it is unfortunate it chose to diverge from
608
all other systems here.</p>
610
<p>Texinfo requires all the sections in a document will have unique names.
611
This makes it very inconvenient as both an input and an output format.</p>
613
<p>Also, differing significance of whitespace in different formats requires that
614
all whitespace emitted by lore must be normalized for the particular output
615
format being used. Blank lines which have no impact on HTML will trigger
616
paragraph breaks in LaTeX.</p>
618
<h3>Event-based XML Parsing Considered Harmful</h3>
620
<p>The first version of the LaTeX output generator was using an event-based
621
XML parsing engine. It quickly turned out one needs to keep a lot of
622
information in stacks and manage many instance variables. For example,
623
though XML gets the name of the closing element (even that is arguably
624
too much information), it does not get the attributes. In <code>span</code>
625
elements, for example, the interesting information is the <code>class</code>
626
attribute. Since a-priory, <code>span</code>s might be nested, the class
627
needs to keep a stack of attribute collections.</p>
629
<p>Quite soon, stacks were needed for proper handling of <code>div</code>
630
tags and for determining proper quoting formats. Moreover, getting the
631
code to function correctly in the face of edge cases, such as cross-references
632
inside <code>pre</code> tags, proved to be quite a challenge.</p>
634
<p>The code was shortened, simplified and became more maintainable when
635
it was moved to <code class="API" base="twisted.web">microdom</code>.</p>
637
<p>We feel that unless there is
638
an inherent reason to do XML event-based parsing, then it is much easier
639
to read the whole thing into a DOM and then process it. The code is both
640
shorter and clearer, and features are much easier to add.</p>
642
<h3>Allow Easy Modification</h3>
644
<p>Lore, out of the box, does not attempt to be all things to all people.
645
Particularly in the LaTeX output format, there is a lot of room for
646
interpretation and personal preferences. Lore chose one specific way, without
647
trying to add half a dozen options to tweak it. However, thanks to the
648
way it is coded, it is easy to add or modify features to suit individual
649
preferences. Many customizations only involve adding or overriding simple data
650
attributes to a subclass; more advanced changes require adding or overriding
653
<p>Likewise, the HTML output is built by running several tree-modification
654
functions which are independent. Completely different HTML output could
655
be build by adding more functions, or not running some of those which
658
<p>We already know of multiple users that have extended Lore for custom LaTeX
659
generation. In each case it was a simple matter of subclassing Lore's LaTeX
662
<h3>Reinventing Wheels Can Be Useful</h3>
664
<p>Documentation generation systems were already a solved problem before Lore
665
was written. However, we know of no system with Lore's unique combination of
666
features -- in particular, portability, having a directly readable source format
667
which is also directly writable in text editors.
668
The common wisdom that a documentation generation
669
system is a hard sell because it requires people to learn a new language was
670
refuted by using an existing language.</p>
672
<p>Wheel reinvention also occurred in a nearby area -- Twisted's XML support,
673
for which Lore is one of the biggest users. Again, the common wisdom was that
674
this was a solved problem, with many existing DOM and SAX implementations.
675
However, implementation of some features, no implementation of other features
676
and API instability have lead the Twisted team to write its own, highly
677
pythonic, DOM-like implementation. In
678
<code class="API" base="twisted.web">microdom</code>, the aim is to be
679
as thin a wrapper over the basic Python wrappers as possible. This feature
680
has been used to the full in Lore, where many of the tree manipulations
681
would have been much more cumbersome had a standard <q>opaque</q> DOM
682
implementation been used. In addition, using
683
<code class="API" base="twisted.web">microdom</code> frees Lore from the
684
dependence on both Python version and whether PyXML is installed.</p>
686
<p>For example, <code class="API" base="twisted.web">microdom</code>
687
exposes the list of child nodes as a plain Python lists. This means that
688
not only all the list operations can be done of it, which could possibly
689
be simulated by a list-like object, but that it is possible to
690
<em>replace</em> it by our own list. As another example,
691
<code class="API" base="twisted.web">microdom</code> allows us to freely
692
copy nodes from one DOM tree into another.</p>
694
<p>Python, as a language well suited to rapid application development,
695
acts as a way to make wheel reinvention far from the horrible mistake
696
which is portrayed in the common software engineering folklore. Indeed, Python
697
makes it easy enough to reinvent wheels that only the best, and easy to
698
use, wheels, get reused at all.</p>
700
<h2>Availability</h2>
702
<p>Lore can be found in Twisted 1.0.1 and higher, in the
703
<code class="API">twisted.lore</code> package. When you install the package,
704
the relevant script, <code class="shell">lore</code>,
705
should be installed in a sane directory,
706
as determined by distutils.</p>
708
<p>For usage examples, see <code class="shell">admin/release-twisted</code>
709
in the Twisted source distribution. It runs the various Lore scripts
710
as part of the package build.</p>
712
<h2>Future Plans</h2>
714
<h3>More Output Formats</h3>
716
<p>It would be nice to have the Docbook output fully working. It would also
717
be nice to have Texinfo in full working order so that GNU info aficionados could
718
read the documents with the info browser. As suggested above, it might
719
also be useful to have a way to directly generate PDF output via
720
<code>pdflib</code> in order to skip LaTeX.</p>
722
<p>In addition, another potential output format is to have high-quality
723
text output. This is non-trivial, but possibly useful: browsers'
724
<q>Save as text</q> feature is usually implemented as an afterthought,
725
and hardly uses the flexibility available in the text format to its
726
full power. The authors are unaware, for example, of an HTML to text
727
converter which uses the underlining with <q>=</q> sign or <q>-</q>
728
to indicate a header, or which uses the <code>/slant/</code> or
729
<code>*asterisk*</code> conventions to indicate emphasis.</p>
731
<p>Another output format we are considering is a split-page HTML with
732
interlinks, so that long documents can be converted into something
733
which is web-friendly. One nice use for that would be in web-based
736
<h3>Image Conversion</h3>
738
<p>Currently all images are converted to EPS format. It would be nice to have
739
the LaTeX converter try to see if there is already an EPS version, via some
740
naming convention, and use that. This would allow better scaling of things like
741
Dia diagrams. The versions in bitmap-based formats (such as PNG)
742
are impossible to scale, because the text would become unreadable.</p>
746
<p>Currently, the only interface to Lore is through the command-line, and
747
even that is somewhat spotty: for example, the man page parser is not directly
748
available via the command line. We hope to remedy that, having at least a full
749
suite of command-line tools and possibly graphical wrappers, particularly
752
<h2>Twisted Integration</h2>
754
<p>When starting with a historical note, it is only fitting to end
755
with a historical note. Since the writing of Lore, Twisted documentation
756
is successfully generated by it and distributed in the tarball. It contains
757
generated HTML from the HOWTO documents, specifications and man pages.
758
It also contains all these documents inside a LaTeX-generated PostScript
759
file and PDF file in an easy to print format, suitable for reading on those
760
long plane flights or train rides.</p>
762
<p>Lore is also used to generate pages with consistent headers and footers for
763
the twistedmatrix.com web site -- not just the Twisted documentation.
764
This is shows the inherent flexibility in Lore's model of being easily
765
configurable via an HTML template,
766
a feature which none of the major
767
document generation systems support for their HTML output.</p>
769
<h2>Further Resources</h2>
773
href="http://twistedmatrix.com/documents/TwistedDocs/TwistedDocs-1.0.3/man/lore-man.xhtml"><code
774
class="shell">lore(1)</code> man page</a></li>
776
href="http://twistedmatrix.com/documents/TwistedDocs/TwistedDocs-1.0.3/howto/doc-standard">Lore guidelines</a></li>
778
href="http://twistedmatrix.com/documents/TwistedDocs/TwistedDocs-1.0.3/howto/lore">Lore HOWTO</a></li>
780
href="http://twistedmatrix.com/documents/TwistedDocs/TwistedDocs-1.0.1/examples/example.html">Skeleton Lore document</a></li>
782
href="http://twistedmatrix.com/users/jh.twistd/viewcvs/cgi/viewcvs.cgi/~checkout~/doc/howto/stylesheet.css?rev=1.16&content-type=text/css&cvsroot=Twisted">stylesheet</a> and <a
783
href="http://twistedmatrix.com/users/jh.twistd/viewcvs/cgi/viewcvs.cgi/~checkout~/doc/howto/template.tpl?rev=1.6&content-type=text/plain&cvsroot=Twisted">template</a> used by the Twisted documentation</li>
784
<li><a href="http://www.w3.org/TR/xhtml1/">The XHTML specification</a></li>
785
<li><a href="http://www.latex-project.org">LaTeX project home page</a></li>
786
<li><a href="http://www.lyx.org">LyX</a></li>
787
<li><a href="http://www.docbook.org">Docbook</a></li>
788
<li><a href="http://www.python10.com/p10-papers/09/index.htm">Zadka, Moshe and Lefkowitz, Glyph, The Twisted Network Framework, The Tenth International Python Conference Proceedings</a></li>
added
removed
vendor/Twisted-10.0.0/doc/historic/2003/pycon/lore/lore.html
