1.1.2
by Colin Watson
Import upstream version 5.7 |
1 |
<!-- $Id: EXTENDING.html 198 2002-09-04 01:17:32Z darren $ -->
|
1
by Colin Watson
Import upstream version 5.5.4 |
2 |
<html> |
3 |
<head> |
|
4 |
<title>Exuberant Ctags: Adding support for a new language</title> |
|
5 |
</head> |
|
6 |
<body> |
|
7 |
||
8 |
<h1>How to Add Support for a New Language to Exuberant Ctags</h1> |
|
9 |
||
10 |
<p> |
|
11 |
<b>Exuberant Ctags</b> has been designed to make it very easy to add your own |
|
12 |
custom language parser. As an exercise, let us assume that I want to add |
|
13 |
support for my new language, <em>Swine</em>, the successor to Perl (i.e. Perl |
|
14 |
before Swine <wince>). This language consists of simple definitions of |
|
15 |
labels in the form "<code>def my_label</code>". Let us now examine the various |
|
16 |
ways to do this. |
|
17 |
</p> |
|
18 |
||
19 |
<h2>Operational background</h2> |
|
20 |
||
21 |
<p> |
|
22 |
As ctags considers each file name, it tries to determine the language of the |
|
23 |
file by applying the following three tests in order: if the file extension has |
|
24 |
been mapped to a language, if the file name matches a shell pattern mapped to |
|
25 |
a language, and finally if the file is executable and its first line specifies |
|
26 |
an interpreter using the Unix-style "#!" specification (if supported on the |
|
27 |
platform). If a language was identified, the file is opened and then the |
|
28 |
appropriate language parser is called to operate on the currently open file. |
|
29 |
The parser parses through the file and whenever it finds some interesting |
|
30 |
token, calls a function to define a tag entry. |
|
31 |
</p> |
|
32 |
||
33 |
<h2>Creating a user-defined language</h2> |
|
34 |
||
35 |
<p> |
|
36 |
The quickest and easiest way to do this is by defining a new language using |
|
37 |
the program options. In order to have Swine support available every time I |
|
38 |
start ctags, I will place the following lines into the file |
|
39 |
<code>$HOME/.ctags</code>, which is read in every time ctags starts: |
|
40 |
||
41 |
<code> |
|
42 |
<pre> |
|
43 |
--langdef=swine |
|
44 |
--langmap=swine:.swn |
|
45 |
--regex-swine=/^def[ \t]*([a-zA-Z0-9_]+)/\1/d,definition/ |
|
46 |
</pre> |
|
47 |
</code> |
|
48 |
The first line defines the new language, the second maps a file extension to |
|
49 |
it, and the third defines a regular expression to identify a language |
|
50 |
definition and generate a tag file entry for it. |
|
51 |
</p> |
|
52 |
||
53 |
<h2>Integrating a new language parser</h2> |
|
54 |
||
55 |
<p> |
|
56 |
Now suppose that I want to truly integrate compiled-in support for Swine into |
|
57 |
ctags. First, I create a new module, <code>swine.c</code>, and add one |
|
58 |
externally visible function to it, <code>extern parserDefinition |
|
59 |
*SwineParser(void)</code>, and add its name to the table in |
|
60 |
<code>parsers.h</code>. The job of this parser definition function is to |
|
61 |
create an instance of the <code>parserDefinition</code> structure (using |
|
62 |
<code>parserNew()</code>) and populate it with information defining how files |
|
63 |
of this language are recognized, what kinds of tags it can locate, and the |
|
64 |
function used to invoke the parser on the currently open file. |
|
65 |
</p> |
|
66 |
||
67 |
<p> |
|
68 |
The structure <code>parserDefinition</code> allows assignment of the following |
|
69 |
fields: |
|
70 |
||
71 |
<code> |
|
72 |
<pre> |
|
73 |
const char *name; /* name of language */ |
|
74 |
kindOption *kinds; /* tag kinds handled by parser */ |
|
75 |
unsigned int kindCount; /* size of `kinds' list */ |
|
76 |
const char *const *extensions; /* list of default extensions */ |
|
77 |
const char *const *patterns; /* list of default file name patterns */ |
|
78 |
parserInitialize initialize; /* initialization routine, if needed */ |
|
79 |
simpleParser parser; /* simple parser (common case) */ |
|
80 |
rescanParser parser2; /* rescanning parser (unusual case) */ |
|
81 |
boolean regex; /* is this a regex parser? */ |
|
82 |
</pre> |
|
83 |
</code> |
|
84 |
</p> |
|
85 |
||
86 |
<p> |
|
87 |
The <code>name</code> field must be set to a non-empty string. Also, unless |
|
88 |
<code>regex</code> is set true (see below), either <code>parser</code> or |
|
89 |
<code>parser2</code> must set to point to a parsing routine which will |
|
90 |
generate the tag entries. All other fields are optional. |
|
91 |
||
92 |
<p> |
|
93 |
Now all that is left is to implement the parser. In order to do its job, the |
|
94 |
parser should read the file stream using using one of the two I/O interfaces: |
|
95 |
either the character-oriented <code>fileGetc()</code>, or the line-oriented |
|
96 |
<code>fileReadLine()</code>. When using <code>fileGetc()</code>, the parser |
|
97 |
can put back a character using <code>fileUngetc()</code>. How our Swine parser |
|
98 |
actually parses the contents of the file is entirely up to the writer of the |
|
99 |
parser--it can be as crude or elegant as desired. You will note a variety of |
|
100 |
examples from the most complex (c.c) to the simplest (make.c). |
|
101 |
</p> |
|
102 |
||
103 |
<p> |
|
104 |
When the Swine parser identifies an interesting token for which it wants to |
|
105 |
add a tag to the tag file, it should create a <code>tagEntryInfo</code> |
|
106 |
structure and initialize it by calling <code>initTagEntry()</code>, which |
|
107 |
initializes defaults and fills information about the current line number and |
|
108 |
the file position of the beginning of the line. After filling in information |
|
109 |
defining the current entry (and possibly overriding the file position or other |
|
110 |
defaults), the parser passes this structure to <code>makeTagEntry()</code>. |
|
111 |
</p> |
|
112 |
||
113 |
<p> |
|
114 |
Instead of writing a character-oriented parser, it may be possible to specify |
|
115 |
regular expressions which define the tags. In this case, instead of defining a |
|
116 |
parsing function, <code>SwineParser()</code>, sets <code>regex</code> to true, |
|
117 |
and points <code>initialize</code> to a function which calls |
|
118 |
<code>addTagRegex()</code> to install the regular expressions which define its |
|
119 |
tags. The regular expressions thus installed are compared against each line |
|
120 |
of the input file and generate a specified tag when matched. It is usually |
|
121 |
much easier to write a regex-based parser, although they can be slower (one |
|
122 |
parser example was 4 times slower). Whether the speed difference matters to |
|
123 |
you depends upon how much code you have to parse. It is probably a good |
|
124 |
strategy to implement a regex-based parser first, and if it is too slow for |
|
125 |
you, then invest the time and effort to write a character-based parser. |
|
126 |
</p> |
|
127 |
||
128 |
<p> |
|
129 |
A regex-based parser is inherently line-oriented (i.e. the entire tag must be |
|
130 |
recognizable from looking at a single line) and context-insensitive (i.e the |
|
131 |
generation of the tag is entirely based upon when the regular expression |
|
132 |
matches a single line). However, a regex-based callback mechanism is also |
|
133 |
available, installed via the function <code>addCallbackRegex()</code>. This |
|
134 |
allows a specified function to be invoked whenever a specific regular |
|
135 |
expression is matched. This allows a character-oriented parser to operate |
|
136 |
based upon context of what happened on a previous line (e.g. the start or end |
|
137 |
of a multi-line comment). Note that regex callbacks are called just before the |
|
138 |
first character of that line can is read via either <code>fileGetc()</code> or |
|
139 |
using <code>fileGetc()</code>. The effect of this is that before either of |
|
140 |
these routines return, a callback routine may be invoked because the line |
|
141 |
matched a regex callback. A callback function to be installed is defined by |
|
142 |
these types: |
|
143 |
||
144 |
<code> |
|
145 |
<pre> |
|
146 |
typedef void (*regexCallback) (const char *line, const regexMatch *matches, unsigned int count); |
|
147 |
||
148 |
typedef struct { |
|
149 |
size_t start; /* character index in line where match starts */ |
|
150 |
size_t length; /* length of match */ |
|
151 |
} regexMatch; |
|
152 |
</pre> |
|
153 |
</code> |
|
154 |
</p> |
|
155 |
||
156 |
<p> |
|
157 |
The callback function is passed the line matching the regular expression and |
|
158 |
an array of <code>count</code> structures defining the subexpression matches |
|
159 |
of the regular expression, starting from \0 (the entire line). |
|
160 |
</p> |
|
161 |
||
162 |
<p> |
|
163 |
Lastly, be sure to add your the name of the file containing your parser (e.g. |
|
164 |
swine.c) to the macro <code>SOURCES</code> in the file <code>source.mak</code> |
|
165 |
and an entry for the object file to the macro <code>OBJECTS</code> in the same |
|
166 |
file, so that your new module will be compiled into the program. |
|
167 |
</p> |
|
168 |
||
169 |
<p> |
|
170 |
This is all there is to it. All other details are specific to the parser and |
|
171 |
how it wants to do its job. There are some support functions which can take |
|
172 |
care of some commonly needed parsing tasks, such as keyword table lookups (see |
|
173 |
keyword.c), which you can make use of if desired (examples of its use can be |
|
174 |
found in c.c, eiffel.c, and fortran.c). Almost everything is already taken care |
|
175 |
of automatically for you by the infrastructure. Writing the actual parsing |
|
176 |
algorithm is the hardest part, but is not constrained by any need to conform |
|
177 |
to anything in ctags other than that mentioned above. |
|
178 |
</p> |
|
179 |
||
180 |
<p> |
|
181 |
There are several different approaches used in the parsers inside <b>Exuberant |
|
182 |
Ctags</b> and you can browse through these as examples of how to go about |
|
183 |
creating your own. |
|
184 |
</p> |
|
185 |
||
186 |
<h2>Examples</h2> |
|
187 |
||
188 |
<p> |
|
189 |
Below you will find several example parsers demonstrating most of the |
|
190 |
facilities available. These include three alternative implementations |
|
191 |
of a Swine parser, which generate tags for lines beginning with |
|
192 |
"<CODE>def</CODE>" followed by some name. |
|
193 |
</p> |
|
194 |
||
195 |
<code> |
|
196 |
<pre> |
|
197 |
/*************************************************************************** |
|
198 |
* swine.c |
|
199 |
* Character-based parser for Swine definitions |
|
200 |
**************************************************************************/ |
|
201 |
/* INCLUDE FILES */ |
|
202 |
#include "general.h" /* always include first */ |
|
203 |
||
204 |
#include <string.h> /* to declare strxxx() functions */ |
|
205 |
#include <ctype.h> /* to define isxxx() macros */ |
|
206 |
||
207 |
#include "parse.h" /* always include */ |
|
208 |
#include "read.h" /* to define file fileReadLine() */ |
|
209 |
||
210 |
/* DATA DEFINITIONS */ |
|
211 |
typedef enum eSwineKinds { |
|
212 |
K_DEFINE |
|
213 |
} swineKind; |
|
214 |
||
215 |
static kindOption SwineKinds [] = { |
|
216 |
{ TRUE, 'd', "definition", "pig definition" } |
|
217 |
}; |
|
218 |
||
219 |
/* FUNCTION DEFINITIONS */ |
|
220 |
||
221 |
static void findSwineTags (void) |
|
222 |
{ |
|
223 |
vString *name = vStringNew (); |
|
224 |
const unsigned char *line; |
|
225 |
||
226 |
while ((line = fileReadLine ()) != NULL) |
|
227 |
{ |
|
228 |
/* Look for a line beginning with "def" followed by name */ |
|
229 |
if (strncmp ((const char*) line, "def", (size_t) 3) == 0 &&
|
|
230 |
isspace ((int) line [3])) |
|
231 |
{ |
|
232 |
const unsigned char *cp = line + 4; |
|
233 |
while (isspace ((int) *cp)) |
|
234 |
++cp; |
|
235 |
while (isalnum ((int) *cp) || *cp == '_') |
|
236 |
{ |
|
237 |
vStringPut (name, (int) *cp); |
|
238 |
++cp; |
|
239 |
} |
|
240 |
vStringTerminate (name); |
|
241 |
makeSimpleTag (name, SwineKinds, K_DEFINE); |
|
242 |
vStringClear (name); |
|
243 |
} |
|
244 |
} |
|
245 |
vStringDelete (name); |
|
246 |
} |
|
247 |
||
248 |
/* Create parser definition stucture */ |
|
249 |
extern parserDefinition* SwineParser (void) |
|
250 |
{ |
|
251 |
static const char *const extensions [] = { "swn", NULL }; |
|
252 |
parserDefinition* def = parserNew ("Swine"); |
|
253 |
def->kinds = SwineKinds;
|
|
254 |
def->kindCount = KIND_COUNT (SwineKinds);
|
|
255 |
def->extensions = extensions;
|
|
256 |
def->parser = findSwineTags;
|
|
257 |
return def; |
|
258 |
} |
|
259 |
</pre> |
|
260 |
</code> |
|
261 |
||
262 |
<p> |
|
263 |
<pre> |
|
264 |
<code> |
|
265 |
/*************************************************************************** |
|
266 |
* swine.c |
|
267 |
* Regex-based parser for Swine |
|
268 |
**************************************************************************/ |
|
269 |
/* INCLUDE FILES */ |
|
270 |
#include "general.h" /* always include first */ |
|
271 |
#include "parse.h" /* always include */ |
|
272 |
||
273 |
/* FUNCTION DEFINITIONS */ |
|
274 |
||
275 |
static void installSwineRegex (const langType language) |
|
276 |
{ |
|
277 |
addTagRegex (language, "^def[ \t]*([a-zA-Z0-9_]+)", "\\1", "d,definition", NULL); |
|
278 |
} |
|
279 |
||
280 |
/* Create parser definition stucture */ |
|
281 |
extern parserDefinition* SwineParser (void) |
|
282 |
{ |
|
283 |
static const char *const extensions [] = { "swn", NULL }; |
|
284 |
parserDefinition* def = parserNew ("Swine"); |
|
285 |
parserDefinition* const def = parserNew ("Makefile"); |
|
286 |
def->patterns = patterns;
|
|
287 |
def->extensions = extensions;
|
|
288 |
def->initialize = installMakefileRegex;
|
|
289 |
def->regex = TRUE;
|
|
290 |
return def; |
|
291 |
} |
|
292 |
</code> |
|
293 |
</pre> |
|
294 |
||
295 |
<p> |
|
296 |
<pre> |
|
297 |
/*************************************************************************** |
|
298 |
* swine.c |
|
299 |
* Regex callback-based parser for Swine definitions |
|
300 |
**************************************************************************/ |
|
301 |
/* INCLUDE FILES */ |
|
302 |
#include "general.h" /* always include first */ |
|
303 |
||
304 |
#include "parse.h" /* always include */ |
|
305 |
#include "read.h" /* to define file fileReadLine() */ |
|
306 |
||
307 |
/* DATA DEFINITIONS */ |
|
308 |
typedef enum eSwineKinds { |
|
309 |
K_DEFINE |
|
310 |
} swineKind; |
|
311 |
||
312 |
static kindOption SwineKinds [] = { |
|
313 |
{ TRUE, 'd', "definition", "pig definition" } |
|
314 |
}; |
|
315 |
||
316 |
/* FUNCTION DEFINITIONS */ |
|
317 |
||
318 |
static void definition (const char *const line, const regexMatch *const matches, |
|
319 |
const unsigned int count) |
|
320 |
{ |
|
321 |
if (count > 1) /* should always be true per regex */
|
|
322 |
{ |
|
323 |
vString *const name = vStringNew (); |
|
324 |
vStringNCopyS (name, line + matches [1].start, matches [1].length); |
|
325 |
makeSimpleTag (name, SwineKinds, K_DEFINE); |
|
326 |
} |
|
327 |
} |
|
328 |
||
329 |
static void findSwineTags (void) |
|
330 |
{ |
|
331 |
while (fileReadLine () != NULL) |
|
332 |
; /* don't need to do anything here since callback is sufficient */ |
|
333 |
} |
|
334 |
||
335 |
static void installSwine (const langType language) |
|
336 |
{ |
|
337 |
addCallbackRegex (language, "^def[ \t]+([a-zA-Z0-9_]+)", NULL, definition); |
|
338 |
} |
|
339 |
||
340 |
/* Create parser definition stucture */ |
|
341 |
extern parserDefinition* SwineParser (void) |
|
342 |
{ |
|
343 |
static const char *const extensions [] = { "swn", NULL }; |
|
344 |
parserDefinition* def = parserNew ("Swine"); |
|
345 |
def->kinds = SwineKinds;
|
|
346 |
def->kindCount = KIND_COUNT (SwineKinds);
|
|
347 |
def->extensions = extensions;
|
|
348 |
def->parser = findSwineTags;
|
|
349 |
def->initialize = installSwine;
|
|
350 |
return def; |
|
351 |
} |
|
352 |
</pre> |
|
353 |
||
354 |
<p> |
|
355 |
<pre> |
|
356 |
/*************************************************************************** |
|
357 |
* make.c |
|
358 |
* Regex-based parser for makefile macros |
|
359 |
**************************************************************************/ |
|
360 |
/* INCLUDE FILES */ |
|
361 |
#include "general.h" /* always include first */ |
|
362 |
#include "parse.h" /* always include */ |
|
363 |
||
364 |
/* FUNCTION DEFINITIONS */ |
|
365 |
||
366 |
static void installMakefileRegex (const langType language) |
|
367 |
{ |
|
368 |
addTagRegex (language, "(^|[ \t])([A-Z0-9_]+)[ \t]*:?=", "\\2", "m,macro", "i"); |
|
369 |
} |
|
370 |
||
371 |
/* Create parser definition stucture */ |
|
372 |
extern parserDefinition* MakefileParser (void) |
|
373 |
{ |
|
374 |
static const char *const patterns [] = { "[Mm]akefile", NULL }; |
|
375 |
static const char *const extensions [] = { "mak", NULL }; |
|
376 |
parserDefinition* const def = parserNew ("Makefile"); |
|
377 |
def->patterns = patterns;
|
|
378 |
def->extensions = extensions;
|
|
379 |
def->initialize = installMakefileRegex;
|
|
380 |
def->regex = TRUE;
|
|
381 |
return def; |
|
382 |
} |
|
383 |
</pre> |
|
384 |
||
385 |
</body> |
|
386 |
</html> |