~ubuntu-branches/ubuntu/trusty/swish-e/trusty : revision 5

1

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"

2

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

3

4

<!--

5

***** GENERATED FILE *** DO NOT EDIT DIRECTLY - any changes will be LOST ******

6

7

swish-e.org mockup based on http://www.oswd.org/design/1773/prosimii/index2.html

8

-->

9

10

11

12

<head>

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

<title>Swish-e :: SWISH-CONFIG - Configuration File Directives</title>

35

</head>

36

37

38

<body>

39

40

41

42

43

44

<div id="top"><a href="#main-copy" class="doNotDisplay doNotPrint">Skip to main content.</a></div>

45

46

47

48

49

50

Related Sites:

51

<a href="http://swishewiki.org/" title="swishe wiki">swish-e wiki</a> |

52

<a href="http://www.xmlsoft.org/" title="libxml2 home page">libxml2</a> |

53

<a href="http://www.zlib.net/" title="zlib home page">zlib</a> |

54

<a href="http://www.foolabs.com/xpdf/" title="xpdf home page">xpdf</a> |

55

<a href="http://cvs.sourceforge.net/viewcvs.py/swishe/" title="View CVS at SourceForge">CVS @ SourceForge</a>

56

</div>

57

58

59

<h1 class="headerTitle" lang="la">Swish-e</h1>

60

<div class="headerSubTitle">Simple Web Indexing System for Humans - Enhanced</div>

61

62

63

64

65

Tools:

66

67

68

69

<a href="http://swish-e.org/download/index.html">download latest version</a>

70

71

72

</div>

73

</div>

74

</div>

75

76

77

78

79

80

<form method="get"

81

action="http://swish-e.org/search/index.html"

82

enctype="application/x-www-form-urlencoded"

83

class="srchform">

84

85

86

<tr>

87

88

<a href="http://swish-e.org/index.html">home</a> |

89

<a href="http://swish-e.org/support.html">support</a> |

90

<a href="http://swish-e.org/download/index.html">download</a>

91

</td>

92

93

94

95

96

<label for="searchfield">Search for</label>

97

98

99

100

</td>

101

</tr>

102

</table>

103

104

</form>

105

106

</div>

107

108

109

110

111

112

113

114

115

116

<a class="menu"

117

href="./index.html"

118

>Doc Overview</a>

119

120

121

122

123

<a class="submenu"

124

href="./readme.html"

125

title="First time users">README</a>

126

127

</li><li class="">

128

129

<a class="submenu"

130

href="./install.html"

131

title="Installation and usage overview">Install</a>

132

133

</li><li class="">

134

135

<a class="submenu"

136

href="./changes.html"

137

title="Important changes from previous versions">Changes</a>

138

139

</li><li class="">

140

141

<a class="thisfile"

142

href="./swish-config.html"

143

title="Directives that go in your Swish-e configuration file">Configuration »</a>

144

145

</li><li class="">

146

147

<a class="submenu"

148

href="./swish-run.html"

149

title="Command line options for Swish-e binary">Running</a>

150

151

</li><li class="">

152

153

<a class="submenu"

154

href="./swish-search.html"

155

title="Swish-e's search language">Searching</a>

156

157

</li><li class="">

158

159

<a class="submenu"

160

href="./swish-faq.html"

161

>FAQ</a>

162

163

</li><li class="">

164

165

<a class="submenu"

166

href="./swish-bugs.html"

167

>Known issues</a>

168

169

</li><li class="">

170

171

<a class="submenu"

172

href="./swish-3.0.html"

173

>The Future</a>

174

175

</li><li class="">

176

177

<a class="submenu"

178

href="./swish-library.html"

179

title="Swish-e C API">C API</a>

180

181

</li><li class="">

182

183

<a class="submenu"

184

href="./api.html"

185

title="Perl interface to the Swish-e library">Perl API</a>

186

187

</li><li class="">

188

189

<a class="submenu"

190

href="./swish.cgi.html"

191

title="Example CGI/mod_perl script">Swish.cgi</a>

192

193

</li><li class="">

194

195

<a class="submenu"

196

href="./search.cgi.html"

197

title="Example Perl script using SWISH::API">Search.cgi</a>

198

199

</li><li class="">

200

201

<a class="submenu"

202

href="./spider.html"

203

title="The Swish-e HTTP spider">Spider.pl</a>

204

205

</li><li class="">

206

207

<a class="submenu"

208

href="./filter.html"

209

title="How to index non-text documents">Filters</a>

210

211

</li></ul>

212

213

214

215

</li></ul>

216

217

218

219

220

</div>

221

222

223

224

225

226

227

<h1>SWISH-CONFIG - Configuration File Directives</h1>

228

Swish-e version 2.4.5

229

230

231

232

233

234

235

<h2>Table of Contents</h2>

236

237

238

239

240

<li>

241

<a href="#overview">OVERVIEW</a>

242

243

</li>

244

245

<li>

246

<a href="#configuration_file">CONFIGURATION FILE</a>

247

248

249

250

<li>

251

<a href="#alphabetical_listing_of_directives">Alphabetical Listing of Directives</a>

252

253

</li>

254

255

<li>

256

<a href="#directives_that_control_swish">Directives that Control Swish</a>

257

258

</li>

259

260

<li>

261

<a href="#administrative_headers_directives">Administrative Headers Directives</a>

262

263

</li>

264

265

<li>

266

<a href="#document_source_directives">Document Source Directives</a>

267

268

</li>

269

270

<li>

271

<a href="#document_contents_directives">Document Contents Directives</a>

272

273

</li>

274

275

<li>

276

<a href="#directives_for_the_file_access_method_only">Directives for the File Access method only</a>

277

278

</li>

279

280

<li>

281

<a href="#directives_for_the_http_access_method_only">Directives for the HTTP Access Method Only</a>

282

283

</li>

284

285

<li>

286

<a href="#directives_for_the_prog_access_method_only">Directives for the prog Access Method Only</a>

287

288

</li>

289

290

<li>

291

<a href="#document_filter_directives">Document Filter Directives</a>

292

293

294

295

<li>

296

<a href="#filtering_with_swish_filter">Filtering with SWISH::Filter</a>

297

298

</li>

299

300

<li>

301

<a href="#filtering_with_the_filefilter_feature">Filtering with the FileFilter feature</a>

302

303

</li>

304

305

</ul>

306

307

</li>

308

309

</ul>

310

311

</li>

312

313

<li>

314

<a href="#document_info">Document Info</a>

315

316

</li>

317

318

</ul>

319

320

</div>

321

322

323

324

325

326

327

328

<hr />

329

330

331

332

333

<h1><a name="overview"></a>OVERVIEW</h1>

334

335

This document lists the available configuration directives available in

336

Swish-e.

337

338

</div>

339

340

341

342

<h1><a name="configuration_file"></a>CONFIGURATION FILE</h1>

343

344

What files Swish-e indexes and how they are indexed, and where the index

345

is written can be controlled by a configuration file.

346

The configuration file is a text file composed of comments, blank

347

lines, and configuration directives. The order of the directives

348

is not important. Some directives may be used more than once in the

349

configuration file, while others can only be used once (e.g. additional

350

directives will overwrite preceding directives). Case of the directive

351

is not important -- you may use upper, lower, or mixed case.

352

Comments are any line that begin with a "#".

353

<pre class="pre-section"> # This is a comment</pre>

354

As of 2.4.3 lines may be continued by placing a backslas as the last character

355

on the line:

356

<pre class="pre-section"> IgnoreWords \

357

am \

358

the \

359

foo</pre>

360

Directives may take more than one parameter. Enclose single parameters

361

that include whitespace in quotes (single or double). Inside of quotes

362

the backslash escapes the next character.

363

<pre class="pre-section"> ReplaceRules append "foo bar" <- define "foo bar" as a single parameter</pre>

364

If you need to include a quote character in the value either use a

365

backslash to escape it, or enclose it in quotes of the other type.

366

Backslashes also have special meaning in regular expressions.

367

<pre class="pre-section"> FileFilterMatch pdftotext "'%p' -" /\.pdf$/</pre>

368

This says that the dot is a real dot (instead of matching any character).

369

If you place the regular expression in quotes then you must use

370

double-backslashes.

371

<pre class="pre-section"> FileFilterMatch pdftotext "'%p' -" "/\\.pdf$/"</pre>

372

Swish-e will convert the double backslash into a single backslash before

373

passing the parameter to the regular expression compiler.

374

Commented example configuration files are included in the conf

375

directory of the Swish-e distribution.

376

Some command line arguments can override directives specified in the

377

configuration file. Please see also the <a href="swish-run.html">SWISH-RUN</a> for

378

instructions on running Swish-e, and the <a href="swish-search.html">SWISH-SEARCH</a> page for

379

information and examples on how to search your index.

380

The configuration file is specified to Swish-e by the <code>-c</code> switch. For

381

example,

382

<pre class="pre-section"> swish-e -c myconfig.conf</pre>

383

You may also split your directives up into different configuration files. This

384

allows you to have a master configuration file used for many different indexes,

385

and smaller configuration files for each separate index. You can specify the

386

different configuration files when running from the command line with the <code>-c</code>

387

switch (see <a href="swish-run.html">SWISH-RUN</a>), or you may include other Configuration

388

file with the IncludeConfigFile directive below.

389

Typically, in a configuration file the directives are grouped together in

390

some logical order -- that is, directives that control the source of the

391

documents would be grouped together first, and directives that control

392

how each document is filtered or its words index in another group of

393

directives. (The directives listed below are grouped in this order).

394

The configuration file directives are listed below in these groups:

395

<ul>

396

<li>

397

<a href="#administrative_headers_directives">Administrative Headers Directives</a> -- You may add administrative

398

information to the header of the index file.

399

</li>

400

<li>

401

<a href="#document_source_directives">Document Source Directives</a> -- Directives for selecting the source

402

documents and the location of the index file.

403

</li>

404

<li>

405

<a href="#document_contents_directives">Document Contents Directives</a> -- Directives that control how a document

406

content is indexed.

407

</li>

408

<li>

409

<a href="#directives_for_the_file_access_method_only">Directives for the File Access method only</a> -- These directives are only

410

applicable to the File Access indexing method.

411

</li>

412

<li>

413

<a href="#directives_for_the_http_access_method_only">Directives for the HTTP Access Method Only</a> -- Likewise, these only apply

414

to the HTTP Access method.

415

</li>

416

<li>

417

<a href="#directives_for_the_prog_access_method_only">Directives for the prog Access Method Only</a> -- These only apply to the prog

418

Access method.

419

</li>

420

<li>

421

<a href="#document_filter_directives">Document Filter Directives</a> -- This is a special section that describes

422

using document filters with Swish-e.

423

</li>

424

</ul>

425

426

</div>

427

428

429

430

<h2><a name="alphabetical_listing_of_directives"></a>Alphabetical Listing of Directives</h2>

431

432

<ul>

433

<li>

434

<a href="#absolutelinks">AbsoluteLinks</a> [yes|NO]

435

</li>

436

<li>

437

<a href="#begincharacters">BeginCharacters</a> *string of characters*

438

</li>

439

<li>

440

<a href="#bumppositioncountercharacters">BumpPositionCounterCharacters</a> *string*

441

</li>

442

<li>

443

<a href="#buzzwords">Buzzwords</a> [*list of buzzwords*|File: path]

444

</li>

445

<li>

446

<a href="#compresspositions">CompressPositions</a> [yes|NO]

447

</li>

448

<li>

449

<a href="#converthtmlentities">ConvertHTMLEntities</a> [YES|no]

450

</li>

451

<li>

452

<a href="#defaultcontents">DefaultContents</a> [TXT|HTML|XML|TXT2|HTML2|XML2|TXT*|HTML*|XML*]

453

</li>

454

<li>

455

<a href="#delay">Delay</a> *seconds*

456

</li>

457

<li>

458

<a href="#dontbumppositiononendtags">DontBumpPositionOnEndTags</a> *list of names*

459

</li>

460

<li>

461

<a href="#dontbumppositiononstarttags">DontBumpPositionOnStartTags</a> *list of names*

462

</li>

463

<li>

464

<a href="#enablealtsearchsyntax">EnableAltSearchSyntax</a> [yes|NO]

465

</li>

466

<li>

467

<a href="#endcharacters">EndCharacters</a> *string of characters*

468

</li>

469

<li>

470

<a href="#equivalentserver">EquivalentServer</a> *server alias*

471

</li>

472

<li>

473

<a href="#extractpath">ExtractPath</a> *metaname* [replace|remove|prepend|append|regex]

474

</li>

475

<li>

476

<a href="#filefilter">FileFilter</a> *suffix* *program* [options]

477

</li>

478

<li>

479

<a href="#filefiltermatch">FileFilterMatch</a> *program* *options* *regex* [*regex* ...]

480

</li>

481

<li>

482

<a href="#fileinfocompression">FileInfoCompression</a> [yes|NO]

483

</li>

484

<li>

485

<a href="#filematch">FileMatch</a> [contains|is|regex] *regular expression*

486

</li>

487

<li>

488

<a href="#filerules">FileRules</a> [contains|is|regex] *regular expression*

489

</li>

490

<li>

491

<a href="#fuzzyindexingmode">FuzzyIndexingMode</a> [NONE|Stemming|Soundex|Metaphone|DoubleMetaphone]

492

</li>

493

<li>

494

<a href="#followsymlinks">FollowSymLinks</a> [yes|NO]

495

</li>

496

<li>

497

<a href="#htmllinksmetaname">HTMLLinksMetaName</a> *metaname*

498

</li>

499

<li>

500

<a href="#ignorefirstchar">IgnoreFirstChar</a> *string of characters*

501

</li>

502

<li>

503

<a href="#ignorelastchar">IgnoreLastChar</a> *string of characters*

504

</li>

505

<li>

506

<a href="#ignorelimit">IgnoreLimit</a> *integer integer*

507

</li>

508

<li>

509

<a href="#ignoremetatags">IgnoreMetaTags</a> *list of names*

510

</li>

511

<li>

512

<a href="#ignorenumberchars">IgnoreNumberChars</a> *list of characters*

513

</li>

514

<li>

515

<a href="#ignoretotalwordcountwhenranking">IgnoreTotalWordCountWhenRanking</a> [YES|no]

516

</li>

517

<li>

518

<a href="#ignorewords">IgnoreWords</a> [*list of stop words*|File: path]

519

</li>

520

<li>

521

<a href="#imagelinksmetaname">ImageLinksMetaName</a> *metaname*

522

</li>

523

<li>

524

<a href="#includeconfigfile">IncludeConfigFile</a>

525

</li>

526

<li>

527

<a href="#indexadmin">IndexAdmin</a> *text*

528

</li>

529

<li>

530

<a href="#indexalttagmetaname">IndexAltTagMetaName</a> *tagname*|as-text

531

</li>

532

<li>

533

<a href="#indexcomments">IndexComments</a> [yes|NO]

534

</li>

535

<li>

536

<a href="#indexcontents">IndexContents</a> [TXT|HTML|XML|TXT2|HTML2|XML2|TXT*|HTML*|XML*] *file

537

extensions*

538

</li>

539

<li>

540

<a href="#indexdescription">IndexDescription</a> *text*

541

</li>

542

<li>

543

<a href="#indexdir">IndexDir</a> [URL|directories or files]

544

</li>

545

<li>

546

<a href="#indexfile">IndexFile</a> *path*

547

</li>

548

<li>

549

<a href="#indexname">IndexName</a> *text*

550

</li>

551

<li>

552

<a href="#indexonly">IndexOnly</a> *list of file suffixes*

553

</li>

554

<li>

555

<a href="#indexpointer">IndexPointer</a> *text*

556

</li>

557

<li>

558

<a href="#indexreport">IndexReport</a> [0|1|2|3]

559

</li>

560

<li>

561

<a href="#maxdepth">MaxDepth</a> *integer*

562

</li>

563

<li>

564

<a href="#maxwordlimit">MaxWordLimit</a> *integer*

565

</li>

566

<li>

567

<a href="#metanamealias">MetaNameAlias</a> *meta name* *list of aliases*

568

</li>

569

<li>

570

<a href="#metanames">MetaNames</a> *list of names*

571

</li>

572

<li>

573

<a href="#minwordlimit">MinWordLimit</a> *integer*

574

</li>

575

<li>

576

<a href="#nocontents">NoContents</a> *list of file suffixes*

577

</li>

578

<li>

579

<a href="#obeyrobotsnoindex">obeyRobotsNoIndex</a> [yes|NO]

580

</li>

581

<li>

582

<a href="#parserwarnlevel">ParserWarnLevel</a> [0|1|2|3]

583

</li>

584

<li>

585

<a href="#presortedindex">PreSortedIndex</a> *list of property names*

586

</li>

587

<li>

588

<a href="#propcompressionlevel">PropCompressionLevel</a> [0-9]

589

</li>

590

<li>

591

<a href="#propertynamealias">PropertyNameAlias</a> *property name* *list of aliases*

592

</li>

593

<li>

594

<a href="#propertynames">PropertyNames</a> *list of meta names*

595

</li>

596

<li>

597

<a href="#propertynamescomparecase">PropertyNamesCompareCase</a> *list of meta names*

598

</li>

599

<li>

600

<a href="#propertynamesignorecase">PropertyNamesIgnoreCase</a> *list of meta names*

601

</li>

602

<li>

603

<a href="#propertynamesnostripchars">PropertyNamesNoStripChars</a> *list of meta names*

604

</li>

605

<li>

606

<a href="#propertynamesdate">PropertyNamesDate</a> *list of meta names*

607

</li>

608

<li>

609

<a href="#propertynamesnumeric">PropertyNamesNumeric</a> *list of meta names*

610

</li>

611

<li>

612

<a href="#propertynamesmaxlength">PropertyNamesMaxLength</a> integer *list of meta names*

613

</li>

614

<li>

615

<a href="#propertynamessortkeylength">PropertyNamesSortKeyLength</a> integer *list of meta names*

616

</li>

617

<li>

618

<a href="#replacerules">ReplaceRules</a> [replace|remove|prepend|append|regex]

619

</li>

620

<li>

621

<a href="#resultextformatname">ResultExtFormatName</a> name -x format string

622

</li>

623

<li>

624

<a href="#spiderdirectory">SpiderDirectory</a> *path*

625

</li>

626

<li>

627

<a href="#storedescription">StoreDescription</a> [XML <tag>|HTML <meta>|TXT size]

628

</li>

629

<li>

630

<a href="#swishprogparameters">"SwishProgParameters</a> *list of parameters*

631

</li>

632

<li>

633

<a href="#swishsearchdefaultrule">SwishSearchDefaultRule</a> [<AND-WORD>|<or-word>]

634

</li>

635

<li>

636

<a href="#tmpdir">TmpDir</a> *path*

637

</li>

638

<li>

639

<a href="#translatecharacters">TranslateCharacters</a> [*string1 string2*|:ascii7:]

640

</li>

641

<li>

642

<a href="#truncatedocsize">TruncateDocSize</a> *number of characters*

643

</li>

644

<li>

645

<a href="#undefinedmetatags">UndefinedMetaTags</a> [error|ignore|INDEX|auto]

646

</li>

647

<li>

648

<a href="#undefinedxmlattributes">UndefinedXMLAttributes</a> [DISABLE|error|ignore|index|auto]

649

</li>

650

<li>

651

<a href="#usestemming">UseStemming</a> [yes|NO]

652

</li>

653

<li>

654

<a href="#usesoundex">UseSoundex</a> [yes|NO]

655

</li>

656

<li>

657

<a href="#usewords">UseWords</a> [*list of words*|File: path]

658

</li>

659

<li>

660

<a href="#wordcharacters">WordCharacters</a> *string of characters*

661

</li>

662

<li>

663

<a href="#xmlclassattributes">XMLClassAttributes</a> *list of XML attribute names*

664

</li>

665

</ul>

666

667

</div>

668

669

670

671

<h2><a name="directives_that_control_swish"></a>Directives that Control Swish</h2>

672

673

These configuration directives control the general behavior of Swish-e.

674

<ul>

675

<li><a name="item_includeconfigfile"></a><a name="includeconfigfile"></a>IncludeConfigFile *path to config file*

676

This directive can be used to include configuration directives located

677

in another file.

678

<pre class="pre-section"> IncludeConfigFile /usr/local/swish/conf/site_config.config</pre>

679

</li>

680

<li><a name="item_indexreport"></a><a name="indexreport"></a>IndexReport [0|1|2|3]

681

This is how detailed you want reporting while indexing. You can specify

682

numbers 0 to 3. 0 is totally silent, 3 is the most verbose. The default

683

is 1.

684

This may be overridden from the command line via the <code>-v</code> switch (see

685

<a href="swish-run.html">SWISH-RUN</a>).

686

</li>

687

<li><a name="item_parserwarnlevel"></a><a name="parserwarnlevel"></a>ParserWarnLevel [0|1|2|3]

688

Sets the error level when using the libxml2 parser for XML and HTML.

689

libxml2 will point out structural errors in your documents.

690

<pre class="pre-section"> 0 = no report

691

1 = fatal errors

692

2 = errors

693

3 = warnings</pre>

694

Currently (as of 2.4.4 - early 2005) libxml2 only reports errors at level 2.

695

The default as of 2.4.4 is "2" which should report any errors that might indicate

696

a problem parsing a document.

697

The exception to this is UTF-8 to Latin-1 conversion errors are reported at

698

level 3 (changed from 1 in 2.4.4). Although these errors indicate a problem indexing

699

text, they are only reported at level 3 because they can be very common.

700

It is recommended that you index at ParserWarnLevel 3 when first starting out to see

701

what errors and warnings are reported. Then reduce the level when you understand what

702

documents are causing parsing problems and why.

703

</li>

704

<li><a name="item_indexfile"></a><a name="indexfile"></a>IndexFile *path*

705

Index file specifies the location of the generated index file. If not

706

specified, Swish-e will create the file index.swish-e in the current

707

directory.

708

<pre class="pre-section"> IndexFile /usr/local/swish/site.index</pre>

709

</li>

710

<li><a name="item_obeyrobotsnoindex"></a><a name="obeyrobotsnoindex"></a>obeyRobotsNoIndex [yes|NO]

711

When enabled, Swish-e will not index any HTML file that contains:

712

713

The default is to ignore these meta tags and index the document.

714

This tag is described at <a href="http://www.robotstxt.org/wc/exclusion.html">http://www.robotstxt.org/wc/exclusion.html</a>.

715

Note: This feature is only available with the libxml2 HTML parser.

716

Also, if you are using the libxml2 parser (HTML2 and XML2) then you can use the following

717

comments in your documents to prevent indexing:

718

719

</pre>

720

and/or these may be used also:

721

722

</pre>

723

For example, these are very helpful to prevent indexing of common headers, footers, and menus.

724

</li>

725

</ul>

726

NOTE: This following items are currently not available. These items

727

require Swish-e to parse the configuration file while searching.

728

<ul>

729

<li><a name="item_enablealtsearchsyntax"></a><a name="enablealtsearchsyntax"></a>EnableAltSearchSyntax [yes|NO]

730

NOTE: This following item is currently not available.

731

Enable alternate search syntax. Allows the usage of a basic

732

"Altavista(c)", "Lycos(c)", etc. like search syntax. This means a search

733

query can contain "+" and "-" as syntax parameter.

734

Example:

735

<pre class="pre-section"> swish-e -w "+word1 +word2 -word3 word4 word5"

736

"+" = following word has to be in all found documents

737

"-" = following word may not be in any document found

738

" " = following word will be searched in documents</pre>

739

</li>

740

<li><a name="item_swishsearhoperators"></a><a name="swishsearhoperators"></a>SwishSearhOperators <and-word> <or-word> <not-word>

741

NOTE: This following item is currently not available.

742

Using this config directive you can change the boolean search operators of

743

Swish-e, e.g. to adapt these to your language.

744

The default is: AND OR NOT

745

Example (german):

746

<pre class="pre-section"> SwishSearchOperators UND ODER NICHT</pre>

747

</li>

748

<li><a name="item_swishsearchdefaultrule"></a><a name="swishsearchdefaultrule"></a>SwishSearchDefaultRule [<AND-WORD>|<or-word>]

749

NOTE: This following item is currently not available.

750

<code>SwishSearchDefaultRule</code> defines the default Boolean operator to use if none

751

is specified between words or phrases. The default is <code>AND</code>.

752

The word you specify must match one of the available <code>SwishSearchOperators</code>.

753

Example:

754

<pre class="pre-section"> SwishSearchOperators UND ODER NICHT

755

# Make it act like a web search engine

756

SwishSearchDefaultRule ODER</pre>

757

</li>

758

<li><a name="item_resultextformatname"></a><a name="resultextformatname"></a>ResultExtFormatName name -x format string

759

NOTE: This following item is currently not available.

760

The output of Swish-e can be defined by specifying a format string with the

761

<code>-x</code> command line argument. Using <code>ResultExtFormatName</code> you can assign a

762

predefined format string to a name.

763

Examples:

764

<pre class="pre-section"> ResultExtFormatName moreinfo "%c|%r|%t|%p|<author>|<publishyear>\n"</pre>

765

Then when searching you can specify the format string's name

766

<pre class="pre-section"> swish-e ... -x moreinfo ...</pre>

767

See the <code>-x</code> switch in <a href="swish-run.html">SWISH-RUN</a> for more information about

768

output formats.

769

</li>

770

</ul>

771

772

</div>

773

774

775

776

<h2><a name="administrative_headers_directives"></a>Administrative Headers Directives</h2>

777

778

Swish-e stores configuration information in the header of the index file.

779

This information can be retrieved while searching or by functions in

780

the Swish-e C library. There are a number of fields available for your

781

own use. None of these fields are required:

782

<ul>

783

<li><a name="item_indexname"></a><a name="indexname"></a>IndexName *text*

784

</li>

785

<li><a name="item_indexdescription"></a><a name="indexdescription"></a>IndexDescription *text*

786

</li>

787

<li><a name="item_indexpointer"></a><a name="indexpointer"></a>IndexPointer *text*

788

</li>

789

<li><a name="item_indexadmin"></a><a name="indexadmin"></a>IndexAdmin *text*

790

These variables specify information that goes into index files to help

791

users and administrators. IndexName should be the name of your index,

792

like a book title. IndexDescription is a short description of the index

793

or a URL pointing to a more full description. IndexPointer should be

794

a pointer to the original information, most likely a URL. IndexAdmin

795

should be the name of the index maintainer and can include name and email

796

information. These values should not be more than 70 or so characters

797

and should be contained in quotes. Note that the automatically generated

798

date in index files is in D/M/Y and 24-hour format.

799

Examples:

800

<pre class="pre-section"> IndexName "Linux Documentation"

801

IndexDescription "This is an index of /usr/doc on our Linux machine."

802

IndexPointer http://localhost/swish/linux/index.html

803

IndexAdmin webmaster</pre>

804

</li>

805

</ul>

806

807

</div>

808

809

810

811

<h2><a name="document_source_directives"></a>Document Source Directives</h2>

812

813

These directives control what documents are indexed and how they are

814

accessed. See also <a href="#directives_for_the_file_access_method_only">Directives for the File Access method only</a> and <a href="#directives_for_the_http_access_method_only">Directives for the HTTP Access Method Only</a> for directives that are

815

specific to those access methods.

816

<ul>

817

<li><a name="item_indexdir"></a><a name="indexdir"></a>IndexDir [directories or files|URL|external program]

818

IndexDir defines the source of the documents for Swish-e. Swish-e

819

currently supports three file access methods: File system, HTTP

820

(also called spidering), and prog for reading files from an

821

external program.

822

The <code>-S</code> command line argument is used to select the file access method.

823

<pre class="pre-section"> swish-e -c swish.config -S fs - file system

824

swish-e -c swish.config -S http - internal http spider

825

swish-e -c swish.config -S prog - external program of any type</pre>

826

For the fs method of access IndexDir is a space-separated

827

list of files and directories to index. Use a forward slash as the path

828

separator in MS Windows.

829

For the http method the IndexDir setting is a list of space-separated

830

URLs.

831

For the prog method the IndexDir setting is a list of space-separated

832

programs to run (which generate documents for swish to index).

833

You may specify more than one IndexDir directive.

834

Any sub-directories of any listed directory will also be indexed.

835

Note: While processing directories, Swish-e will ignore any files or

836

directories that begin with a dot ("."). You may index files or directories

837

that begin with a dot by specifying their name with <code>IndexDir</code> or <code>-i</code>.

838

Examples:

839

<pre class="pre-section"> # Index this directory an any subdirectories

840

IndexDir /usr/local/home/http

841

842

# Index the docs directory in current directory

843

IndexDir ./docs

844

845

# Index these files in the current directory

846

IndexDir ./index.html ./page1.html ./page2.html

847

# and index this directory, too

848

IndexDir ../public_html</pre>

849

For the HTTP method of access specify the URL's from which

850

you want the spidering to begin.

851

Example:

852

<pre class="pre-section"> IndexDir http://www.my-site.com/index.html

853

IndexDir http://localhost/index.html</pre>

854

Obviously, using the HTTP method to index is much slower than indexing

855

local files. Be well aware that some sites do not appreciate spidering and may

856

block your IP address. You may wish to contact the remote site before

857

spidering their web site. More information about spidering can be found in

858

<a href="#directives_for_the_http_access_method_only">Directives for the HTTP Access Method Only</a> below.

859

For the <a href="swish-run.html#item_prog">prog</a> method of access IndexDir specifies

860

the path to the program(s) to execute. The external program must correctly

861

format the documents being passed back to Swish-e. Examples of external

862

programs are provided in the prog-bin directory.

863

<pre class="pre-section"> IndexDir ./myprogram.pl</pre>

864

See <a href="swish-run.html#item_prog">prog</a> for details.

865

Note: Not all directives work with all methods.

866

</li>

867

<li><a name="item_nocontents"></a><a name="nocontents"></a>NoContents *list of file suffixes*

868

Files with these suffixes will not have their contents indexed,

869

but will have their path name (file name) indexed instead.

870

If the file's type is HTML or HTML2 (as set by <code>IndexContents</code> or

871

<code>DefaultContents</code>) then the file will be parsed for a HTML title and that

872

title will be indexed. Note that you must set the file's type with

873

<code>IndexContents</code> or <code>DefaultContents</code>: <code>.html</code> and <code>.htm</code> are NOT type HTML

874

by default. For example:

875

<pre class="pre-section"> IndexContents HTML* .htm .html</pre>

876

If a title is found, it will still be checked for <code>FileRules title</code>, and the

877

file will be skipped if a match is found. See <code>FileRules</code>.

878

If the file's type is not HTML, or it is HTML and no title is found,

879

then the file's path will be indexed.

880

For example, this will allow searching by image file name.

881

<pre class="pre-section"> NoContents .gif .xbm .au .mov .mpg .pdf .ps</pre>

882

Note: Using this directive will not cause files with those suffixes to be

883

indexed. That is, if you use <code>IndexOnly</code> to limit the types of files that are

884

indexed, then you must specify in <code>IndexOnly</code> the same suffixes listed in

885

<code>NoContents</code>.

886

This does not work:

887

<pre class="pre-section"> # Wrong!

888

IndexOnly .htm .html

889

NoContents .gif .xbm .au .mov .mpg .pdf .ps</pre>

890

A <code>-S prog</code> program may set the <code>No-Contents:</code> header to enable this feature

891

for a specific document (although it would be smarter for the <code>-S prog</code>

892

program to simply only send the pathname or title to be indexed.

893

</li>

894

<li><a name="item_replacerules"></a><a name="replacerules"></a>ReplaceRules [replace|remove|prepend|append|regex]

895

ReplaceRules allows you to make changes to file pathnames before

896

they're indexed. These changed file names or URLs will be returned in

897

search results.

898

For example, you may index your files locally (with the File system

899

indexing method), yet return a URL in search results. This directive can

900

be used to map the file names to their respective URLs on your web server.

901

There are five operations you can specify: replace, append,

902

remove, prepend, and regex They will parse the pathname in the

903

order you've typed these commands.

904

This directive uses C library regex.h regular expressions.

905

<pre class="pre-section"> replace "the string you want replaced" "what to change it to"

906

remove "a string to remove"

907

prepend "a string to add before the result"

908

append "a string to add after the result"

909

regex "/search string/replace string/options"</pre>

910

Remember, quotes are needed if an expression contains white space,

911

and backslashes have special meaning.

912

Regex is an Extended Regular Expression. The first character found is

913

the delimiter (but it's not smart enough to use matched chars such as [],

914

(), and {}).

915

The replace string may use substitution variables:

916

<pre class="pre-section"> $0 the entire matched (sub)string

917

$1-$9 returns patterns captured in "(" ")" pairs

918

$` the string before the matched pattern

919

$' the string after the matched pattern</pre>

920

The options change the behavior of expression:

921

<pre class="pre-section"> i ignore the case when matching

922

g repeat the substitution for the entire pattern</pre>

923

Examples:

924

<pre class="pre-section"> ReplaceRules replace testdir/ anotherdir/

925

ReplaceRules replace [a-z_0-9]*_m.*\.html index.html

926

927

ReplaceRules remove testdir/

928

929

ReplaceRules prepend http://localhost/

930

ReplaceRules append .html

931

932

ReplaceRules regex !^/web/(.+)/!http://$1.domain.com/!

933

replaces a file path:

934

/web/search/foo/index.html

935

with

936

http://search.domain.com/foo/index.html

937

938

ReplaceRules regex #^#http://localhost/www#

939

ReplaceRules prepend http://localhost/www (same thing)

940

941

# Remove all extensions from C source files

942

ReplaceRules remove .c # ERROR! That "." is *any char*

943

ReplaceRules remove \.c # much better...

944

945

ReplaceRules remove "\\.c" # if in quotes you need double-backslash!

946

ReplaceRules remove "\.c" # ERROR! "\." -> "." and is *any char*</pre>

947

</li>

948

<li><a name="item_indexcontents"></a><a name="indexcontents"></a>IndexContents [TXT|HTML|XML|TXT2|HTML2|XML2|TXT*|HTML*|XML*] *file extensions*

949

The <code>IndexContents</code> directive assigns one of Swish-e's document parsers to a

950

document, based on the its extension. Swish-e currently knows how to parse

951

TXT, HTML, and XML documents.

952

The XML2, HTML2, and TXT2 parsers are currently only available when

953

Swish-e is configured to use libxml2.

954

You may use XML*, HTML*, and TXT* to select the parser automatically.

955

If libxml2 is installed then it will be used to parse the content. Otherwise,

956

Swish-e's internal parsers will be used.

957

Documents that are not assigned a parser with <code>IndexContents</code> will, by

958

default, use the HTML2 parser if libxml2 is installed, otherwise will use

959

Swish-e's internal HTML parser. The <code>DefaultContents</code> directive may be used

960

to assign a parser to documents that do not match a file extension defined with

961

the <code>IndexContents</code> directive.

962

Example:

963

<pre class="pre-section"> IndexContents HTML* .htm .html .shtml

964

IndexContents TXT* .txt .log .text

965

IndexContents XML* .xml</pre>

966

HTML* is the default type for all files, unless otherwise specified (and this

967

default can be changed by the DefaultContents directive. Swish-e parses

968

titles from HTML files, if available, and keeps track of the context of the

969

text for context searching (see <code>-t</code> in <a href="swish-run.html">SWISH-RUN</a>).

970

If using filters (with the <code>FileFilter</code> directive) to convert documents you

971

should include those extensions, too. For example, if using a filter to

972

convert .pdf to .html, you need to tell Swish-e that .pdf should be indexed by

973

the internal HTML parser:

974

<pre class="pre-section"> FileFilter .pdf pdf2html

975

IndexContent HTML .pdf</pre>

976

See also <a href="#document_filter_directives">Document Filter Directives</a>.

977

Note: Some of this may be changed in the future to use content-types instead

978

of file extensions. See <a href="swish-3.0.html">SWISH-3.0</a>

979

</li>

980

<li><a name="item_defaultcontents"></a><a name="defaultcontents"></a>DefaultContents [TXT|HTML|XML|TXT2|HTML2|XML2|TXT*|HTML*|XML*]

981

This sets the default parser for documents that are not specified in

982

IndexContents. If not specified the default is HTML.

983

The XML2, HTML2, and TXT2 parsers are currently only available when

984

Swish-e is configured to use libxml2.

985

You may use XML*, HTML*, and TXT* to select the parser automatically.

986

If libxml2 is installed then it will be used to parse the content. Otherwise,

987

Swish-e's internal parsers will be used.

988

Example:

989

<pre class="pre-section"> DefaultContents HTML</pre>

990

The <code>DefaultContents</code> directive should be used when spidering, as HTML

991

files may be returned without a file extension (such as when requesting a

992

directory and the default index.html is returned).

993

</li>

994

<li><a name="item_fileinfocompression"></a><a name="fileinfocompression"></a>FileInfoCompression [yes|NO]

995

** This directive is currently not supported **

996

Setting FileInfoCompression to <code>yes</code> will compress the index file to save

997

disk space. This may result in longer indexing times. The default is <code>no</code>.

998

Also see the <code>-e</code> switch in <a href="swish-run.html">SWISH-RUN</a> for saving RAM during

999

indexing.

1000

</li>

1001

</ul>

1002

1003

</div>

1004

1005

1006

1007

<h2><a name="document_contents_directives"></a>Document Contents Directives</h2>

1008

1009

These directives control what information is extracted from your source

1010

documents, and how that information is made available during searching.

1011

<ul>

1012

<li><a name="item_converthtmlentities"></a><a name="converthtmlentities"></a>ConvertHTMLEntities [YES|no]

1013

ASCII entities can be converted automatically while indexing documents of

1014

type HTML (not for HTML2). For performance reasons you may wish to set this to

1015

<code>no</code> if your documents do not contain HTML entities. The default is <code>yes</code>.

1016

If <code>ConvertHTMLEntities</code> is set <code>no</code> the entities will be indexed without

1017

conversion.

1018

NOTE: Entities within XML files and files parsed with libxml2 (HTML2) are

1019

converted regardless of this setting.

1020

</li>

1021

<li><a name="item_metanames"></a><a name="metanames"></a>MetaNames *list of names*

1022

META names are a way to define "fields" in your XML and HTML documents. You

1023

can use the META names in your queries to limit the search to just the words

1024

contained in that META name of your document. For example, you might have a

1025

META tagged field in your documents called <code>subjects</code> and then you can search

1026

your documents for the word "foo" but only return documents where "foo" is

1027

within the <code>subjects</code> META tag.

1028

<pre class="pre-section"> swish-e -w subjects=foo</pre>

1029

(See also the <code>-t</code> switch in <a href="swish-run.html">SWISH-RUN</a> for information about

1030

context searching in HTML documents.)

1031

The MetaNames directive is a space separated list. For example:

1032

<pre class="pre-section"> MetaNames meta1 meta2 keywords subjects</pre>

1033

You may also use <code>UndefinedMetaTags</code> to specify automatic extraction of meta

1034

names from your HTML and XML documents, and also to ignore indexing content of

1035

meta tags.

1036

META tags can have two formats in your HTML source documents:

1037

1038

and (if using the HTML2/libxml2 parser)

1039

1040

some content

1041

</meta1></pre>

1042

But this second version is invalid HTML, and will generate a warning if

1043

ParserWarningLevel is set (libxml2 only).

1044

And in XML documents, use the format:

1045

1046

Some Content

1047

</meta1></pre>

1048

Then you can limit your search to just META meta1 like this:

1049

<pre class="pre-section"> swish-e -w 'meta1=(apples or oranges)'</pre>

1050

You may nest the XML and the start/end tag versions:

1051

1052

<tag1>

1053

some content

1054

</tag1>

1055

<tag2>

1056

some other content

1057

</tag2>

1058

1059

Then you can search in both tag2 and tag2 with:

1060

<pre class="pre-section"> swish-e -w 'keywords=(query words)'</pre>

1061

Swish-e indexes all text as some metaname. The default is <code>swishdefault</code>, so

1062

these two queries are the same:

1063

<pre class="pre-section"> swish-e -w foo

1064

swish-e -w swishdefault=foo</pre>

1065

When indexing HTML Swish-e indexes the HTML title as default text, so

1066

when searching Swish-e will find matches in both the HTML body and the

1067

HTML title. Swish also, by default, indexes content of meta tags. So:

1068

<pre class="pre-section"> swish-e -w foo</pre>

1069

will find "foo" in the body, the title, or any meta tags.

1070

Currently, there's no way to prevent Swish-e from indexing the title contents

1071

along with the body contents, but see <code>UndefinedMetaTags</code> for how to control

1072

the indexing of meta tags.

1073

If you would like to search just the title text, you may use:

1074

<pre class="pre-section"> MetaNames swishtitle</pre>

1075

This will index the title text separately under the built-in swish

1076

internal meta name "swishtitle". You may then search like

1077

<pre class="pre-section"> swish-e -w foo -- search for "foo" in title, body (and undefined meta tags)

1078

swish-e -w swishtitle=foo -- search for "foo" in title only</pre>

1079

In addition to swishtitle, you can limit searches to documents' path with:

1080

<pre class="pre-section"> MetaNames swishdocpath</pre>

1081

Then to search for "foo" but also limit searches to documents that include

1082

"manual" or "tutorial" in their path:

1083

<pre class="pre-section"> swish-e -w foo swishdocpath=(manual or tutorial)</pre>

1084

See also <code>ExtractPath</code>.

1085

</li>

1086

<li><a name="item_metanamealias"></a><a name="metanamealias"></a>MetaNameAlias *meta name* *list of aliases*

1087

MetaNameAlias assigns aliases for a meta name. For example, if your

1088

documents contain meta tags "description", "summary", and "overview"

1089

that all give a summary of your documents you could do this:

1090

<pre class="pre-section"> MetaNames summary

1091

MetaNameAlias summary description overview</pre>

1092

Then all three tags will get indexed as meta tag "summary". You can

1093

then search all the fields as:

1094

<pre class="pre-section"> -w summary=foo</pre>

1095

The Alias work at search time, too. So these will also limit the search

1096

to the "summary" meta name.

1097

<pre class="pre-section"> -w description=foo

1098

-w overview=foo</pre>

1099

</li>

1100

<li><a name="item_metanamesrank"></a><a name="metanamesrank"></a>MetaNamesRank integer *list of meta names*

1101

You can assign a bias to metanames that will affect how ranking is

1102

calculated. The range of values is from -10 to +10, with zero being

1103

no bias.

1104

<pre class="pre-section"> MetaNamesRank 4 subject

1105

MetaNamesRank 3 swishdefault

1106

MetaNamesRank 2 author publisher

1107

MetaNamesRank -5 wrongwords</pre>

1108

This feature is still considered experimental. If you use it, please send feedback

1109

to the discussion list.

1110

</li>

1111

<li><a name="item_htmllinksmetaname"></a><a name="htmllinksmetaname"></a>HTMLLinksMetaName *metaname*

1112

Allows indexing of HTML links. Normally, HTML links (href tags) are

1113

not indexed by Swish-e. This directive defines a metaname, and links

1114

will be indexed under this meta name.

1115

Example:

1116

<pre class="pre-section"> HTMLLinksMetaName links</pre>

1117

Now, to limit searches to files with a link to "home.html" do this:

1118

<pre class="pre-section"> -w links='"home.html"'</pre>

1119

The double quotes force a phrase search.

1120

To make Swish-e index links as normal text, you may use:

1121

<pre class="pre-section"> HTMLLinksMetaName swishdefault</pre>

1122

This feature is only available with the libxml2 HTML parser.

1123

</li>

1124

<li><a name="item_imagelinksmetaname"></a><a name="imagelinksmetaname"></a>ImageLinksMetaName *metaname*

1125

Allows indexing of image links under a metaname. Normally, image URLs

1126

are not indexed.

1127

Example:

1128

<pre class="pre-section"> ImagesLinksMetaName images</pre>

1129

Now, if you would like to find pages that include a nice image of a beach:

1130

<pre class="pre-section"> -w images='beach'</pre>

1131

To make Swish-e index links as normal text, you may use:

1132

<pre class="pre-section"> ImageLinksMetaName swishdefault</pre>

1133

This feature is only available with the libxml2 HTML parser.

1134

</li>

1135

<li><a name="item_indexalttagmetaname"></a><a name="indexalttagmetaname"></a>IndexAltTagMetaName *tagname*|as-text

1136

Allows indexing of images <IMG> ALT tag text. Specify either a tag name which will be

1137

used as a metaname, or the special text "as-text" which says to index the ALT text as

1138

if it were plain text at the current location.

1139

For example, by specifying a tag name:

1140

<pre class="pre-section"> IndexAltTagMetaName bar</pre>

1141

would make this markup:

1142

1143

1144

</foo></pre>

1145

appear like

1146

1147

1148

</foo></pre>

1149

Then the normal rules (<code>MetaNames</code> and <code>PropertyNames</code>) apply to how that

1150

text is indexed.

1151

If you use the special tag "as-text" then

1152

1153

1154

</foo></pre>

1155

simply becomes

1156

1157

Alt text here

1158

</foo></pre>

1159

This feature is only available when using the libxml2 parser (HTML2 and XML2).

1160

</li>

1161

<li><a name="item_absolutelinks"></a><a name="absolutelinks"></a>AbsoluteLinks [yes|NO]

1162

If this is set true then Swish-e will attempt to convert relative URIs

1163

extracted from HTML documents for use with <code>HTMLLinksMetaName</code> and

1164

<code>ImageLinksMetaName</code> into absolute URIs. Swish-e will use any <BASE> tag

1165

found in the document, otherwise it will use the file's pathname. The pathname

1166

used will be the pathname *after* <code>ReplaceRules</code> has been applied to the

1167

document's pathname.

1168

For example, say you wish to index image links under the metaname

1169

"images".

1170

<pre class="pre-section"> ImageLinksMetaName images</pre>

1171

If an image is located in <a href="http://localhost/vacations/france/index.html">http://localhost/vacations/france/index.html</a> and

1172

<code>AbsoluteLinks</code> is set to no, then a image within that document:

1173

1174

will only index "beach.jpeg".

1175

But, if you want more detail when searching, you can enable <code>AbsoluteLinks</code>

1176

and Swish-e will index "<a href="http://localhost/vacations/france/beach.jpeg">http://localhost/vacations/france/beach.jpeg</a>". You can

1177

then look for images of beaches, but only in France:

1178

<pre class="pre-section"> -w images=(beach and france)</pre>

1179

This also means you can search for any images within France:

1180

<pre class="pre-section"> -w images=(france)</pre>

1181

This feature is only available with the libxml2 HTML parser.

1182

</li>

1183

<li><a name="item_undefinedmetatags"></a><a name="undefinedmetatags"></a>UndefinedMetaTags [error|ignore|INDEX|auto]

1184

This directive defines the behavior of Swish-e during indexing when a

1185

meta name is found but is not listed in MetaNames. There are

1186

four choices:

1187

<ul>

1188

<li><a name="item_error"></a><a name="error"></a>error

1189

If a meta name is found that is not listed in MetaNames

1190

then indexing will be halted and an error reported.

1191

</li>

1192

<li><a name="item_ignore"></a><a name="ignore"></a>ignore

1193

The contents of the meta tag are ignored and not indexed unless a metaname

1194

has been defined with the <code>MetaNames</code> directive.

1195

</li>

1196

<li><a name="item_index"></a><a name="index"></a>index

1197

The contents of the meta tag are indexed, but placed in the

1198

main index unless there's an enclosing metatag already in force. This

1199

is the default.

1200

</li>

1201

<li><a name="item_auto"></a><a name="auto"></a>auto

1202

This method create meta tags automatically for HTML meta names

1203

and XML elements. Using this is the same as specifying all the meta

1204

names explicitly in a MetaNames directive.

1205

</li>

1206

</ul>

1207

</li>

1208

<li><a name="item_undefinedxmlattributes"></a><a name="undefinedxmlattributes"></a>UndefinedXMLAttributes [DISABLE|error|ignore|index|auto]

1209

This is similar to <code>UndefinedMetaTags</code>, but only applies to XML documents

1210

(parsed with libxml2). This allows indexing of attribute content, and provides

1211

a way to index the content under a metaname. For example,

1212

<code>UndefinedXMLAttributes</code> can make

1213

1214

John Doe

1215

</person></pre>

1216

look like the following to swish:

1217

1218

<person.age>

1219

23

1220

</person.age>

1221

John Doe

1222

</person></pre>

1223

What happens to the text "23" will depend on the setting of

1224

<code>UndefinedXMLAttributes</code>:

1225

<ul>

1226

<li><a name="item_disable"></a><a name="disable"></a>disable

1227

XML attributes are not parsed and not indexed. This is the default.

1228

</li>

1229

<li><a name="item_error"></a><a name="error"></a>error

1230

If the concatenated meta name (e.g. person.age) is not listed in

1231

MetaNames then indexing will be halted and an error reported.

1232

</li>

1233

<li><a name="item_ignore"></a><a name="ignore"></a>ignore

1234

The contents of the meta tag are ignored and not indexed unless a metaname

1235

has been defined with the <code>MetaNames</code> directive.

1236

</li>

1237

<li><a name="item_index"></a><a name="index"></a>index

1238

The contents of the meta tag are indexed, but placed in the main index

1239

unless there's an enclosing metatag already in force.

1240

</li>

1241

<li><a name="item_auto"></a><a name="auto"></a>auto

1242

This method will create meta tags from the combined element and attributes

1243

(and XML Class name) This options should be used with caution as it can

1244

generate a lot of metaname entries.

1245

See also the example below <code>XMLClassAttribues</code>.

1246

</li>

1247

</ul>

1248

</li>

1249

<li><a name="item_xmlclassattributes"></a><a name="xmlclassattributes"></a>XMLClassAttributes *list of XML attribute names*

1250

Combines an XML class name with the element name to make up a metaname.

1251

For example:

1252

<pre class="pre-section"> XMLClassAttributes class

1253

1254

1255

John

1256

</person>

1257

1258

Doe

1259

</person></pre>

1260

Will appear to Swish-e as:

1261

1262

<person.first>

1263

John

1264

</person.first>

1265

</person>

1266

1267

<person.last>

1268

Doe

1269

</person.last>

1270

</person></pre>

1271

How the data is indexed depends on <code>MetaNames</code> and <code>UndefinedMetaTags</code>.

1272

Here's an example using the following configuration which combines the two

1273

directives <code>XMLClassAttributes</code> and <code>UndefinedXMLAttributes</code>.

1274

<pre class="pre-section"> XMLClassAttributes class

1275

UndefinedMetaTags auto

1276

UndefinedXMLAttributes auto

1277

IndexContents XML2 .xml</pre>

1278

The source XML file looks like:

1279

1280

1281

Swish-e parses as:

1282

<pre class="pre-section"> ./swish-e -c 2 -i 1.xml -T parsed_tags parsed_text -v 0

1283

Indexing Data Source: "File-System"

1284

1285

<xml> (MetaName)

1286

1287

<person> (MetaName)

1288

<person.student> (MetaName)

1289

<person.student.phone> (MetaName)

1290

555-1212

1291

</person.student.phone>

1292

<person.student.age> (MetaName)

1293

102

1294

</person.student.age>

1295

John

1296

</person>

1297

1298

<person> (MetaName)

1299

<person.greeting> (MetaName)

1300

howdy

1301

</person.greeting>

1302

Bill

1303

</person>

1304

1305

</xml>

1306

Indexing done!</pre>

1307

One thing to note is that the first <person> block finds a class name

1308

"student" so all metanames that are created from attributes use the

1309

combined name "person.student". The second <person> block doesn't contain

1310

a "class" so, the attribute name is combined directly with the element

1311

name (e.g. "person.greeting").

1312

</li>

1313

<li><a name="item_extractpath"></a><a name="extractpath"></a>ExtractPath *metaname* [replace|remove|prepend|append|regex]

1314

This directive can be used to index extracted parts of a document's path.

1315

A common use would be to limit searches to specific areas of your

1316

file tree.

1317

The extracted string will be indexed under the specified meta name.

1318

See <code>ReplaceRules</code> for a description of the various pattern replacement

1319

methods, but you will use the regex method.

1320

For example, say your file system (or web tree) was organized into departments:

1321

<pre class="pre-section"> /web/sales/foo...

1322

/web/parts/foo...

1323

/web/accounting/foo...</pre>

1324

And you wanted a way to limit searches to just documents under "sales".

1325

<pre class="pre-section"> ExtractPath department regex !^/web/([^/]+)/.*$!$1!</pre>

1326

Which says, extract out the department name (as substring $1) and index it as

1327

meta name <code>department</code>. Then to limit a search to the sales department:

1328

<pre class="pre-section"> swish-e -w foo AND department=sales</pre>

1329

Note that the <code>regex</code> method uses a substitution pattern, so to index only a

1330

sub-string match the entire document path in the regular expression, as

1331

shown above. Otherwise any part that is not matched will end up in the

1332

substitution pattern.

1333

See the <code>ExtractPathDefault</code> option for a way to set a value if not patterns

1334

match.

1335

Although unlikely, you may use more than one <code>ExtractPath</code> directive. More

1336

than one directive of the same meta name will operate successively (in order

1337

listed in the configuration file) on the path. This allows you to use regular

1338

expressions on the results of the previous pattern substitution (as if piping

1339

the output from one expression to the patter of the next).

1340

<pre class="pre-section"> ExtractPath foo regex !^(...).+$!$1!

1341

ExtractPath foo regex !^.+(.)$!$1!</pre>

1342

So, the third letter is indexed as meta name "foo" if both patterns match.

1343

<pre class="pre-section"> ExtractPath foo regex !^X(...).+$!$1!

1344

ExtractPath foo regex !^.+(.)$!$1!</pre>

1345

Now (not the "X"), if the first pattern doesn't match, the last character of

1346

the path name is indexed. You must be clear on this behavior if you are using

1347

more than one <code>ExtractPath</code> directive with the same metaname.

1348

The document path operated on is the real path swish used to access the

1349

document. That is, the <code>ReplaceRules</code> directive has no effect on the path

1350

used with <code>ExtractPath</code>.

1351

The full path is used for each meta name if more than one <code>ExtractPath</code>

1352

directive is used. That is, changes to the path used in <code>ExtractPath foo</code> do

1353

not affect the path used by <code>ExtractPath bar</code>.

1354

</li>

1355

<li><a name="item_extractpathdefault"></a><a name="extractpathdefault"></a>ExtractPathDefault *metaname* default_value

1356

This can be used with <code>ExtractPath</code> to set a default string to index under the

1357

given metaname if none of the <code>ExtractPath</code> patterns match.

1358

For example, say your want to index each document with a metaname

1359

"department" based on the following path examples:

1360

<pre class="pre-section"> /web/sales/foo...

1361

/web/parts/foo...

1362

/web/accounting/foo...</pre>

1363

But you are also indexing documents that do not follow that pattern and you want to search those

1364

separately, too.

1365

<pre class="pre-section"> ExtractPath department regex !^/web/([^/]+)/.*$!$1!

1366

ExtractPathDefault department other</pre>

1367

Now, you may search like this:

1368

<pre class="pre-section"> -w foo department=(sales) - limit searches to the sales documents

1369

-w foo department=(parts) - limit searches to the parts documents

1370

-w foo department=(accounting) - limit searches to the accounting documents

1371

-w foo department=(other) - everything but sales, parts, and accounting.</pre>

1372

This basically is a shortcut for:

1373

<pre class="pre-section"> -w foo not department=(sales or parts or accounting)</pre>

1374

but you don't need to keep track of what was extracted.

1375

</li>

1376

<li><a name="item_propertynames"></a><a name="propertynames"></a>PropertyNames *list of meta names*

1377

</li>

1378

<li><a name="item_propertynamescomparecase"></a><a name="propertynamescomparecase"></a>PropertyNamesCompareCase *list of meta names*

1379

</li>

1380

<li><a name="item_propertynamesignorecase"></a><a name="propertynamesignorecase"></a>PropertyNamesIgnoreCase *list of meta names*

1381

Swish-e allows you to specify certain META tags that can be used as document

1382

properties. The contents of any META tag that has been identified as a

1383

document property can be returned as part of the search results along with the

1384

rank, file name, title, and document size (see the <code>-p</code> and <code>-x</code> switches in

1385

<a href="swish-run.html">SWISH-RUN</a>).

1386

Properties are useful for returning additional data from documents in

1387

search results -- this saves the effort of reading and parsing the source

1388

files while reading Swish-e search results, and is especially useful

1389

when the source documents are no longer available or slow to access

1390

(e.g. over http).

1391

Another feature of properties is that Swish-e can use the PropertyNames for

1392

sorting the search results (see the <code>-s</code> switch).

1393

<pre class="pre-section"> PropertyNames author subjects</pre>

1394

Two variations are available. <code>PropertyNamesCompareCase</code> and

1395

<code>PropertyNamesIgnoreCase</code>. These tell Swish-e to either ignore or compare

1396

case when sorting results. The default for <code>PropertyNames</code> is to ignore the

1397

case.

1398

<pre class="pre-section"> PropertyNamesIgnoreCase subject

1399

PropertyNamesCompareCase keyword</pre>

1400

The defaults for "internal" properties are:

1401

<pre class="pre-section"> swishtitle -- ignore the case

1402

swishdocpath -- compare case

1403

swishdescription -- compare case</pre>

1404

These can be overridden with <code>PropertyNamesCompareCase</code> and

1405

<code>PropertyNamesIgnoreCase</code>.

1406

<pre class="pre-section"> PropertyNamesCompareCase swishtitle </pre>

1407

Use of PropertyNames will increase the size of your index files, sometimes

1408

significantly. Properties will be compressed if Swish-e is compiled with zlib

1409

as described in the <a href="install.html">INSTALL</a> manual page.

1410

If Swish-e finds more than one property of the same name in a document

1411

the property's contents will be concatinated for strings, and a warning

1412

issues for numeric (or date) properties.

1413

</li>

1414

<li><a name="item_propertynamesnostripchars"></a><a name="propertynamesnostripchars"></a>PropertyNamesNoStripChars

1415

PropertyNamesNoStripChars specifies that the listed properties should not

1416

have strings of low ASCII characters replaced with a space character.

1417

Properties will be stored as found in the document.

1418

When printing properties with the swish-e binary newlines are replaced with

1419

a space character. Use the swish-e library (or SWISH::API perl module) to

1420

fetch properties without newlines replaced.

1421

</li>

1422

<li><a name="item_propertynamesnumeric"></a><a name="propertynamesnumeric"></a>PropertyNamesNumeric

1423

This directive is similar to <code>PropertyNames</code>, but it flags the property as

1424

being a string of digits (integer value) that will be stored as binary data

1425

instead of a string. This allows sorting with <code>-s</code> and limiting with <code>-L</code> to

1426

sort and limit the property correctly.

1427

Swish-e uses <code>strtoul(3)</code> to convert the string into an unsigned long integer.

1428

Therefore, only positive integers can be stored.

1429

Future versions of Swish-e may be able to store different property types

1430

(such as negative integers and real numbers). This directive may change

1431

in future releases of Swish.

1432

</li>

1433

<li><a name="item_propertynamesdate"></a><a name="propertynamesdate"></a>PropertyNamesDate

1434

This directive is exactly like <code>PropertyNamesNumeric</code>, but it also flags the

1435

number as a machine timestamp (seconds since Epoch), and will print a formatted

1436

date when returning this property. See <code>-x</code> in <a href="swish-run.html">SWISH-RUN</a>.

1437

Swish-e will not parse dates when indexing; you must use a timestamp.

1438

</li>

1439

<li><a name="item_propertynamealias"></a><a name="propertynamealias"></a>PropertyNameAlias *property name* *list of aliases*

1440

This allows aliases for a property name. For example, if you are indexing

1441

HTML files, plus XML files that are written in English, German, and

1442

Spanish and thus use the tags "title", "titel", and "t�tulo" you can use:

1443

<pre class="pre-section"> PropertyNameAlias swishtitle title titel t�tulo titulo</pre>

1444

Note that "swishtitle" is the built-in property used to store the title of

1445

a document, and therefore you do not need to specify it as a PropertyName

1446

before use.

1447

</li>

1448

<li><a name="item_propertynamesmaxlength"></a><a name="propertynamesmaxlength"></a>PropertyNamesMaxLength integer *list of meta names*

1449

This option will set the max length of the text stored in a property.

1450

You must specify a number between 0 and the max integer size on your

1451

platform, and a list of properties. The properties specified must not

1452

be aliases.

1453

If any of the property names do not exist they will be created (e.g. you

1454

do not need to define the property with PropertyNames first).

1455

In general, this feature will only be useful when parsing HTML or XML

1456

with the libxml2 parser.

1457

For example:

1458

<pre class="pre-section"> PropertyNamesMaxLength 1000 swishdescription

1459

PropertyNameAlias swishdescription body</pre>

1460

Is somewhat like

1461

<pre class="pre-section"> StoreDescription HTML <body> 1000

1462

StoreDescription XML <body> 1000

1463

StoreDescription HTML2 <body> 1000

1464

StoreDescription XML2 <body> 1000</pre>

1465

but StoreDescription allows setting the tag for each parser type.

1466

<pre class="pre-section"> PropertyNamesMaxLength 1000 headings

1467

PropertyNameAlias headings h1 h2 h3 h4</pre>

1468

collects all the heading text into a single property called "headings", not

1469

to exceed 1000 characters.

1470

</li>

1471

<li><a name="item_propertynamessortkeylength"></a><a name="propertynamessortkeylength"></a>PropertyNamesSortKeyLength integer *list of meta names*

1472

Sets the length of the string used when sorting.

1473

The default is 100 characters. The -T metanames debugging option will

1474

list the current values for an index.

1475

This setting is used when sorting during indexing, and perhaps when sorting

1476

while searching. It also effects the order when limiting to a range of values

1477

with the -L option.

1478

</li>

1479

<li><a name="item_presortedindex"></a><a name="presortedindex"></a>PreSortedIndex *list of property names*

1480

By default Swish-e generates presorted tables while indexing for each

1481

property name. This allows faster sorting when generating results.

1482

On large document collections this presorting may add to the indexing

1483

time, and also adds to the total size of the index. This directive can

1484

be used to customize exactly which properties will be presorted.

1485

If <code>PreSortedIndex</code> it is not present in the config file (default action),

1486

all the properties will be presorted at indexing time. If it is present

1487

without any parameter, no properties will be presorted. Otherwise, only the

1488

property names specified will be presorted.

1489

For example, if you only wish to sort results by a property called <code>title</code>:

1490

<pre class="pre-section"> PropertyNames title age time

1491

PreSortedIndex title</pre>

1492

</li>

1493

<li><a name="item_storedescription"></a><a name="storedescription"></a>StoreDescription [XML <tag> size|HTML <meta> size|TXT size]

1494

StoreDescription allows you to store a document description in the index

1495

file. This description can be returned in your search results when the <code>-x</code>

1496

switch is used to include the swishdescription for extended results, or by

1497

using <code>-p swishdescription</code>.

1498

The document type (XML, HTML and TXT) must match the document type currently

1499

being indexed as set by <code>IndexContents</code> or <code>DefaultContents</code>. See those

1500

directives for possible values. A common problem is using <code>StoreDescription</code>

1501

yet not setting the document's type with <code>IndexContents</code> or

1502

<code>DefaultContents</code>. Another problem is different types:

1503

<pre class="pre-section"> IndexContents HTML2 .html

1504

StoreDescription HTML <body></pre>

1505

Then .html documents are assigned a type of HTML2 (and parsed by the libxml2 parser), but the

1506

description will not be stored since it is type HTML instead of HTML2.

1507

For text documents you specify the type TXT (or TXT2 or TXT*) and the number of characters to capture.

1508

<pre class="pre-section"> StoreDescription TXT 20</pre>

1509

The above stores only the first twenty characters from the text file in the Swish-e index

1510

file.

1511

For HTML, and XML file types, specify the tag to use for the

1512

description, and optionally the number of characters to capture. If not

1513

specified will capture the entire contents of the tag.

1514

<pre class="pre-section"> StoreDescription HTML <body> 20000

1515

StoreDescription XML <desc> 40</pre>

1516

Again, note that documents must be assigned a document type with

1517

<code>IndexContents</code> or <code>DefaultContents</code> to use this feature.

1518

Swish-e will compress the descriptions (or any other large property) if

1519

compiled to use zlib (see <a href="install.html">INSTALL</a>). This is recommended when using

1520

StoreDescription and a large number of documents. Compression of 30% to 50% is

1521

not uncommon with HTML files.

1522

</li>

1523

<li><a name="item_propcompressionlevel"></a><a name="propcompressionlevel"></a>PropCompressionLevel [0-9]

1524

This directive sets the compression level used when storing properties

1525

to disk. A setting of zero is no compression, and a setting of nine is

1526

the most compression.

1527

The default depends on the default setting compiled with zlib, but is

1528

typically six.

1529

This option is useful when using <code>StoreDescription</code> to store a large amount

1530

text in properties (or if using <code>PropertyNames</code> with large property sizes).

1531

Properties must be over a value defined in config.h (100 is the

1532

default) before compression will be attempted. Swish-e will never store

1533

the results of the compression if the compressed data is larger than

1534

the original data.

1535

This option is only available when Swish-e is compiled with zlib support.

1536

</li>

1537

<li><a name="item_truncatedocsize"></a><a name="truncatedocsize"></a>TruncateDocSize *number of characters*

1538

TruncateDocSize limits the size of a document while indexing documents

1539

and/or using filters. This config directive truncates the numbers of

1540

read bytes of a document to the specified size. This means: if a document

1541

is larger, read only the specified numbers of bytes of the document.

1542

Example:

1543

<pre class="pre-section"> TruncateDocSize 10000000</pre>

1544

The default is zero, which means read all data.

1545

Warning: If you use TruncateDocSize, use it with care! TruncateDocSize

1546

is a safety belt only, to limit e.g. filteroutput, when accessing

1547

databases, or to limit "runnaway" filters. Truncating doc input may

1548

destroy document structures for Swish-e (e.g. swish may miss closing

1549

tags for XML or HTML documents).

1550

TruncateDocSize does not currently work with the <code>prog</code> input source method.

1551

</li>

1552

<li><a name="item_fuzzyindexingmode"></a><a name="fuzzyindexingmode"></a>FuzzyIndexingMode NONE|Stemming|Soundex|Metaphone|DoubleMetaphone

1553

Selects the type of index to create. Only one type of index may be created.

1554

It's a good idea to create both a normal index and a fuzzy index and

1555

allow your search interface select which index to use. Many people find the

1556

fuzzy searches to be too fuzzy.

1557

The available fuzzy indexing options can be displayed by running

1558

<pre class="pre-section"> swish-e -T LIST_FUZZY_MODES</pre>

1559

Available options include:

1560

<ul>

1561

<li><a name="item_none"></a><a name="none"></a>None

1562

Words are stored in the index without any conversion. This is the default.

1563

</li>

1564

<li><a name="item_stemming__"></a><a name="stemming__"></a>Stemming_*

1565

This options uses one of the installed Snowball stemmers (<a href="http://snowball.tartarus.org/">http://snowball.tartarus.org/</a>).

1566

The installed stemmers can be viewed by running

1567

<pre class="pre-section"> swish-e -T LIST_FUZZY_MODES</pre>

1568

For example, to use the Spanish stemming module:

1569

<pre class="pre-section"> FuzzyIndexingMode Stemming_es</pre>

1570

</li>

1571

<li><a name="item_stem"></a><a name="stem"></a>Stem or Stemming_en

1572

**This option is no longer supported.**

1573

Selects the legacy Swish-e English stemmer.

1574

This is deprecated in favor of the Snowball English stemmer Stemming_en1.

1575

Words are converted using the Porter stemming algorithm.

1576

From: <a href="http://www.tartarus.org/~martin/PorterStemmer/">http://www.tartarus.org/~martin/PorterStemmer/</a>

1577

<pre class="pre-section"> The Porter stemming algorithm (or Porter stemmer) is a

1578

process for removing the commoner morphological and inflexional

1579

endings from words in English. Its main use is as part of a

1580

term normalisation process that is usually done when setting up

1581

Information Retrieval systems.</pre>

1582

This will help a search for "running" to also find "run" and "runs", for example.

1583

The stemming function does not convert words to their root, rather

1584

programmatically removes endings on words in an attempt to make similar

1585

words with different endings stem to the same string of characters.

1586

It's not a perfect system, and searches on stemmed indexes often return

1587

curious results. For example, two entirely different words may stem to

1588

the same word.

1589

Stemming also can be confusing when used with a wildcard (truncation).

1590

For example, you might expect to find the word "running" by searching for

1591

"runn*". But this fails when using a stemmed index, as "running" stems to

1592

"run", yet searching for "runn*" looks for words that start with "runn".

1593

</li>

1594

<li><a name="item_soundex"></a><a name="soundex"></a>Soundex

1595

Soundex was developed in the 1880s so records for people with similar

1596

sounding names could be found more readily. Soundex is a coded surname

1597

based on the way a surname sounds rather than spelling. Surnames that

1598

sound similar, like Smith and Smyth, are filed together under the same

1599

Soundex code. This is mostly useful for US English.

1600

Soundex should not be used to search for sound-alike words. Metaphone

1601

would be more appropriate for generic sound matching of words. Soundex

1602

should only be used where you need to search multiple documents for

1603

proper names which sound similar. This is primarily used for indexing

1604

genealogical records. This may be useful for indexing other collections

1605

of data consisting mostly of names. Many common name variations are

1606

matched by Soundex. The only notable exception is the first letter of

1607

the name. The first letter is not matched for sound.

1608

</li>

1609

<li><a name="item_metaphone"></a><a name="metaphone"></a>Metaphone and DoubleMetaphone

1610

Words are transformed into a short series of letters representing the sound of the word (in English).

1611

Metaphone algorithms are often used for looking up mis-spelled words in dictionary programs.

1612

From: <a href="http://aspell.sourceforge.net/metaphone/">http://aspell.sourceforge.net/metaphone/</a>

1613

<pre class="pre-section"> Lawrence Philips' Metaphone Algorithm is an algorithm which returns

1614

the rough approximation of how an English word sounds.</pre>

1615

The <code>DoubleMetaphone</code> mode will sometimes generate two different metaphones

1616

for the same word. This is supposed to be useful when a word may be pronounced

1617

more than one way.

1618

A metaphone index should give results somewhere in between Soundex and Stemming.

1619

</li>

1620

</ul>

1621

</li>

1622

<li><a name="item_usestemming"></a><a name="usestemming"></a>UseStemming [yes|NO]

1623

Put yes to apply word stemming algorithm during indexing, else no.

1624

<pre class="pre-section"> UseStemming no

1625

UseStemming yes</pre>

1626

When UseStemming is set to <code>yes</code> every word is stemmed before placing it in to

1627

the index.

1628

This option is deprecated. It has been superceded by <code>FuzzyIndexingMode</code>.

1629

</li>

1630

<li><a name="item_usesoundex"></a><a name="usesoundex"></a>UseSoundex [yes|NO]

1631

When UseSoundex is set to <code>yes</code> every word is converted to a Soundex code

1632

before placing it in to the index.

1633

This option is deprecated. It has been superceded by <code>FuzzyIndexingMode</code>.

1634

</li>

1635

<li><a name="item_ignoretotalwordcountwhenranking"></a><a name="ignoretotalwordcountwhenranking"></a>IgnoreTotalWordCountWhenRanking [YES|no]

1636

Put yes to ignore the total number of words in the file when calculating

1637

ranking. Often better with merges and small files. Default is yes.

1638

<pre class="pre-section"> IgnoreTotalWordCountWhenRanking no</pre>

1639

The default was changed from no to yes in version 2.2.

1640

NOTE: must be set to no if you intend to use the -R 1 option when

1641

searching.

1642

</li>

1643

<li><a name="item_minwordlimit"></a><a name="minwordlimit"></a>MinWordLimit *integer*

1644

Set the minimum length of an word. Shorter words will not be indexed.

1645

The default is 1 (as defined in src/config.h).

1646

<pre class="pre-section"> MinWordLimit 5</pre>

1647

</li>

1648

<li><a name="item_maxwordlimit"></a><a name="maxwordlimit"></a>MaxWordLimit *integer*

1649

Set the maximum length of an indexable word. Every longer word will not

1650

be indexed. The Default is 40 (as defined in src/config.h).

1651

</li>

1652

<li><a name="item_wordcharacters"></a><a name="wordcharacters"></a>WordCharacters *string of characters*

1653

</li>

1654

<li><a name="item_ignorefirstchar"></a><a name="ignorefirstchar"></a>IgnoreFirstChar *string of characters*

1655

</li>

1656

<li><a name="item_ignorelastchar"></a><a name="ignorelastchar"></a>IgnoreLastChar *string of characters*

1657

</li>

1658

<li><a name="item_begincharacters"></a><a name="begincharacters"></a>BeginCharacters *string of characters*

1659

</li>

1660

<li><a name="item_endcharacters"></a><a name="endcharacters"></a>EndCharacters *string of characters*

1661

These settings define what a word consists of to the Swish-e indexing engine.

1662

Compiled in defaults are in src/config.h.

1663

When indexing Swish-e uses WordCharacters to split up the document

1664

into words. Words are defined by any string of non-blank characters

1665

that contain only the characters listed in WordCharacters. If a string

1666

of characters includes a character that is not in WordCharacters then

1667

the word will be spit into two or more separate words.

1668

For example:

1669

<pre class="pre-section"> WordCharacters abde</pre>

1670

Would turn "abcde" into two words "ab" and "de".

1671

Next, of these words, any characters defined in IgnoreFirstChar are

1672

stripped off the start of the word, and IgnoreLastChar characters

1673

are stripped off the end of the word. This allows, for example,

1674

periods within a word (www.slashdot.com), but not at the end of

1675

a word. Characters in IgnoreFirstChar and IgnoreLastChar must be in

1676

WordCharacters.

1677

Finally, the resulting words MUST begin with one of the characters

1678

listed in BeginCharacters and end with one of the characters listed in

1679

EndCharacters. BeginCharacters and EndCharacters must be a subset of

1680

the characters in WordCharacters. Often, WordCharacters, BeginCharacters

1681

and EndCharacters will all be the same.

1682

Note that the same process applies to the query while searching.

1683

Getting these settings correct will take careful consideration and practice.

1684

It's helpful to create an index of a single test file, and then look at the

1685

words that are placed in the index (see the <code>-v 4</code>, <code>-D</code> and <code>-k</code> searching

1686

switches).

1687

Currently there is only support for eight-bit characters.

1688

Example:

1689

<pre class="pre-section"> WordCharacters .abcdefghijklmnopqrstuvwxyz

1690

BeginCharacters abcdefghijklmnopqrstuvwxyz

1691

EndCharacters abcdefghijklmnopqrstuvwxyz

1692

IgnoreFirstChar .

1693

IgnoreLastChar .</pre>

1694

So the string

1695

<pre class="pre-section"> Please visit http://www.example.com/path/to/file.html.</pre>

1696

will be indexed as the following words:

1697

<pre class="pre-section"> please

1698

visit

1699

http

1700

www.example.com

1701

path

1702

to

1703

file.html</pre>

1704

Which means that you can search for <code>www.example.com</code> as a single word, but

1705

searching for just <code>example</code> will not find the document.

1706

Note: when indexing HTML documents HTML entities are converted to their

1707

character equivalents before being processed with these directives. This is a

1708

change from previous versions of Swish-e where you were required to include the

1709

characters <code>0123456789&#;</code> to index entities. See also <code>ConvertHTMLEntities</code>

1710

</li>

1711

<li><a name="item_buzzwords"></a><a name="buzzwords"></a>Buzzwords [*list of buzzwords*|File: path]

1712

The Buzzwords option allows you to specify words that will be indexed

1713

regardless of WordCharacters, BeginCharacters, EndCharacters, stemming,

1714

soundex and many of the other checks done on words while indexing.

1715

Buzzwords are case insensitive.

1716

Buzzwords should be separated by spaces and may span multiple directives. If

1717

the special format <code><a href="File:filename">File:filename</a></code> is used then the Buzzwords will be read

1718

from an external file during indexing.

1719

Examples:

1720

<pre class="pre-section"> Buzzwords C++ TCP/IP

1721

1722

Buzzwords File: ./buzzwords.lst</pre>

1723

If a Buzzword contains search operator characters they must be backslashed

1724

when searching. For example:

1725

<pre class="pre-section"> Buzzwords C++ TCP/IP web=http

1726

1727

./swish-e -w 'web\=http'</pre>

1728

Buzzwords are found by splitting the text on whitespace, removing

1729

<code>IgnoreFirstChar</code> and <code>IgnoreLastChar</code> characters from the word, and then

1730

comparing with the list of <code>Buzzwords</code>. Therefore, if adding <code>Buzzwords</code> to

1731

an index you will probably want to define <code>IgnoreFirstChar</code> and

1732

<code>IgnoreLastChar</code> settings.

1733

Note: Buzzwords specific settings for <code>IgnoreFirstChar</code> and <code>IgnoreLastChar</code>

1734

may be used in the future.

1735

</li>

1736

<li><a name="item_compresspositions"></a><a name="compresspositions"></a>CompressPositions [yes|NO]

1737

This option enables zlib compression for individual word data in the index file.

1738

The default is NO, that is the index word data is not compressed by default.

1739

Enabling this option can reduced the size of the index file, but at the expense of

1740

slower wildcard search times.

1741

The default changed from YES to NO starting with version 2.4.3.

1742

</li>

1743

<li><a name="item_ignorewords"></a><a name="ignorewords"></a>IgnoreWords [*list of stop words*|File: path]

1744

The IgnoreWords option allows you to specify words to ignore, called

1745

stopwords. The default is to not use any stopwords.

1746

Words should be separated by spaces and may span multiple directives. If the

1747

special format <code><a href="File:filename">File:filename</a></code> is used then the stop words will be read from

1748

an external file during indexing.

1749

In previous versions of Swish-e you could use the directive

1750

<pre class="pre-section"> IgnoreWords swishdefault - obsolete!</pre>

1751

to include a default list of compiled in stopwords. This keyword is no

1752

longer supported.

1753

Examples:

1754

<pre class="pre-section"> IgnoreWords www http a an the of and or

1755

1756

IgnoreWords File: ./stopwords.de</pre>

1757

</li>

1758

<li><a name="item_usewords"></a><a name="usewords"></a>UseWords [*list of words*|File: path]

1759

UseWords defines the words that Swish-e will index. Only the words

1760

listed will be indexed.

1761

You can specify a list of words following the directive (you may specify more

1762

than one <code>UseWords</code> directive in a config file), and/or use the <code>File:</code> form

1763

to specify a path to a file containing the words:

1764

<pre class="pre-section"> UseWords perl python pascal fortran basic cobal php

1765

UseWords File: /path/to/my/wordlist</pre>

1766

Please drop the Swish-e list a note if you actually use this feature.

1767

It may be removed from future versions.

1768

</li>

1769

<li><a name="item_ignorelimit"></a><a name="ignorelimit"></a>IgnoreLimit *integer integer*

1770

This automatically omits words that appear too often in the files (these

1771

words are called stopwords). Specify a whole percentage and a number,

1772

such as "80 256". This omits words that occur in over 80% of the files

1773

and appear in over 256 files. Comment out to turn off auto-stopwording.

1774

<pre class="pre-section"> IgnoreLimit 50 1000</pre>

1775

Swish-e must do extra processing to adjust the entire index when this

1776

feature is used. It is recommended that instead of using this feature

1777

that you decided what words are stopwords and add them to IngoreWords

1778

in your configuration file. To do this, use IgnoreLimit one time and

1779

note the stop words that are found while indexing. Add this list to

1780

IgnoreWords, and then remove IgnoreLimit from the configuration file.

1781

</li>

1782

<li><a name="item_ignoremetatags"></a><a name="ignoremetatags"></a>IgnoreMetaTags *list of names*

1783

<code>IgnoreMetaTags</code> defines a list of metatags to ignore while indexing XML files

1784

(and HTML files if using libxml2 for parsing HTML). All text within the tags

1785

will be ignored -- both for indexing (<code>MetaNames</code>) and properties

1786

(<code>PropertyNames</code>). To still parse properties, yet do not index the text, see

1787

<code>UndefinedMetaTags</code>.

1788

This option is useful to avoid indexing specific data from a file.

1789

For example:

1790

1791

<first_name>

1792

William

1793

</first_name> <last_name>

1794

Shakespeare

1795

</last_name> <updated_date>

1796

April 25, 1999

1797

</updated_date>

1798

</person></pre>

1799

In the above example you might not want to index the updated date,

1800

and therefore prevent finding this record by searching

1801

<pre class="pre-section"> -w 'person=(April)'</pre>

1802

This is solved by:

1803

<pre class="pre-section"> IgnoreMetaTags updated_date</pre>

1804

See also <code>UndefinedMetaTags</code>.

1805

</li>

1806

<li><a name="item_ignorenumberchars"></a><a name="ignorenumberchars"></a>IgnoreNumberChars *list of characters*

1807

Experimental Feature

1808

This experimental feature can be used to define a set of characters that

1809

describe a number. If a word is found to contain only those characters it will

1810

not be indexed. The characters listed must be part of <code>WordCharacters</code>

1811

settings. In other words, the "word" checked is a word that Swish-e would

1812

otherwise index.

1813

For example,

1814

<pre class="pre-section"> IgnoreNumberChars 0123456789$.,</pre>

1815

Then Swish-e would not index the following:

1816

<pre class="pre-section"> 123

1817

123,456.78

1818

$123.45</pre>

1819

You might be tempted to avoid indexing hex numbers with:

1820

<pre class="pre-section"> IgnoreNumberChars 0123456789abcdef</pre>

1821

which will not index 0D31, but will also not index the word "bad".

1822

This is an experimental feature that may change in future versions.

1823

One possible change is to use regular expressions instead.

1824

</li>

1825

<li><a name="item_indexcomments"></a><a name="indexcomments"></a>IndexComments [NO|yes]

1826

This option allows the user decide if to index the contents of HTML

1827

comments. Default is no. Set to yes if comment indexing is required.

1828

<pre class="pre-section"> IndexComments yes</pre>

1829

Note: This is a change in the default behavior prior to version 2.2.

1830

</li>

1831

<li><a name="item_translatecharacters"></a><a name="translatecharacters"></a>TranslateCharacters [*string1 string2*|:ascii7:]

1832

The TranslateCharacters directive maps the characters in string1 to the

1833

characters listed in string2.

1834

For example:

1835

<pre class="pre-section"> # This will index a_b as a-b and �mo as amo

1836

TranslateCharacters _� -a</pre>

1837

<code>TranslateCharacters :ascii7:</code> is a predefined set of characters that will

1838

translate eight bit characters to ascii7 characters. Using the :ascii7: rule

1839

will translate "��" to "aac". This means: searching "�elik", "�elik" or

1840

"celik" will all match the same word.

1841

TranslateCharacters is done early in the indexing process, after

1842

converting HTML entities but before splitting the input text into words

1843

based on WordCharacters. So characters you are translating from

1844

do not need to be listed in word characters.

1845

The same character translations take place when searching.

1846

</li>

1847

<li><a name="item_bumppositioncountercharacters"></a><a name="bumppositioncountercharacters"></a>BumpPositionCounterCharacters *string*

1848

When indexing Swish-e assigns a word position to each word. This enables

1849

phrase searching. There may be cases where you would like to prevent

1850

phrase matching. The BumpPositionCounterCharacters directive allows

1851

you to specify a set of characters that when found in the text will

1852

increment the word position -- effectively preventing phrase matches

1853

across that character.

1854

For example, if you have a tag:

1855

1856

computer programming | apple computers

1857

</subjects></pre>

1858

You might want to prevent matching "programming apple" in that meta name.

1859

<pre class="pre-section"> BumpPositionCounterCharacters |</pre>

1860

There is no default, and you may list a string of characters.

1861

</li>

1862

<li><a name="item_dontbumppositiononendtags"></a><a name="dontbumppositiononendtags"></a>DontBumpPositionOnEndTags *list of names*

1863

</li>

1864

<li><a name="item_dontbumppositiononstarttags"></a><a name="dontbumppositiononstarttags"></a>DontBumpPositionOnStartTags *list of names*

1865

Since metatags are typically separate data fields, the word position counter is

1866

automatically bumped between metatags (actually, bumped when a start tag is

1867

found and when an end tag is found). This prevents matching a phrase that

1868

spans more than one metaname. <code>DontBumpPositionOnEndTags</code> and

1869

<code>DontBumpPositionOnStartTags</code> disables this feature for the listed metanames.

1870

For example,

1871

1872

<first_name>

1873

William

1874

</first_name>

1875

<last_name>

1876

Shakespeare

1877

</last_name>

1878

<updated_date>

1879

April 25, 1999

1880

</updated_date>

1881

</person></pre>

1882

In the configuration file:

1883

<pre class="pre-section"> DontBumpPositionOnEndTags first_name

1884

DontBumpPositionOnStartTags last_name</pre>

1885

This configuration allows this phrase search

1886

<pre class="pre-section"> -w 'person=("william shakespeare")'</pre>

1887

but this phrase search will fail

1888

<pre class="pre-section"> -w 'person=("shakespeare april")'</pre>

1889

</li>

1890

</ul>

1891

1892

</div>

1893

1894

1895

1896

<h2><a name="directives_for_the_file_access_method_only"></a>Directives for the File Access method only</h2>

1897

1898

Some directives have different uses depending on the source of the

1899

documents. These directives are only valid when using the File system

1900

method of indexing.

1901

<ul>

1902

<li><a name="item_indexonly"></a><a name="indexonly"></a>IndexOnly *list of file suffixes*

1903

This directive specifies the allowable file suffixes (extensions) while

1904

indexing. The default is to index all files specified in IndexDir.

1905

<pre class="pre-section"> # Only index .html .htm and .q files

1906

IndexOnly .html .htm .q</pre>

1907

<code>IndexOnly</code> checks that the file end in the characters listed. It does not

1908

check "extensions". <code>IndexOnly</code> is tested right before <code>FileRules</code> is

1909

processed.

1910

</li>

1911

<li><a name="item_followsymlinks"></a><a name="followsymlinks"></a>FollowSymLinks [yes|NO]

1912

Put "yes" to follow symbolic links in indexing, else "no". Default is no.

1913

<pre class="pre-section"> FollowSymLinks no

1914

FollowSymLinks yes</pre>

1915

Note that when set to <code>no</code> extra stat(2) system calls must be made for each

1916

file. For large number of files you may see a small reduction in indexing time

1917

by setting this to <code>yes</code>.

1918

See also the <code>-l</code> switch in <a href="swish-run.html">SWISH-RUN</a>.

1919

</li>

1920

<li><a name="item_filerules"></a><a name="filerules"></a>FileRules [type] [contains|is|regex] *regular expression*

1921

</li>

1922

<li><a name="item_filematch"></a><a name="filematch"></a>FileMatch [type] [contains|is|regex] *regular expression*

1923

FileRules and FileMatch are used to, respectively, exclude and include files

1924

and directories to index. Since, by default, Swish-e indexes all files and

1925

recurses all directories (but see also <code>FollowSymLinks</code>) you will typically

1926

only use <code>FileRules</code> to exclude files or directories. <code>FileMatch</code> is useful

1927

in a few cases, for example, to override the behavior of <code>IndexOnly</code>. Some

1928

examples are included below.

1929

Except for <code>FileRules title ...</code>, this feature is only available for file

1930

access method (-S fs), which is the default indexing mode. Also, any pathname

1931

modification with <code>ReplaceRules</code> happens after the check for <code>FileRules</code>.

1932

(It's unlikely that you would exclude files with <code>FileRules</code> based on text you

1933

added with <code>ReplaceRules</code>!)

1934

The regular expression is a C regex.h extended regular expression.

1935

You may supply more than one regular expression per line, or use

1936

separate directives. Preceding the regular expression with the word

1937

"not" negates the match.

1938

The regular expression is compared against [type] as described below.

1939

For historical reasons, you can specify <code>contains</code> or <code>is</code>. <code>is</code> simply

1940

forces the regular expression to match at the start and end of the string (by

1941

internally prepending "^" and appending "$" to the regular expression).

1942

The <code>regex</code> option requires delimiter characters:

1943

<pre class="pre-section"> FileRules title regex /^private/i</pre>

1944

The only advantage of <code>regex</code> is if you want to do case insensitive matches,

1945

or simply like your regular expressions to look like perl regular expressions.

1946

You must use matching delimiters; (), {}, and [], are not currently supported

1947

for no good reason other than laziness.

1948

Use quotes (" or ') around a pattern if it contains any white space.

1949

Note that the backslash character becomes the escape character within

1950

quotes.

1951

For example, these sets generate the same regular expressions.

1952

<pre class="pre-section"> FileRules title is hello

1953

FileRules title contains ^hello$

1954

FileRules title regex /^hello$/</pre>

1955

These all need quotes due to the included space character

1956

<pre class="pre-section"> FileRules title is "hello there"

1957

FileRules title contains "^hello there$"

1958

FileRules title regex "!^hello there$!"</pre>

1959

These show how the backslash must be doubled inside of quotes.

1960

Swish-e converts a double-backslash into a single backslash, and then

1961

passes that single onto the regular expression compiler.

1962

<pre class="pre-section"> FileRules filename regex /\.pdf/

1963

FileRules filename regex "/\\.pdf/"

1964

1965

FileRules filename regex !hello\\there! # need double for real backslash

1966

FileRules filename regex "!hello\\\\there!" # need double-double inside of quotes</pre>

1967

Matching Types

1968

The following types of match strings my be supplied:

1969

<pre class="pre-section"> FileRules pathname

1970

FileRules dirname

1971

FileRules filename

1972

FileRules directory

1973

FileRules title

1974

1975

FileMatch pathname

1976

FileMatch filename

1977

FileMatch dirname

1978

FileMatch directory</pre>

1979

pathname matches the regular expression against the current pathname. The

1980

pathname may or may not be absolute depending on what you supplied to

1981

<code>IndexDir</code>.

1982

Example:

1983

<pre class="pre-section"> # Don't index paths that contain private or hidden

1984

FileRules pathname contains (private|hidden)

1985

1986

# Same thing

1987

FileRules pathname regex /(private|hidden)/

1988

1989

# Don't index exe files

1990

FileRules pathname contains \.exe$</pre>

1991

dirname and filename split the path name by the last delimiter

1992

character into a directory name, and a file name. Then these are compared

1993

against the patterns supplied. Directory names do not have a trailing

1994

slash. All path names use the forward slash as a delimiter within Swish-e.

1995

Example:

1996

<pre class="pre-section"> # Same as last example - don't index *.exe files.

1997

FileRules filename contains \.exe$

1998

1999

# Don't index any file called test.html files

2000

FileRules filename contains ^test\.html$

2001

2002

# Same thing

2003

FileRules filename is test\.html

2004

2005

# Don't index any directories that contain "old" (/usr/local/myold/docs)

2006

FileRules dirname contains old

2007

2008

# Don't index any directories that contain the path segment "old" (/usr/local/old/foo)

2009

FileRules dirname contains /old/

2010

2011

# Index only .htm, .html, plus any all-digit file names

2012

IndexOnly .htm .html

2013

FileMatch filename contains ^\d+$

2014

2015

# Same as previous, but maybe a little slower

2016

FileRules filename regex not !\.(htm|html)$!

2017

FileMatch filename contains ^\d+$</pre>

2018

Swish-e checks these settings in the order of <code>pathname</code>, <code>dirname</code>, and

2019

<code>filename</code>, and <code>FileMatch</code> patterns are checked before <code>FileRules</code>, in

2020

general. This allows you to exclude most files with <code>FileRules</code>, yet allow in

2021

a few special cases with <code>FileMatch</code>. For example:

2022

<pre class="pre-section"> # Exclude all files of .exe, .bin, and .bat

2023

FileRules filename contains \.(exe|bin|bat)$

2024

# But, let these two in

2025

FileMatch filename is baseball\.bat incoming_mail\.bin

2026

2027

# Same, but as a single pattern

2028

FileMatch filename is (baseball\.bat|incoming_mail\.bin)</pre>

2029

The <code>directory</code> type is somewhat unique. When Swish-e recurses into a

2030

directory it will compare all the files in the directory with the pattern

2031

and then decide if that entire directory should or should not be indexed (or

2032

recursed). Note that you are matching against file names in a directory -- and

2033

some of those names may be directory names.

2034

A <code>FileRules directory</code> match will cause Swish-e to ignore all files and

2035

sub-directories in the current directory.

2036

Warning: A match with <code>FileMatch directory</code> says to index everything in the

2037

*current* directory and ignore any FileRules for this directory.

2038

Example:

2039

<pre class="pre-section"> # Don't index any directories (and sub directories) that contain

2040

# a file (or sub-directory) called "index.skip"

2041

FileRules directory contains ^index\.skip$

2042

2043

# Don't index directories that contain a .htaccess file.

2044

FileRules directory contains ^\.htaccess</pre>

2045

Note: While processing directories, Swish-e will ignore any files or

2046

directories that begin with a dot ("."). You may index files or directories

2047

that begin with a dot by specifying their name with <code>IndexDir</code> or <code>-i</code>.

2048

<code>title</code> checks for a pattern match in an HTML title.

2049

Example:

2050

<pre class="pre-section"> FileRules title contains construction example pointers

2051

2052

# This example says to ignore case

2053

FileRules title regex "/^Internal document/i"</pre>

2054

Note: <code>FileRules title</code> works for any input method (fs, prog, or http) that is

2055

parsed as HTML, and where a title was found in the document.

2056

In case all this seems a bit confusing, processing a directory happens

2057

in the following order.

2058

First the directory name is checked:

2059

<pre class="pre-section"> FileRules dirname - reject entire directory if matches</pre>

2060

Next the directory is scanned and each file name (which might be the

2061

name of a sub-directory) is checked:

2062

<pre class="pre-section"> FileRules directory - reject entire dir if *any* files match

2063

FileMatch directory - accept entire dir if *any* files match</pre>

2064

Then, unless <code>FileMatch directory</code> matched, each file is tested with

2065

FileMatch. A match says to index the file without further testing (i.e.

2066

overrides FileRules and IndexOnly):

2067

<pre class="pre-section"> FileMatch pathname \

2068

FileMatch dirname - file is accepted if any match

2069

FileMatch filename /</pre>

2070

otherwise

2071

<pre class="pre-section"> IndexOnly - file is checked for the correct file extension

2072

2073

FileRules pathname \

2074

FileRules dirname - file is rejected if any match

2075

FileRules filename /</pre>

2076

finally, the file is indexed.

2077

Files (not directories) listed with <code>IndexDir</code> or <code>-i</code> are processed in a

2078

similar way:

2079

<pre class="pre-section"> FileMatch pathname \

2080

FileMatch dirname - file is accepted if any match

2081

FileMatch filename /</pre>

2082

otherwise, the file is rejected if it doesn't have the correct extension

2083

or a FileRules matches.

2084

<pre class="pre-section"> IndexOnly - file is checked for the correct file extension

2085

2086

FileRules pathname \

2087

FileRules dirname - file is rejected if any match

2088

FileRules filename /</pre>

2089

Note: If things are not indexing as you expect, create a directory with some

2090

test files and use the <code>-T regex</code> trace option to see how file names are

2091

checked. Start with very simple tests!

2092

</li>

2093

</ul>

2094

2095

</div>

2096

2097

2098

2099

<h2><a name="directives_for_the_http_access_method_only"></a>Directives for the HTTP Access Method Only</h2>

2100

2101

The HTTP Access method is enabled by the "-S http" switch when indexing. It works by

2102

running a Perl program called SwishSpider which fetches documents from a web server.

2103

Only text files (content-type of "text/*") are indexed with the HTTP Access Method.

2104

Other document types (e.g. PDF or MSWord) may be indexed as well. The SwishSpider will

2105

attempt to make use of the SWISH::Filter module (included with the Swish-e distribution) to

2106

convert documents into a format that Swish-e can index.

2107

Note: The -S prog method of spidering (using spider.pl) can be a replacement for the -S http method.

2108

It offers more configuration options and better spidering speed.

2109

These directives below are available when using the HTTP Access Method of indexing.

2110

<ul>

2111

<li><a name="item_maxdepth"></a><a name="maxdepth"></a>MaxDepth *integer*

2112

MaxDepth defines how many links the spider should follow before stopping.

2113

A value of 0 configures the spider to traverse all links. The default

2114

is MaxDepth 0.

2115

<pre class="pre-section"> MaxDepth 5</pre>

2116

Note: The default was changed from 5 to 0 in release 2.4.0

2117

</li>

2118

<li><a name="item_delay"></a><a name="delay"></a>Delay *seconds*

2119

The number of seconds to wait between issuing requests to a server.

2120

This setting allows for more friendly spidering of remote sites.

2121

The default is 5 seconds.

2122

<pre class="pre-section"> Delay 1</pre>

2123

Note: The default was changed from 60 to 5 seconds in release 2.4.0

2124

</li>

2125

<li><a name="item_tmpdir"></a><a name="tmpdir"></a>TmpDir *path*

2126

The location of a writable temp directory on your system. The HTTP access

2127

method tells the Perl helper to place its files in this location, and the <code>-e</code>

2128

switch causes Swish-e to use this directory while indexing. There is no

2129

default.

2130

<pre class="pre-section"> TmpDir /tmp/swish</pre>

2131

If this directory does not exist or is not writable Swish-e will fail

2132

with an error during indexing.

2133

Note, the environment variables of <code>TMPDIR</code>, <code>TMP</code>, and <code>TEMP</code> (in that

2134

order) will override this setting.

2135

</li>

2136

<li><a name="item_spiderdirectory"></a><a name="spiderdirectory"></a>SpiderDirectory *path*

2137

The location of the Perl helper script called swishspider. If you

2138

use a relative directory, it is relative to your directory when you run

2139

Swish-e, not to the directory that Swish-e is in.

2140

The default is the location swishspider was installed.

2141

Normally this does not need to be set.

2142

<pre class="pre-section"> SpiderDirectory /usr/local/swish</pre>

2143

</li>

2144

<li><a name="item_equivalentserver"></a><a name="equivalentserver"></a>EquivalentServer *server alias*

2145

Often times the same site may be referred to by different names.

2146

A common example is that often <a href="http://www.some-server.com">http://www.some-server.com</a> and

2147

<a href="http://some-server.com">http://some-server.com</a> are the same. Each line should have a list of

2148

all the method/names that should be considered equivalent. Multiple

2149

EquivalentServer directives may be used. Each directive defines its

2150

own set of equivalent servers.

2151

<pre class="pre-section"> EquivalentServer http://library.berkeley.edu http://www.lib.berkeley.edu

2152

EquivalentServer http://sunsite.berkeley.edu:2000 http://sunsite.berkeley.edu</pre>

2153

</li>

2154

</ul>

2155

2156

</div>

2157

2158

2159

2160

<h2><a name="directives_for_the_prog_access_method_only"></a>Directives for the prog Access Method Only</h2>

2161

2162

This section details the directives that are only available for the

2163

"prog" document source feature of Swish-e. The "prog" access method runs

2164

an external program that "feeds" documents to Swish-e. This allows indexing

2165

and filtering of documents from any source.

2166

See <a href="swish-run.html#item_prog">prog - general purpose access method</a> in the

2167

SWISH-RUN man page for more information.

2168

A number of example programs for use with the "prog" access method are

2169

provided in the prog-bin directory. Please see those example if you

2170

have questions about implementing a "prog" input program.

2171

<ul>

2172

<li><a name="item_swishprogparameters"></a><a name="swishprogparameters"></a>SwishProgParameters *list of parameters*

2173

This is a list of parameters that will be sent to the external program

2174

when running with the "prog" document source method.

2175

<pre class="pre-section"> SwishProgParameters /path/to/config hello there

2176

IndexDir /path/to/program.pl</pre>

2177

Then running:

2178

<pre class="pre-section"> swish-e -c config -S prog</pre>

2179

Swish-e will execute <code>/path/to/program.pl</code> and pass <code>/path/to/config hello

2180

there</code> as three command line arguments to the program. This directive makes it

2181

easy to pass settings from the Swish-e configuration file to the external

2182

program.

2183

For example, the <code>spider.pl</code> program (included in the <code>prog-bin</code> directory)

2184

uses the <code>SwishProgParameters</code> to specify what file to read for configuration

2185

information.

2186

<pre class="pre-section"> SwishProgParameters spider.config

2187

IndexDir ./spider.pl</pre>

2188

The <code>spider.pl</code> program also has a default action so you can avoid using a

2189

configuration file:

2190

<pre class="pre-section"> SwishProgParameters default http://www.swishe.org/ http://some.other.site/

2191

IndexDir ./spider.pl</pre>

2192

And the spider program will use default settings for spidering those sites.

2193

Swish-e can read documents from standard input, so another way to run an external program

2194

with parameters is:

2195

<pre class="pre-section"> ./spider.pl spider.conf | ./swish-e -S prog -i stdin</pre>

2196

</li>

2197

</ul>

2198

Notes when using MS Windows

2199

You should use unix style path separators to specify your external program.

2200

Swish will convert forward slashes to backslashes before calling the external

2201

program. This is only true for the program name specified with <code>IndexDir</code> or

2202

the <code>-i</code> command line option.

2203

In addition, Swish-e will make sure the program specified actually exists,

2204

which means you need to use the full name of the program.

2205

For example, to run the perl spider program spider.pl you would need

2206

a Swish-e configuration file such as:

2207

<pre class="pre-section"> IndexDir e:/perl/bin/perl.exe

2208

SwishProgParameters prog-bin/spider.pl default http://swish-e.org</pre>

2209

and run indexing with the command:

2210

<pre class="pre-section"> swish-e -c swish.cfg -S prog -v 9</pre>

2211

The <code>IndexDir</code> command tells Swish-e the name of the program to run. Under

2212

unix you can just specify the name of the script, since unix will figure out

2213

the program from the first line of the script.

2214

The <code>SwishProgParameters</code> are the parameters passed to the program specified

2215

by <code>IndexDir</code> (perl.exe in this case). The first parameter is the perl script

2216

to run (prog-bin/spider.pl). Perl passes the rest of the parameters

2217

directly to the perl script. The second parameter default tells the

2218

spider.pl program to use default settings for spidering (or you could

2219

specify a spider config file -- see <code>perldoc spider.pl</code> for details), and

2220

lastly, the URL is passed into the spider program.

2221

2222

</div>

2223

2224

2225

2226

<h2><a name="document_filter_directives"></a>Document Filter Directives</h2>

2227

2228

Internally, Swish-e knows how to parse only text, HTML, and XML documents.

2229

With "filters" you can index other types of documents. For example,

2230

if all your web pages are in gzip format a filter can uncompress these

2231

on the fly for indexing.

2232

You may wish to read the Swish-e FAQ question on filtering before continuing

2233

here. <a href="swish-faq.html#how_do_i_filter_documents_">How Do I filter documents?</a>

2234

There are two suggested methods for filtering.

2235

2236

</div>

2237

2238

2239

2240

<h3><a name="filtering_with_swish_filter"></a>Filtering with SWISH::Filter</h3>

2241

2242

The Swish-e distribution includes a Perl module called SWISH::Filter and individual

2243

filters located in the filters directory. This system uses plug-in filters to

2244

extend the types of documents that Swish-e can index. The plug-in filters do not

2245

actually do the filtering, but rather provide a standard interface for accessing programs that

2246

can filter or convert documents. The programs that do the filtering are not part of

2247

the Swish-e distribution; they must be downloaded and installed separately.

2248

The advantage of this method is that new filtering methods can be installed easily.

2249

This system is designed to work with the -S http and -prog methods, but may

2250

also be used with the <code>FileFilter</code> feature and -S fs indexing method. See

2251

$prefix/share/doc/swish-e/examples/filter-bin/swish_filter.pl for an

2252

example.

2253

See the filters/README file for more information.

2254

2255

</div>

2256

2257

2258

2259

<h3><a name="filtering_with_the_filefilter_feature"></a>Filtering with the FileFilter feature</h3>

2260

2261

A filter is an external program that Swish-e executes while processing

2262

a document of a given type. Swish-e will execute the filter program

2263

for each file that matches the file suffix (extension) set in the

2264

FileFilter or FileFilterMatch directives. FileFilterMatch

2265

matches using regular expressions and is described below.

2266

Filters may be used with any type of input method (i.e. -S fs, -S http, or -S prog).

2267

But because

2268

Swish-e calls the external program passing as default arguments:

2269

<ul>

2270

<li><a name="item__0"></a><a name="_0"></a>$0

2271

the name of the filter program

2272

</li>

2273

<li><a name="item__1"></a><a name="_1"></a>$1

2274

the physical path name of the file to read. This may be a temporary

2275

file location if indexing by the http method.

2276

</li>

2277

<li><a name="item__2"></a><a name="_2"></a>$2

2278

When indexing under the file system this will be the same as $1 (the

2279

path to the source file), but when indexing under the http method this

2280

will be the URL of the source document.

2281

</li>

2282

</ul>

2283

Swish-e can also pass other parameters to the filter program. These

2284

parameters can be defined using the FileFilter or FileFilterMatch

2285

directives. See Filter Options below.

2286

The filter program must open the file, process its contents, and return

2287

it to Swish-e by printing to STDOUT.

2288

Note that this can add a significant amount of time to the indexing

2289

process if your external program is a perl or shell script. If you

2290

have many files to filter you should consider writing your filter in C

2291

instead of a shell or perl script, or using the "prog" Access Method

2292

along with SWISH::Filter.

2293

<ul>

2294

<li><a name="item_filterdir"></a><a name="filterdir"></a>FilterDir *path-to-directory*

2295

Deprecated.

2296

This is the path to a directory where the filter programs are stored.

2297

Swish-e looks in this directory to find the filter specified in the

2298

FileFilter directive.

2299

This directive is not needed if the filter program can be found in

2300

your system's path. Even if your filter is not in your system's path

2301

you can specify the full path to the filter in the FileFilter or

2302

FileFilterMatch directives.

2303

Example:

2304

<pre class="pre-section"> FilterDir /usr/local/swish/filters</pre>

2305

</li>

2306

<li><a name="item_filefilter"></a><a name="filefilter"></a>FileFilter *suffix* "filter-prog" ["filter-options"]

2307

This maps file suffix (extension) to a filter program. If filter-prog

2308

starts with a directory delimiter (absolute path), Swish-e doesn't use

2309

the FilterDir settings, but uses the given filter-prog path directly.

2310

On systems that have a working fork(2) system call the filter program

2311

is run by forking swish then executing the filter. This mean the shell

2312

is not used for running the filter and no arguments are passed through the

2313

shell.

2314

On other systems (e.g. Windows) the arguments are double-quoted and

2315

popen(3) is used to run the program. This does pass argument though

2316

the shell and may be a security concern depending on the abilities of

2317

the shell.

2318

Filter options:

2319

Filter options are a string passed as arguments to the filter-prog.

2320

Filter options can contain variables, replaced by Swish-e. If you omit

2321

filter-options Swish-e will use default parameters for the options

2322

listed above.

2323

<pre class="pre-section"> Default: %p %P

2324

Which means: pass "workfile path" and "documentfile path" to filter.</pre>

2325

Variables in filter options:

2326

<pre class="pre-section"> %% = %

2327

%P = Full document pathname (e.g. URL, or path on filesystem)

2328

%p = Full pathname to work file (maybe a tmpfile or the real document path on filesystem)

2329

%F = Filename stripped from full document pathname

2330

%f = Filename stripped from "work" pathname

2331

%D = Directoryname stripped from full document pathname

2332

%d = Directoryname stripped from full "work" pathname</pre>

2333

Examples of strings passed:

2334

<pre class="pre-section"> %P = document pathname: http://myserver/path1/mydoc.txt

2335

%p = work pathname: /tmp/tmp.1234.mydoc.txt

2336

%F = mydoc.txt

2337

%f = tmp.1234.mydoc.txt

2338

%D = http://myserver/path1

2339

%d = /tmp</pre>

2340

Notes when using MS Windows

2341

Windows uses double quotes to escape shell metacharacters, so if you need

2342

to use quotes then use single quotes around the entire option string.

2343

<pre class="pre-section"> FileFiler .mydoc mydocfilter.exe '--title "text with spaces"'</pre>

2344

You can specify the filter program using forward slashes (unix style).

2345

Swish will convert the slashes to backslashes before running your program.

2346

<pre class="pre-section"> FileFilter .mydoc c:/some/path/mydocfilter.exe '-d "%d" -example -url "%P" "%f"'</pre>

2347

Examples of filters:

2348

<pre class="pre-section"> FileFilter .doc /usr/local/bin/catdoc "-s8859-1 -d8859-1 %p"

2349

FileFilter .pdf pdftotext "%p -"

2350

FileFilter .html.gz gzip "-c %p"

2351

FileFilter .mydoc "/some/path/mydocfilter" "-d %d -example -url %P %f"</pre>

2352

The above examples are running a binary filter program. For more

2353

complicated filtering needs you may use a scripting language such as

2354

Perl or a shell script. Here's some examples of calling a shell and

2355

perl script:

2356

<pre class="pre-section"> FileFilter .pdf pdf2html.sh

2357

FileFilter .ps ghostscript-filter.pl</pre>

2358

Using a scripting language (or any language that has a large startup cost) can

2359

greatly increase the indexing time. For small indexing jobs, this may not

2360

be an issue, but for large collections of files that require processing by a

2361

scripting language, you may be better off using the <code>-S prog</code> access method

2362

where the script will only be compiled once, instead of for each document.

2363

Filters are probably easier to write than a <code>-S prog</code> program. Which you

2364

decide to use depends on your requirements. Examples of filter scripts can be

2365

found in the filter-bin directory, and examples of <code>-S prog</code> programs can

2366

be found in the prog-bin directory.

2367

</li>

2368

<li><a name="item_filefiltermatch"></a><a name="filefiltermatch"></a>FileFilterMatch *filter-prog* *filter-options* *regex* [*regex* ...]

2369

This is similar to <code>FileMatch</code> except uses regular expressions to match

2370

against the file name. *filter-prog* is the path to the program. Unlike

2371

<code>FileFilter</code> this does not use the <code>FilterDir</code> option. Also unlike

2372

<code>FileFilter</code> you must specify the *filter-options*.

2373

Examples:

2374

<pre class="pre-section"> FileFilterMatch ./pdftotext "%p -" /\.pdf$/</pre>

2375

Note that will also match a file called ".pdf", so you may want to use

2376

something that requires a filename that has more than just an extension.

2377

For example:

2378

<pre class="pre-section"> FileFilterMatch ./pdftotext "%p -" /.\.pdf$/</pre>

2379

To specify more than one extension:

2380

<pre class="pre-section"> FileFilterMatch ./check_title.pl "%p" /\.html$/ /\.htm$/</pre>

2381

Or a few ways to do the same thing:

2382

<pre class="pre-section"> FileFilterMatch ./check_title.pl %p /\.(html|html)$/

2383

FileFilterMatch ./check_title.pl %p /\.html?$/</pre>

2384

And to ignore case:

2385

<pre class="pre-section"> FileFilterMatch ./check_title.pl %p /\.html?$/i</pre>

2386

You may also precede an expression with "not" to negate regular expression

2387

that follow. For example, to match files that do not have an extension:

2388

<pre class="pre-section"> FileFilterMatch ./convert "%p %P" not /\..+$/</pre>

2389

</li>

2390

</ul>

2391

2392

</div>

2393

2394

2395

2396

<h1><a name="document_info"></a>Document Info</h1>

2397

2398

$Id: SWISH-CONFIG.pod,v 1.91 2006/10/20 20:18:30 whmoseley Exp $

2399

.

2400

2401

</div>

2402

2403

2404

2405

2406

2407

2408

2409

2410

2411

2412

2413

2414

2415

2416

2417

</div>

2418

2419

2420

2421

2422

2423

2424

Swish-e is distributed with no warranty under the terms of the

2425

<a href='/license.html'>Swish-e License</a>.

2426

Questions may be posted to the

2427

<a href="http://swish-e.org/discuss.html" title="email list and list archive">Swish-e Discussion list</a>.

2428

2429

2430

2431

<SCRIPT type='text/javascript' language='JavaScript'

2432

src='http://www.ohloh.net/projects/3196;badge_js'></SCRIPT>

2433

2434

2435

2436

URI » http://swish-e.org/

2437

• Updated » Fri, 20 Oct 2006 20:51:46 UTC

2438

2439

</div>

2440

2441

2442

2443

2444

</body>

2445

</html>