~ubuntu-branches/ubuntu/warty/swish-e/warty

require the modules know where to find the modules, but the <EM>perldoc</EM> program used for reading documentation does not. This can be corrected by

716

adding $prefix/lib/swish-e and $prefix/lib/swish-e/perl to the PERL5LIB

717

environment variable.

718

719

<P>

720

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

721

<HR>

722

<H2><A NAME="Documentation">Documentation</A></H2>

723

<P>

724

Documentation can be found in the $prefix/share/doc/swish-e directory.

725

Documentation can also be read on-line at the Swish-e web site:

726

727

<P>

728

729

<table>

730

<tr>

731

732

733

734

</td>

735

736

<td>

737

<pre> <A HREF="http://swish-e.org/">http://swish-e.org/</A></pre>

738

</td>

739

740

</tr>

741

</table>

742

<P>

743

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

744

<HR>

745

<H2><A NAME="The_Swish_e_documentation_as_man_1_pages">The Swish-e documentation as man(1) pages</A></H2>

746

<P>

747

Running ``make install'' installs some of the Swish-e documentation as man

748

pages. The following man pages are installed:

749

750

<P>

751

752

<table>

753

<tr>

754

755

756

757

</td>

758

759

<td>

760

<pre> SWISH-FAQ(1)

761

SWISH-CONFIG(1)

762

SWISH-RUN(1)

763

SWISH-LIBRARY(1)</pre>

764

</td>

765

766

</tr>

767

</table>

768

<P>

769

The man pages are installed in the system man directory. This directory is

770

determined by running ./configure and can be set by passing the directory

771

when running ./configure.

772

773

<P>

774

For example,

775

776

<P>

777

778

<table>

779

<tr>

780

781

782

783

</td>

784

785

<td>

786

<pre> ./configure --mandir=/usr/local/doc/man</pre>

787

</td>

788

789

</tr>

790

</table>

791

<P>

792

Information on running ./configure can be found by typing:

793

794

<P>

795

796

<table>

797

<tr>

798

799

800

801

</td>

802

803

<td>

804

<pre> ./configure --help</pre>

805

</td>

806

807

</tr>

808

</table>

809

<P>

810

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

811

<HR>

812

<H2><A NAME="Join_the_Swish_e_discussion_list">Join the Swish-e discussion list</A></H2>

813

<P>

814

The final step when installing Swish-e is to join the Swish-e discussion

815

list.

816

817

<P>

818

The Swish-e discussion list is the place to ask questions about installing

819

and using Swish-e, see or post bug fixes or security announcements, and a

820

place where <STRONG>you</STRONG> can offer help to others. Please do not contact the developers directly.

821

822

<P>

823

The list is typically <EM>very low traffic</EM>, so it won't overload your inbox. Please take time to subscribe. See <A

824

HREF="http://Swish-e.org.">http://Swish-e.org.</A>

825

826

<P>

827

If you are using Swish-e on a public site, please let the list know so it

828

can be added to the list of sites that use Swish-e!

829

830

<P>

831

Please review the next section before posting a question to the Swish-e

832

list.

833

834

<P>

835

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

836

<HR>

837

<H1><A NAME="QUESTIONS_AND_TROUBLESHOOTING">QUESTIONS AND TROUBLESHOOTING</A></H1>

838

<P>

839

Support for installation, configuration and usage is available via the

840

Swish-e discussion list. Visit <A

841

HREF="http://swish-e.org">http://swish-e.org</A> for information. Do not

842

contact developers directly for help -- always post your question to the

843

list.

844

845

<P>

846

It's very important to provide the right information when asking for help.

847

848

<P>

849

Please search the Swish-e list archive before posting a question, and check

850

the <A HREF="././SWISH-FAQ.html">SWISH-FAQ</A> to see if your question hasn't already been asked.

851

852

<P>

853

Before posting use tools available to narrow down the problem.

854

855

<P>

856

Swish-e has the -T, -v, and -k switches that may help resolve issues. These

857

switches are described on the <A HREF="././SWISH-RUN.html">SWISH-RUN</A> page. For example, if you cannot find a document by a keyword that you

858

believe should be indexed try indexing just that single file, and use the

859

-T INDEXED_WORDS option to see if the word is actually being indexed. First

860

try without any changes to default settings:

861

862

<P>

863

864

<table>

865

<tr>

866

867

868

869

</td>

870

871

<td>

872

<pre> swish-e -i testdoc.html -T indexed_words | less</pre>

873

</td>

874

875

</tr>

876

</table>

877

<P>

878

if that works then add in your configuration file:

879

880

<P>

881

882

<table>

883

<tr>

884

885

886

887

</td>

888

889

<td>

890

<pre> swish-e -i testdoc.html -c swish.conf -T indexed_words | less</pre>

891

</td>

892

893

</tr>

894

</table>

895

<P>

896

If that still isn't working as you expect try to reduce the test document

897

to a very small example. This will be very helpful when asking for help.

898

899

<P>

900

Other tools are to use -H9 when searching to display full headers in search

901

results. Look at the ``Parsed Words'' header to see what words swish-e is

902

searching for.

903

904

<P>

905

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

906

<HR>

907

<H2><A NAME="When_posting_please_provide_the_following_information_">When posting please provide the following information:</A></H2>

908

<UL>

909

<P><LI>

910

<P>

911

The exact version of Swish-e that you are using. Running Swish-e with the

912

<CODE>-V</CODE> switch will print the version number. Also, supply the output from

913

<CODE>uname -a</CODE> or similar command that identifies the operating system you are running on.

914

If you are running an old version of swish be prepared for a response to

915

your question of ``upgrade.''

916

917

<P><LI>

918

<P>

919

A summary of the problem. This should include the commands issued (e.g. for

920

indexing or searching) and their output, and why you don't think it's

921

working correctly. Please cut-n-paste the exact commands and their output

922

instead of retyping to avoid errors.

923

924

<P><LI>

925

<P>

926

Include a copy of the configuration file you are using, if any. Swish-e has

927

reasonable defaults so in many cases you can run it without using a

928

configuration file. But, if you need to use a configuration file,

929

<STRONG>reduce it down</STRONG> to the absolute minimum number of commands required to demonstrate your

930

problem. Again, cut-n-paste.

931

932

<P><LI>

933

<P>

934

A small copy of a source document that demonstrates the problem.

935

936

<P>

937

If you are having problems spidering a web server, use lwp-download or wget

938

to copy the file locally to make sure you can index the document using the

939

file system method. This will help determine if the problem is with

940

spidering or with indexing.

941

942

<P>

943

If you expect help with spidering, don't post fake URLs, as it makes it

944

impossible to test. If you don't want to expose your web page to the people

945

on the Swish-e list, find some other site to test spidering on. If that

946

works, but you still cannot spider your own site then post your real URL if

947

you want help, or make a test document available via some other source.

948

949

<P><LI>

950

<P>

951

If you are having trouble building Swish-e please cut-n-paste the output

952

from make (or from ./configure if that's where the problem is).

953

954

</UL>

955

<P>

956

The key is to provide enough information so that others may reproduce the

957

problem.

958

959

<P>

960

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

961

<HR>

962

<H1><A NAME="ADDITIONAL_INSTALLATION_OPTIONS">ADDITIONAL INSTALLATION OPTIONS</A></H1>

963

<P>

964

These steps are not required for normal use of Swish-e.

965

966

<P>

967

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

968

<HR>

969

<H2><A NAME="The_SWISH_API_Perl_Module">The SWISH::API Perl Module</A></H2>

970

<P>

971

The Swish-e distribution includes a module that provides a Perl interface

972

to the Swish-e C library. This module provides a way to search a Swish-e

973

index without running the swish-e program. Searching an index will be many

974

times faster when running under a persistent environment such as

975

Apache/mod_perl with the SWISH::API module.

976

977

<P>

978

See the <EM>perl/README</EM> file for information on installing and using the SWISH::API Perl module.

979

980

<P>

981

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

982

<HR>

983

<H2><A NAME="Creating_PDF_and_Postscript_documentation">Creating PDF and Postscript documentation</A></H2>

984

<P>

985

The Swish-e documentation in HTML format was created with Pod::HtmlPsPdf, a

986

package of Perl modules written and/or modified by Stas Bekman to automate

987

the conversion of documents in pod format (see perldoc perlpod) to HTML,

988

Postscript, and PDF. A slightly modified version of this package is

989

included with the Swish-e distribution and used for building the HTML.

990

991

<P>

992

If your system has the <STRONG>necessary tools</STRONG> to build Postscript and the converter ps2pdf installed, you may be able to

993

build the Postscript and PDF versions of the documentation. After you have

994

run ./configure, type from the <EM>doc</EM> directory of the distribution:

995

996

<P>

997

998

<table>

999

<tr>

1000

1001

1002

1003

</td>

1004

1005

<td>

1006

1007

</td>

1008

1009

</tr>

1010

</table>

1011

<P>

1012

And with any luck you will end up with the these two files in the top-level

1013

directory:

1014

1015

<P>

1016

1017

<table>

1018

<tr>

1019

1020

1021

1022

</td>

1023

1024

<td>

1025

<pre> swish-e_documentation.pdf

1026

swish-e_documentation.ps</pre>

1027

</td>

1028

1029

</tr>

1030

</table>

1031

<P>

1032

Most people find reading the documentation in HTML most convenient.

1033

1034

<P>

1035

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

1036

<HR>

1037

<H1><A NAME="GENERAL_CONFIGURATION_AND_USAGE">GENERAL CONFIGURATION AND USAGE</A></H1>

1038

<P>

1039

This section should give you a basic overview of indexing and searching

1040

with <STRONG>Swish-e</STRONG>. Other examples can be found in the <EM>conf</EM> directory which will step you through a number of different configurations.

1041

Also, please review the <A HREF="././SWISH-FAQ.html">SWISH-FAQ</A>.

1042

1043

<P>

1044

Swish-e is a command line program. The program is controlled by passing

1045

switches on the command line. A configuration file may be used, but often

1046

is not required. Swish-e does not include a graphical user interface. There

1047

are example CGI scripts provided in the distribution, but they require

1048

additional setup to use.

1049

1050

<P>

1051

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

1052

<HR>

1053

<H2><A NAME="Introduction_to_Indexing_and_Searching">Introduction to Indexing and Searching</A></H2>

1054

<P>

1055

Swish-e can index files on the local file system. For example, running:

1056

1057

<P>

1058

1059

<table>

1060

<tr>

1061

1062

1063

1064

</td>

1065

1066

<td>

1067

<pre> swish-e -i /var/www/htdocs</pre>

1068

</td>

1069

1070

</tr>

1071

</table>

1072

<P>

1073

will index <EM>all</EM> files in the /var/www/htdocs directory. You may specify one or more files

1074

or directories with the -i option. By default this will create an index

1075

(which is made up of more than one file) in the current directory called <EM>index.swish-e</EM>.

1076

1077

<P>

1078

Then to search the resulting index for a given word:

1079

1080

<P>

1081

1082

<table>

1083

<tr>

1084

1085

1086

1087

</td>

1088

1089

<td>

1090

<pre> swish-e -w apache</pre>

1091

</td>

1092

1093

</tr>

1094

</table>

1095

<P>

1096

This will find the word ``apache'' in the body or title of the indexed

1097

documents.

1098

1099

<P>

1100

As mentioned above, Swish-e will index all files in a directory unless

1101

instructed otherwise. So if /var/www/htdocs contains non-HTML then you will

1102

need a configuration file to limit the files that Swish-e indexes. Create a

1103

file called ``swish.conf'':

1104

1105

<P>

1106

1107

<table>

1108

<tr>

1109

1110

1111

1112

</td>

1113

1114

<td>

1115

<pre> # Example configuration file</pre>

1116

</td>

1117

1118

</tr>

1119

</table>

1120

<P>

1121

1122

<table>

1123

<tr>

1124

1125

1126

1127

</td>

1128

1129

<td>

1130

<pre> # Tell swish what to index (same as -i switch above)

1131

IndexDir /var/www/htdocs</pre>

1132

</td>

1133

1134

</tr>

1135

</table>

1136

<P>

1137

1138

<table>

1139

<tr>

1140

1141

1142

1143

</td>

1144

1145

<td>

1146

<pre> # Only index HTML and text files

1147

IndexOnly .htm .html .txt</pre>

1148

</td>

1149

1150

</tr>

1151

</table>

1152

<P>

1153

1154

<table>

1155

<tr>

1156

1157

1158

1159

</td>

1160

1161

<td>

1162

<pre> # Tell swish that .txt files are to use the text parser.

1163

IndexContents TXT* .txt</pre>

1164

</td>

1165

1166

</tr>

1167

</table>

1168

<P>

1169

1170

<table>

1171

<tr>

1172

1173

1174

1175

</td>

1176

1177

<td>

1178

<pre> # Otherwise, use the HTML parser

1179

DefaultContents HTML*</pre>

1180

</td>

1181

1182

</tr>

1183

</table>

1184

<P>

1185

Save that as ``swish.conf'' and reindex:

1186

1187

<P>

1188

1189

<table>

1190

<tr>

1191

1192

1193

1194

</td>

1195

1196

<td>

1197

<pre> swish-e -c swish.conf</pre>

1198

</td>

1199

1200

</tr>

1201

</table>

1202

<P>

1203

The Swish-e configuration settings are described in the

1204

<A HREF="././SWISH-CONFIG.html">SWISH-CONFIG</A> manual page. Order of statements in the configuration file is typically not

1205

important, although some statements depend on previously set statements.

1206

There are many possible settings. Good advice is to use as few settings as

1207

possible when first starting out with Swish-e.

1208

1209

<P>

1210

The runtime options (switches) are described in the <A HREF="././SWISH-RUN.html">SWISH-RUN</A>

1211

manual page. You may also see a summary of options by running:

1212

1213

<P>

1214

1215

<table>

1216

<tr>

1217

1218

1219

1220

</td>

1221

1222

<td>

1223

<pre> swish-e -h</pre>

1224

</td>

1225

1226

</tr>

1227

</table>

1228

<P>

1229

Swish-e has two other methods reading input files. One method uses a Perl

1230

helper script and the LWP Perl library to spider remote web sites:

1231

1232

<P>

1233

1234

<table>

1235

<tr>

1236

1237

1238

1239

</td>

1240

1241

<td>

1242

<pre> swish-e -S http -i <A HREF="http://localhost/index.html">http://localhost/index.html</A> -v2</pre>

1243

</td>

1244

1245

</tr>

1246

</table>

1247

<P>

1248

This will spider the web server running on the local host. The <CODE>-S</CODE> option defines the input source method to be ``http'', <CODE>-i</CODE> specifies the URL to spider, and <CODE>-v</CODE> sets the verbose level to two. There are a number of configuration options

1249

specific to the <CODE>-S</CODE>

1250

http input source. See <A HREF="././SWISH-CONFIG.html">SWISH-CONFIG</A>. Note that only files of Content-Type text/* will be indexed.

1251

1252

<P>

1253

The <CODE>-S http</CODE> method is depreciated in favor of the next input method.

1254

1255

<P>

1256

The other method is a general purpose input method where Swish-e reads

1257

input from a program that produces documents in a special format. The

1258

program might read and format data stored in a database, or parse and

1259

format messages in a mailing list archive, or run a program that spiders

1260

web sites like the previous method.

1261

1262

<P>

1263

The Swish-e distribution includes a spider program that uses this method of

1264

input. This spider program is much more configurable and feature-rich than

1265

the previous -S http method.

1266

1267

<P>

1268

To duplicate the previous example create a configuration file called

1269

``swish2.conf''

1270

1271

<P>

1272

1273

<table>

1274

<tr>

1275

1276

1277

1278

</td>

1279

1280

<td>

1281

<pre> # Example for spidering

1282

# Use the "spider.pl" program included with Swish-e

1283

IndexDir spider.pl</pre>

1284

</td>

1285

1286

</tr>

1287

</table>

1288

<P>

1289

1290

<table>

1291

<tr>

1292

1293

1294

1295

</td>

1296

1297

<td>

1298

<pre> # Define what site to index

1299

SwishProgParameters default <A HREF="http://localhost/index.html">http://localhost/index.html</A></pre>

1300

</td>

1301

1302

</tr>

1303

</table>

1304

<P>

1305

Then create the index using the command:

1306

1307

<P>

1308

1309

<table>

1310

<tr>

1311

1312

1313

1314

</td>

1315

1316

<td>

1317

<pre> swish-e -S prog -c swish2.conf</pre>

1318

</td>

1319

1320

</tr>

1321

</table>

1322

<P>

1323

This says to use the <CODE>-S prog</CODE> input source method. Note that in this case the IndexDir settings does not

1324

list a file or directory to index, but a program name run. This program,

1325

spider.pl, does the work of fetching the documents from the web server and

1326

passing them to Swish-e for indexing.

1327

1328

<P>

1329

The SwishProgParameters options is a special feature that allows passing

1330

command line parameters to the program specified with IndexDir. In this

1331

case passing the word ``default'' which tells spider.pl to use default

1332

settings, and the URL to spider.

1333

1334

<P>

1335

Running a script under Windows requires specifying the interpreter (e.g.

1336

perl.exe) and then use SwishPropParameters to specify the script and the

1337

script's parameters. See <EM>Notes when using -S prog on MS Windows</EM> on the

1338

<A HREF="././SWISH-RUN.html">SWISH-RUN</A> page.

1339

1340

<P>

1341

The advantage of the <CODE>-S prog</CODE> method of spidering (over the previous <CODE>-S http</CODE> method) is that the Perl code is only compiled once instead of for every

1342

document fetched from the web server. In addition it is a much more

1343

advanced spider with many, many features. Still, as used here, spider.pl

1344

will automatically index PDF or MS Word documents if (when) Xpdf and Catdoc

1345

are installed.

1346

1347

<P>

1348

A special form of the <CODE>-S prog</CODE> input source method is:

1349

1350

<P>

1351

1352

<table>

1353

<tr>

1354

1355

1356

1357

</td>

1358

1359

<td>

1360

<pre> ./myprog --option | swish-e -S prog -i stdin -c config</pre>

1361

</td>

1362

1363

</tr>

1364

</table>

1365

<P>

1366

This allows running Swish-e from a program (instead of running the external

1367

program from Swish-e). Thus, this also can be done:

1368

1369

<P>

1370

1371

<table>

1372

<tr>

1373

1374

1375

1376

</td>

1377

1378

<td>

1379

<pre> ./myprog --option > outfile

1380

swish-e -S prog -i stdin -c config < outfile</pre>

1381

</td>

1382

1383

</tr>

1384

</table>

1385

<P>

1386

1387

1388

<P>

1389

1390

<table>

1391

<tr>

1392

1393

1394

1395

</td>

1396

1397

<td>

1398

<pre> ./myprog --option > outfile

1399

cat outfile | swish-e -S prog -i stdin -c config</pre>

1400

</td>

1401

1402

</tr>

1403

</table>

1404

<P>

1405

One final note about the <CODE>-S prog</CODE> input source method. The program specified with -i or IndexDir needs to be

1406

an absolute path. The exception is when the program is installed in the

1407

``libexecdir'' directory and then a plain program name may be specified (as

1408

in the example showing spider.pl above).

1409

1410

<P>

1411

All three input source methods are described in more detail on the <A HREF="././SWISH-RUN.html">SWISH-RUN</A> page.

1412

1413

<P>

1414

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

1415

<HR>

1416

<H2><A NAME="Metanames_and_Properties">Metanames and Properties</A></H2>

1417

<P>

1418

There's two key Swish-e concepts that you need to be familiar with:

1419

Metanames and Properties.

1420

1421

<UL>

1422

<P><LI><STRONG><A NAME="item_Metanames">Metanames</A></STRONG>

1423

<P>

1424

Swish-e creates a reverse index. Just like an index in a book, you look up

1425

a word and it lists the pages (or documents) where that word can be found.

1426

1427

<P>

1428

Swish-e can create multiple index tables within the same index file. For

1429

example, you might want to create an index of just words in HTML titles so

1430

searches can be limited to just titles. Or you might have descriptive words

1431

in a meta tag called ``keywords'' you would like to search.

1432

1433

<P>

1434

Some database systems might call these different ``fields'' or ``columns'',

1435

but swish-e calls them <EM>MetaNames</EM> (as a result of first indexing HTML meta tags).

1436

1437

<P>

1438

To find documents with ``foo'' in their title you might run:

1439

1440

<P>

1441

1442

<table>

1443

<tr>

1444

1445

1446

1447

</td>

1448

1449

<td>

1450

<pre> swish-e -w swishtitle=foo</pre>

1451

</td>

1452

1453

</tr>

1454

</table>

1455

<P>

1456

or more advanced:

1457

1458

<P>

1459

1460

<table>

1461

<tr>

1462

1463

1464

1465

</td>

1466

1467

<td>

1468

<pre> swish-e -w swishtitle=(foo or bar) or swishdefault=(baz)</pre>

1469

</td>

1470

1471

</tr>

1472

</table>

1473

<P>

1474

The Metaname ``swishdefault'' is the name used by Swish-e if no other name

1475

is specified. The following two searches are the same:

1476

1477

<P>

1478

1479

<table>

1480

<tr>

1481

1482

1483

1484

</td>

1485

1486

<td>

1487

<pre> swish-e -w foo

1488

swish-e -w swishdefault=foo</pre>

1489

</td>

1490

1491

</tr>

1492

</table>

1493

<P>

1494

When indexing HTML documents Swish-e indexes words in the body and title

1495

under the Metaname ``swishdefault''.

1496

1497

<P><LI><STRONG><A NAME="item_Properties">Properties</A></STRONG>

1498

<P>

1499

Swish-e search results is a list of files -- actually internally swish uses

1500

file numbers. Data can be associated with each file number when indexing.

1501

For example, by default Swish-e associates the file's name, title, last

1502

modified date, and size with the file number and these items can be printed

1503

in search results. In Swish-e this associated data is called a file's

1504

<EM>Properties</EM>. Properties can be any data you wish to associated with a document -- even

1505

the entire text of the document can be stored in the index. What data is

1506

stored as a Property is controlled by the <EM>PropertyNames</EM> (and others) configuration directive.

1507

1508

<P>

1509

What properties are printed with search results depends on the -x or -p

1510

switches. By default Swish-e returns the rank, path/URL, title and file

1511

size in bytes for each result.

1512

1513

</UL>

1514

<P>

1515

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

1516

<HR>

1517

<H2><A NAME="Getting_Started_With_Swish_e">Getting Started With Swish-e</A></H2>

1518

<P>

1519

Swish-e reads a configuration file (see <A HREF="././SWISH-CONFIG.html">SWISH-CONFIG</A>) for directives that control what and how Swish-e indexes files. Swish-e

1520

is also controlled by command line arguments (see

1521

<A HREF="././SWISH-RUN.html">SWISH-RUN</A>). Many of the command line arguments have equivalent configuration

1522

directives (e.g. -i and IndexDir).

1523

1524

<P>

1525

Swish-e does not require a configuration file, but most people need to

1526

change the default behavior by placing settings in a configuration file.

1527

1528

<P>

1529

To try the examples below you may change to the <EM>tests</EM> subdirectory of the distribution. The tests will use the *.html files in

1530

this directory when creating the test index. You may wish to review these

1531

*.html files to get an idea of the various native file formats that Swish-e

1532

supports.

1533

1534

<P>

1535

You may also use your own test documents. It's recommended to use small

1536

test documents when first using Swish-e.

1537

1538

<P>

1539

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

1540

<HR>

1541

<H2><A NAME="Step_1_Create_a_Configuration_File">Step 1: Create a Configuration File</A></H2>

1542

<P>

1543

The configuration file controls what and how Swish-e indexes. The

1544

configuration file consists of directives, comments, and blank lines. The

1545

configuration file can be any name you like.

1546

1547

<P>

1548

This example will work with the documents in the <EM>tests</EM> directory. You may wish to review the <EM>tests/test.config</EM> configuration file used for the <CODE>make test</CODE> tests.

1549

1550

<P>

1551

For example, a simple configuration file (<EM>swish-e.conf</EM>):

1552

1553

<P>

1554

1555

<table>

1556

<tr>

1557

1558

1559

1560

</td>

1561

1562

<td>

1563

<pre> # Example Swish-e Configuration file</pre>

1564

</td>

1565

1566

</tr>

1567

</table>

1568

<P>

1569

1570

<table>

1571

<tr>

1572

1573

1574

1575

</td>

1576

1577

<td>

1578

<pre> # Define *what* to index

1579

# IndexDir can point to a directories and/or a files

1580

# Here it's pointing to the current directory

1581

# Swish-e will also recurse into sub-directories.

1582

IndexDir .</pre>

1583

</td>

1584

1585

</tr>

1586

</table>

1587

<P>

1588

1589

<table>

1590

<tr>

1591

1592

1593

1594

</td>

1595

1596

<td>

1597

<pre> # But only index the .html files

1598

IndexOnly .html</pre>

1599

</td>

1600

1601

</tr>

1602

</table>

1603

<P>

1604

1605

<table>

1606

<tr>

1607

1608

1609

1610

</td>

1611

1612

<td>

1613

<pre> # Show basic info while indexing

1614

IndexReport 1</pre>

1615

</td>

1616

1617

</tr>

1618

</table>

1619

<P>

1620

And that's a simple configuration file. It says to index all the .html

1621

files in the current directory and sub-directories, if any, and provide

1622

some basic output while indexing.

1623

1624

<P>

1625

As mentioned above, the complete list of all configuration file directives

1626

are described in <A HREF="././SWISH-CONFIG.html">SWISH-CONFIG</A>.

1627

1628

<P>

1629

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

1630

<HR>

1631

<H2><A NAME="Step_2_Index_your_Files">Step 2: Index your Files</A></H2>

1632

<P>

1633

Run Swish-e using the <CODE>-c</CODE> switch to specify the name of the configuration file.

1634

1635

<P>

1636

1637

<table>

1638

<tr>

1639

1640

1641

1642

</td>

1643

1644

<td>

1645

<pre> swish-e -c swish-e.conf</pre>

1646

</td>

1647

1648

</tr>

1649

</table>

1650

<P>

1651

1652

<table>

1653

<tr>

1654

1655

1656

1657

</td>

1658

1659

<td>

1660

<pre> Indexing Data Source: "File-System"

1661

Indexing "."

1662

Removing very common words...

1663

no words removed.

1664

Writing main index...

1665

Sorting words ...

1666

Sorting 55 words alphabetically

1667

Writing header ...

1668

Writing index entries ...

1669

Writing word text: Complete

1670

Writing word hash: Complete

1671

Writing word data: Complete

1672

55 unique words indexed.

1673

4 properties sorted.

1674

5 files indexed. 1252 total bytes. 140 total words.

1675

Elapsed time: 00:00:00 CPU time: 00:00:00

1676

Indexing done!</pre>

1677

</td>

1678

1679

</tr>

1680

</table>

1681

<P>

1682

This created the index file <EM>index.swish-e</EM>. This is the default index file name unless the <STRONG>IndexFile</STRONG> directive is specified in the configuration file:

1683

1684

<P>

1685

1686

<table>

1687

<tr>

1688

1689

1690

1691

</td>

1692

1693

<td>

1694

<pre> IndexFile ./website.index</pre>

1695

</td>

1696

1697

</tr>

1698

</table>

1699

<P>

1700

You may use the -f switch to specify a index file at indexing time. The -f

1701

option overrides a IndexFile setting in the configuration file.

1702

1703

<P>

1704

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

1705

<HR>

1706

<H2><A NAME="Step_3_Search">Step 3: Search</A></H2>

1707

<P>

1708

You specify your search terms with the <CODE>-w</CODE> switch. For example, to find the files that contain the word <STRONG>sample</STRONG> you would issue the command:

1709

1710

<P>

1711

1712

<table>

1713

<tr>

1714

1715

1716

1717

</td>

1718

1719

<td>

1720

<pre> swish-e -w sample</pre>

1721

</td>

1722

1723

</tr>

1724

</table>

1725

<P>

1726

This example assumes that you are in the <EM>tests</EM> directory. Swish-e returns in response to that command the following:

1727

1728

<P>

1729

1730

<table>

1731

<tr>

1732

1733

1734

1735

</td>

1736

1737

<td>

1738

<pre> swish-e -w sample</pre>

1739

</td>

1740

1741

</tr>

1742

</table>

1743

<P>

1744

1745

<table>

1746

<tr>

1747

1748

1749

1750

</td>

1751

1752

<td>

1753

<pre> # SWISH format: 2.4.0

1754

# Search words: sample

1755

# Number of hits: 2

1756

# Search time: 0.000 seconds

1757

# Run time: 0.005 seconds

1758

1000 ./test_xml.html "If you are seeing this, the METATAG XML search was successful!" 159

1759

1000 ./test.html "If you are seeing this, the test was successful!" 437

1760

.</pre>

1761

</td>

1762

1763

</tr>

1764

</table>

1765

<P>

1766

So the word <STRONG>sample</STRONG> was found in two documents. The first number shown is the relevance or rank

1767

of the search term, followed by the file containing the search term, the

1768

title of the document, and finally the length of the document.

1769

1770

<P>

1771

The period (``.'') alone at the end marks the end of results.

1772

1773

<P>

1774

Much more information may be retrieved while searching by using the <CODE>-x</CODE> and <CODE>-H</CODE> switches (see <A HREF="././SWISH-RUN.html">SWISH-RUN</A>) and by using Document Properties (see <A HREF="././SWISH-CONFIG.html">SWISH-CONFIG</A>).

1775

1776

<P>

1777

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

1778

<HR>

1779

<H2><A NAME="Phrase_Searching">Phrase Searching</A></H2>

1780

<P>

1781

To search for a phrase in a document use double-quotes to delimit your

1782

search terms. (The default phrase delimiter is set in src/swish.h.)

1783

1784

<P>

1785

You must protect the quotes from the shell.

1786

1787

<P>

1788

For example, under Unix:

1789

1790

<P>

1791

1792

<table>

1793

<tr>

1794

1795

1796

1797

</td>

1798

1799

<td>

1800

<pre> swish-e -w '"this is a phrase" or (this and that)'

1801

swish-e -w 'meta1=("this is a phrase") or (this and that)'</pre>

1802

</td>

1803

1804

</tr>

1805

</table>

1806

<P>

1807

Or under Windows <EM>command.com</EM> shell.

1808

1809

<P>

1810

1811

<table>

1812

<tr>

1813

1814

1815

1816

</td>

1817

1818

<td>

1819

<pre> swish-e -w \"this is a phrase\" or (this and that)</pre>

1820

</td>

1821

1822

</tr>

1823

</table>

1824

<P>

1825

The phrase delimiter can be set with the <CODE>-P</CODE> switch.

1826

1827

<P>

1828

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

1829

<HR>

1830

<H2><A NAME="Boolean_Searching">Boolean Searching</A></H2>

1831

<P>

1832

You can use the Boolean operators <STRONG>and</STRONG>, <STRONG>or</STRONG>, or <STRONG>not</STRONG> in searching. Without these Boolean, Swish-e will assume you're <STRONG>and</STRONG>ing the words together.

1833

1834

<P>

1835

Here are some examples:

1836

1837

<P>

1838

1839

<table>

1840

<tr>

1841

1842

1843

1844

</td>

1845

1846

<td>

1847

<pre> swish-e -w 'apples oranges'

1848

swish-e -w 'apples and oranges' ( Same thing )</pre>

1849

</td>

1850

1851

</tr>

1852

</table>

1853

<P>

1854

1855

<table>

1856

<tr>

1857

1858

1859

1860

</td>

1861

1862

<td>

1863

<pre> swish-e -w 'apples or oranges'</pre>

1864

</td>

1865

1866

</tr>

1867

</table>

1868

<P>

1869

1870

<table>

1871

<tr>

1872

1873

1874

1875

</td>

1876

1877

<td>

1878

<pre> swish-e -w 'apples or oranges not juice' -f myIndex </pre>

1879

</td>

1880

1881

</tr>

1882

</table>

1883

<P>

1884

retrieves first the files that contain both the words ``apples'' and

1885

``oranges''; then among those the ones that do not contain the word

1886

``juice''

1887

1888

<P>

1889

A few others to ponder:

1890

1891

<P>

1892

1893

<table>

1894

<tr>

1895

1896

1897

1898

</td>

1899

1900

<td>

1901

<pre> swish-e -w 'apples and oranges or pears'

1902

swish-e -w '(apples and oranges) or pears' ( Same thing )

1903

swish-e -w 'apples and (oranges or pears)' ( Not the same thing )</pre>

1904

</td>

1905

1906

</tr>

1907

</table>

1908

<P>

1909

Swish processes the query left to right.

1910

1911

<P>

1912

See <A HREF="././SWISH-SEARCH.html">SWISH-SEARCH</A> for more information.

1913

1914

<P>

1915

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

1916

<HR>

1917

<H2><A NAME="Context_Searching">Context Searching</A></H2>

1918

<P>

1919

The <CODE>-t</CODE> option in the search command line allows you to search for words that exist

1920

only in specific HTML tags. Each character in the string you specify in the

1921

argument to this option represents a different tag in which the word is

1922

searched; that is you can use any combinations of the following characters:

1923

1924

<P>

1925

1926

<table>

1927

<tr>

1928

1929

1930

1931

</td>

1932

1933

<td>

1934

<pre> H search in all <HEAD> tags

1935

B search in the <BODY> tags

1936

t search in <TITLE> tags

1937

h is <H1> to <H6> (header) tags

1938

e is emphasized tags (this may be <B>, <I>, <EM>, or <STRONG>)

1939

c is HTML comment tags ()</pre>

1940

</td>

1941

1942

</tr>

1943

</table>

1944

<P>

1945

For example:

1946

1947

<P>

1948

1949

<table>

1950

<tr>

1951

1952

1953

1954

</td>

1955

1956

<td>

1957

<pre> # Find only documents with the word "linux" in the <TITLE> tags.

1958

swish-e -w linux -t t</pre>

1959

</td>

1960

1961

</tr>

1962

</table>

1963

<P>

1964

1965

<table>

1966

<tr>

1967

1968

1969

1970

</td>

1971

1972

<td>

1973

<pre> # Find the word "apple" in titles or comments

1974

swish-e -w apple -t tc</pre>

1975

</td>

1976

1977

</tr>

1978

</table>

1979

<P>

1980

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

1981

<HR>

1982

1983

<P>

1984

As mentioned above, Metanames are a way to define ``fields'' in your

1985

documents. You can use the Metanames in your queries to limit the search to

1986

just the words contained in that META name of your document. For example,

1987

you might have a META tagged field in your documents called

1988

<CODE>subjects</CODE> and then you can search your documents for the word ``foo'' but only return

1989

documents where ``foo'' is within the <CODE>subjects</CODE> META tag.

1990

1991

<P>

1992

Document <EM>Properties</EM> are somewhat related: Properties allow the content of a META tag in a

1993

source document to be stored within the index, and that text to be returned

1994

along with search results.

1995

1996

<P>

1997

META tags can have two formats in your documents.

1998

1999

<P>

2000

2001

<table>

2002

<tr>

2003

2004

2005

2006

</td>

2007

2008

<td>

2009

2010

</td>

2011

2012

</tr>

2013

</table>

2014

<P>

2015

And in XML format

2016

2017

<P>

2018

2019

<table>

2020

<tr>

2021

2022

2023

2024

</td>

2025

2026

<td>

2027

2028

Some Content

2029

</keyName></pre>

2030

</td>

2031

2032

</tr>

2033

</table>

2034

<P>

2035

If using libxml, you can optionally use a non-html tag as a metaname:

2036

2037

<P>

2038

2039

<table>

2040

<tr>

2041

2042

2043

2044

</td>

2045

2046

<td>

2047

2048

<body>

2049

Hello swish users!

2050

2051

this is meta data

2052

</keyName>.

2053

</body></pre>

2054

</td>

2055

2056

</tr>

2057

</table>

2058

<P>

2059

This, of course, is invalid HTML.

2060

2061

<P>

2062

To continue with our sample <EM>Swish-e.conf</EM> file, add the following lines:

2063

2064

<P>

2065

2066

<table>

2067

<tr>

2068

2069

2070

2071

</td>

2072

2073

<td>

2074

<pre> # Define META tags

2075

MetaNames meta1 meta2 meta3</pre>

2076

</td>

2077

2078

</tr>

2079

</table>

2080

<P>

2081

Reindex to include the changes:

2082

2083

<P>

2084

2085

<table>

2086

<tr>

2087

2088

2089

2090

</td>

2091

2092

<td>

2093

<pre> swish-e -c swish-e.conf</pre>

2094

</td>

2095

2096

</tr>

2097

</table>

2098

<P>

2099

Now search, but this time limit your search to META tag ``meta1'':

2100

2101

<P>

2102

2103

<table>

2104

<tr>

2105

2106

2107

2108

</td>

2109

2110

<td>

2111

<pre> swish-e -w 'meta1=metatest1'</pre>

2112

</td>

2113

2114

</tr>

2115

</table>

2116

<P>

2117

Again, please see <A HREF="././SWISH-RUN.html">SWISH-RUN</A> and <A HREF="././SWISH-CONFIG.html">SWISH-CONFIG</A>

2118

for complete documentation of the various indexing and searching options.

2119

2120

<P>

2121

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

2122

<HR>

2123

<H2><A NAME="Spidering_and_Searching_with_a_Web_form_">Spidering and Searching with a Web form.</A></H2>

2124

<P>

2125

This example demonstrates how to spider a web site and setup the included

2126

CGI script to provide a web-based search page. This example uses Perl

2127

programs included in the Swish-e distribution: <EM>spider.pl</EM> will be used for reading files from the web server, and <EM>swish.cgi</EM> will provide the web search form and display results.

2128

2129

<P>

2130

As an example we will index the Apache Web Server documentation installed

2131

on the local computer at <A

2132

HREF="http://localhost/apache_docs/index.html">http://localhost/apache_docs/index.html</A>

2133

2134

2135

<OL>

2136

<P><LI><STRONG><A NAME="item_Make_a_Working_Directory">Make a Working Directory</A></STRONG>

2137

<P>

2138

Create a directory to store the Swish-e configuration and the Swish-e

2139

index.

2140

2141

<P>

2142

2143

<table>

2144

<tr>

2145

2146

2147

2148

</td>

2149

2150

<td>

2151

<pre> ~$ mkdir web_index

2152

~$ cd web_index/

2153

~/web_index$</pre>

2154

</td>

2155

2156

</tr>

2157

</table>

2158

<P><LI><STRONG><A NAME="item_Create_a_Swish_e_Configuration_file">Create a Swish-e Configuration file</A></STRONG>

2159

<P>

2160

2161

<table>

2162

<tr>

2163

2164

2165

2166

</td>

2167

2168

<td>

2169

<pre> ~/web_index$ cat swish.conf

2170

# Swish-e config to index the Apache documentation

2171

2172

# Use spider.pl for indexing (location of spider.pl set at installation time)

2173

IndexDir spider.pl</pre>

2174

</td>

2175

2176

</tr>

2177

</table>

2178

<P>

2179

2180

<table>

2181

<tr>

2182

2183

2184

2185

</td>

2186

2187

<td>

2188

<pre> # Use spider.pl's default configuration and specify the URL to spider

2189

SwishProgParameters default <A HREF="http://localhost/apache_docs/index.html">http://localhost/apache_docs/index.html</A></pre>

2190

</td>

2191

2192

</tr>

2193

</table>

2194

<P>

2195

2196

<table>

2197

<tr>

2198

2199

2200

2201

</td>

2202

2203

<td>

2204

<pre> # Allow extra searching by title, path

2205

Metanames swishtitle swishdocpath</pre>

2206

</td>

2207

2208

</tr>

2209

</table>

2210

<P>

2211

2212

<table>

2213

<tr>

2214

2215

2216

2217

</td>

2218

2219

<td>

2220

<pre> # Set StoreDescription for each parser

2221

# to display context with search results

2222

StoreDescription TXT* 10000

2223

StoreDescription HTML* <body> 10000</pre>

2224

</td>

2225

2226

</tr>

2227

</table>

2228

<P><LI><STRONG><A NAME="item_Generate_the_Index">Generate the Index</A></STRONG>

2229

<P>

2230

Now run swish-e to create the index:

2231

2232

<P>

2233

2234

<table>

2235

<tr>

2236

2237

2238

2239

</td>

2240

2241

<td>

2242

<pre> ~/web_index$ swish-e -S prog -c swish.conf </pre>

2243

</td>

2244

2245

</tr>

2246

</table>

2247

<P>

2248

2249

<table>

2250

<tr>

2251

2252

2253

2254

</td>

2255

2256

<td>

2257

<pre> Indexing Data Source: "External-Program"

2258

Indexing "spider.pl"

2259

/usr/local/lib/swish-e/spider.pl: Reading parameters from 'default'</pre>

2260

</td>

2261

2262

</tr>

2263

</table>

2264

<P>

2265

2266

<table>

2267

<tr>

2268

2269

2270

2271

</td>

2272

2273

<td>

2274

<pre> Summary for: <A HREF="http://localhost/apache_docs/index.html">http://localhost/apache_docs/index.html</A>

2275

Duplicates: 4,188 (349.0/sec)

2276

Off-site links: 276 (23.0/sec)

2277

Skipped: 1 (0.1/sec)

2278

Total Bytes: 2,090,125 (174177.1/sec)

2279

Total Docs: 147 (12.2/sec)

2280

Unique URLs: 149 (12.4/sec)

2281

Removing very common words...

2282

no words removed.

2283

Writing main index...

2284

Sorting words ...

2285

Sorting 7736 words alphabetically

2286

Writing header ...

2287

Writing index entries ...

2288

Writing word text: Complete

2289

Writing word hash: Complete

2290

Writing word data: Complete

2291

7736 unique words indexed.

2292

5 properties sorted.

2293

147 files indexed. 2090125 total bytes. 200783 total words.

2294

Elapsed time: 00:00:13 CPU time: 00:00:02

2295

Indexing done!</pre>

2296

</td>

2297

2298

</tr>

2299

</table>

2300

<P>

2301

The above output is actually a mix of output from both swish-e and from

2302

spider.pl. Spider.pl reports the ``Summary for: <A

2303

HREF="http://localhost/apache_docs/index.html">http://localhost/apache_docs/index.html</A>''.

2304

2305

2306

<P>

2307

Also note that swish-e knows to find spider.pl at

2308

/usr/local/lib/swish-e/spider.pl. The script installation directory (called

2309

libexecdir) is set at configure time. You can see your setting by running

2310

swish-e -h:

2311

2312

<P>

2313

2314

<table>

2315

<tr>

2316

2317

2318

2319

</td>

2320

2321

<td>

2322

<pre> ~/web_index$ swish-e -h | grep libexecdir

2323

Scripts and Modules at: (libexecdir) = /usr/local/lib/swish-e</pre>

2324

</td>

2325

2326

</tr>

2327

</table>

2328

<P>

2329

This directory will be needed when setting up the CGI script in the next

2330

step.

2331

2332

<P>

2333

Finally, verify that the index can be searched from the command line:

2334

2335

<P>

2336

2337

<table>

2338

<tr>

2339

2340

2341

2342

</td>

2343

2344

<td>

2345

<pre> ~/web_index$ swish-e -w installing -m3

2346

# SWISH format: 2.4.0

2347

# Search words: installing

2348

# Removed stopwords:

2349

# Number of hits: 17

2350

# Search time: 0.018 seconds

2351

# Run time: 0.050 seconds

2352

1000 <A HREF="http://localhost/apache_docs/install.html">http://localhost/apache_docs/install.html</A> "Compiling and Installing Apache" 17960

2353

718 <A HREF="http://localhost/apache_docs/install-tpf.html">http://localhost/apache_docs/install-tpf.html</A> "Installing Apache on TPF" 25734

2354

680 <A HREF="http://localhost/apache_docs/windows.html">http://localhost/apache_docs/windows.html</A> "Using Apache with Microsoft Windows" 27165

2355

.</pre>

2356

</td>

2357

2358

</tr>

2359

</table>

2360

<P>

2361

Now try limiting the search to the title:

2362

2363

<P>

2364

2365

<table>

2366

<tr>

2367

2368

2369

2370

</td>

2371

2372

<td>

2373

<pre> ~/web_index$ swish-e -w swishtitle=installing -m3

2374

# SWISH format: 2.3.5

2375

# Search words: swishtitle=installing

2376

# Removed stopwords:

2377

# Number of hits: 2

2378

# Search time: 0.018 seconds

2379

# Run time: 0.048 seconds

2380

1000 <A HREF="http://localhost/apache_docs/install-tpf.html">http://localhost/apache_docs/install-tpf.html</A> "Installing Apache on TPF" 25734

2381

1000 <A HREF="http://localhost/apache_docs/install.html">http://localhost/apache_docs/install.html</A> "Compiling and Installing Apache" 17960

2382

.</pre>

2383

</td>

2384

2385

</tr>

2386

</table>

2387

<P>

2388

Note that the above can also be done using the -t option:

2389

2390

<P>

2391

2392

<table>

2393

<tr>

2394

2395

2396

2397

</td>

2398

2399

<td>

2400

<pre> ~/web_index$ swish-e -w installing -m3 -tH</pre>

2401

</td>

2402

2403

</tr>

2404

</table>

2405

<P><LI><STRONG><A NAME="item_Setup_the_CGI_script">Setup the CGI script</A></STRONG>

2406

<P>

2407

Swish-e does not include a web server, therefore you must use your locally

2408

installed web server. Apache is highly recommended, of course.

2409

2410

<P>

2411

Locate your web server's CGI directory. This may be a cgi-bin directory in

2412

your home directory or a central cgi-bin directory setup by the web server

2413

administrator. Once located copy the swish.cgi script into the cgi-bin

2414

directory.

2415

2416

<P>

2417

Where CGI scripts can be located depends completely on the web server used

2418

and how it is configured. See your web server's documentation or your

2419

site's administrator for additional information.

2420

2421

<P>

2422

This example will use a site cgi-bin directory located at /usr/lib/cgi-bin.

2423

Copy the swish.cgi script into the cgi-bin directory. Again, we will need

2424

the location of the libexecdir directory:

2425

2426

<P>

2427

2428

<table>

2429

<tr>

2430

2431

2432

2433

</td>

2434

2435

<td>

2436

<pre> ~/web_index$ swish-e -h | grep libexecdir

2437

Scripts and Modules at: (libexecdir) = /usr/local/lib/swish-e</pre>

2438

</td>

2439

2440

</tr>

2441

</table>

2442

<P>

2443

2444

<table>

2445

<tr>

2446

2447

2448

2449

</td>

2450

2451

<td>

2452

<pre> ~/web_index$ cd /usr/lib/cgi-bin

2453

/usr/lib/cgi-bin$ su

2454

Password:

2455

/usr/lib/cgi-bin# cp /usr/local/lib/swish-e/swish.cgi .</pre>

2456

</td>

2457

2458

</tr>

2459

</table>

2460

<P>

2461

If your operating system supports symbolic links, AND your web server

2462

allows programs to be symbolic links, then you may wish to create a link to

2463

the swish.cgi program instead.

2464

2465

<P>

2466

2467

<table>

2468

<tr>

2469

2470

2471

2472

</td>

2473

2474

<td>

2475

<pre> /usr/lib/cgi-bin# ln -s /usr/local/lib/swish-e/swish.cgi</pre>

2476

</td>

2477

2478

</tr>

2479

</table>

2480

<P>

2481

We need to tell the swish.cgi script where to look for the index created in

2482

the previous step. It's also recommended to enter the path to the swish-e

2483

binary, otherwise the swish.cgi script will look for the binary in the

2484

PATH, and that may change when running under the CGI environment.

2485

2486

<P>

2487

Here's the configuration file:

2488

2489

<P>

2490

2491

<table>

2492

<tr>

2493

2494

2495

2496

</td>

2497

2498

<td>

2499

<pre> /usr/lib/cgi-bin# cat .swishcgi.conf

2500

return {

2501

title => 'Search Apache Documentation',

2502

swish_binary => '/usr/local/bin/swish-e',

2503

swish_index => '/home/moseley/web_index/index.swish-e',

2504

}</pre>

2505

</td>

2506

2507

</tr>

2508

</table>

2509

<P>

2510

Now, test the script from the command line as a normal user:

2511

2512

<P>

2513

2514

<table>

2515

<tr>

2516

2517

2518

2519

</td>

2520

2521

<td>

2522

<pre> /usr/lib/cgi-bin# exit

2523

exit</pre>

2524

</td>

2525

2526

</tr>

2527

</table>

2528

<P>

2529

2530

<table>

2531

<tr>

2532

2533

2534

2535

</td>

2536

2537

<td>

2538

<pre> /usr/lib/cgi-bin$ ./swish.cgi | head

2539

Content-Type: text/html; charset=ISO-8859-1</pre>

2540

</td>

2541

2542

</tr>

2543

</table>

2544

<P>

2545

2546

<table>

2547

<tr>

2548

2549

2550

2551

</td>

2552

2553

<td>

2554

2555

<html>

2556

<head>

2557

<title>

2558

Search Apache Documentation

2559

</title>

2560

</head>

2561

2562

</td>

2563

2564

</tr>

2565

</table>

2566

<P>

2567

Notice that the CGI script returns the HTTP header (Content-Type) and the

2568

body of the web page, just like a well behaved CGI scrip should do.

2569

2570

<P>

2571

Now test using the web server (this step depends on the location of your

2572

cgi-bin directory). This example uses the ``GET'' command that is part of

2573

the LWP Perl library, but any web browser can run this test.

2574

2575

<P>

2576

2577

<table>

2578

<tr>

2579

2580

2581

2582

</td>

2583

2584

<td>

2585

<pre> /usr/lib/cgi-bin$ GET <A HREF="http://localhost/cgi-bin/swish.cgi">http://localhost/cgi-bin/swish.cgi</A> | head

2586

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Tranitional//EN">

2587

<html>

2588

<head>

2589

<title>

2590

Search Apache Documentation

2591

</title>

2592

</head>

2593

<body>

2594

2595

</td>

2596

2597

</tr>

2598

</table>

2599

<P>

2600

The script reports errors to stderr, so consult the web server's error log

2601

if problems occur. The message ``Service currently unavailable'' reported

2602

by running swish.cgi typically indicates a configuration error, and the

2603

exact problem will be listed in the web server's error log.

2604

2605

<P>

2606

Detailed instructions on using the <EM>swish.cgi</EM> script and debugging tips can be found by running:

2607

2608

<P>

2609

2610

<table>

2611

<tr>

2612

2613

2614

2615

</td>

2616

2617

<td>

2618

<pre> $ perldoc swish.cgi</pre>

2619

</td>

2620

2621

</tr>

2622

</table>

2623

<P>

2624

while in the cgi-bin directory where swish.cgi was copied.

2625

2626

<P>

2627

The spider program <EM>spider.pl</EM> also has a large number of configuration options.

2628

2629

<P>

2630

Documentation is also available in the directory $prefix/share/doc/swish-e

2631

or at <A HREF="http://swish-e.org.">http://swish-e.org.</A>

2632

2633

<P>

2634

Note: Also check out the search.cgi script found at the same location as

2635

the swish.cgi script. This is more of a skeleton script for those that want

2636

to create a custom search script.

2637

2638

</OL>

2639

<P>

2640

Now you are ready to search.

2641

2642

<P>

2643

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

2644

<HR>

2645

<H1><A NAME="Indexing_Other_Types_of_Documents_Filtering">Indexing Other Types of Documents - Filtering</A></H1>

2646

<P>

2647

Swish-e can only index HTML, XML and text documents. In order to index

2648

other documents such as PDF or MS Word documents you must use a utility to

2649

convert or ``filter'' those documents.

2650

2651

<P>

2652

How documents are filtered with Swish-e has changed over time. This has

2653

resulting in a bit of confusion. It's also a somewhat complex process as

2654

different programs need to communicate with each other.

2655

2656

<P>

2657

You may wish to read the Swish-e FAQ question on filtering before

2658

continuing here.

2659

<A HREF="././SWISH-FAQ.html#How_Do_I_filter_documents_">How Do I filter documents?</A>

2660

2661

2662

2663

<P>

2664

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

2665

<HR>

2666

<H2><A NAME="Filtering_Overview">Filtering Overview</A></H2>

2667

<P>

2668

There's two ways to filter documents with Swish-e. Both are described in

2669

the <A HREF="././SWISH-CONFIG.html">SWISH-CONFIG</A> man page. They are using the FileFilter directive and the SWISH::Filter

2670

perl module.

2671

2672

<P>

2673

The FileFilter directive is a general purpose method of filtering. It

2674

allows running of an external program for each document processed (based on

2675

file extension), and requires. The external programs open an input file,

2676

convert as needed, and write their output to standard output.

2677

2678

<P>

2679

Previous versions of Swish-e (before 2.4.0) used a collection of filter

2680

programs for converting files such as PDF or MS Word documents. The

2681

external programs call other program do to the work of filtering (e.g.

2682

pdftotext to extract the contents from PDF files). Although these filter

2683

programs are still included with the Swish-e distribution as examples, it

2684

is not recommended to use the SWISH::Filter method instead.

2685

2686

<P>

2687

One disadvantage of using FileFilter is that the filter is called once for

2688

every document that needs to be filtered. This can slow down the indexing

2689

process.

2690

2691

<P>

2692

The SWISH::Filter Perl module works very much like the old system and uses

2693

the same helper programs. But, it provides a single interface for filtering

2694

all types of documents. The advantage of SWISH::Filter is that it is built

2695

into the program used for spidering web sites (spider.pl), so all that's

2696

required is installing the filter programs that do the actual work of

2697

filtering (e.g. catdoc, xpdf). (The Windows binary includes some of the

2698

filter programs.)

2699

2700

<P>

2701

But, Swish-e will not use SWISH::Filter by default when using the file

2702

system method of indexing. To use SWISH::Filter when indexing by file

2703

system method (-S fs) you can use a FileFilter directive with the

2704

``swish_filter.pl'' filter (which is just a program that uses

2705

SWISH::Filter), or use the -S prog method of indexing and use the

2706

DirTree.pl program for fetching documents. DirTree.pl is included with the

2707

Swish-e distribution and is designed to work with SWISH::Filter. Using

2708

DirTree.pl will likely be faster way to index since the SWISH::Filter set

2709

of modules do not need to be compiled for every document that needs to be

2710

filtered.

2711

2712

<P>

2713

See the contents of swish_filter.pl and DirTree.pl for specifics on their

2714

use.

2715

2716

<P>

2717

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

2718

<HR>

2719

<H2><A NAME="Filtering_Examples">Filtering Examples</A></H2>

2720

<P>

2721

The FileFilter directive can be used in your config file to convert

2722

documents based on their extension. This is the old way of filtering, but

2723

provides an easy way to add filters to swish-e.

2724

2725

<P>

2726

For example:

2727

2728

<P>

2729

2730

<table>

2731

<tr>

2732

2733

2734

2735

</td>

2736

2737

<td>

2738

<pre> FileFilter .pdf pdftotext "'%p' -"

2739

IndexContents TXT* .pdf</pre>

2740

</td>

2741

2742

</tr>

2743

</table>

2744

<P>

2745

will cause all .pdf files to be filtered through the pdftotext program

2746

(part of the Xpdf package), and to parse the resulting output from

2747

pdftotext with the text (``TXT'') parser.

2748

2749

<P>

2750

The other ways to filter documents is to use a -S prog program and convert

2751

the documents before passing them onto Swish-e.

2752

2753

<P>

2754

For example, spider.pl makes use of the Perl module included with the

2755

Swish-e distribution called SWISH::Filter. SWISH::Filter is passed a

2756

document and the document's content type and then looks for modules and

2757

utilities to convert the document into one of the types that Swish-e can

2758

index.

2759

2760

<P>

2761

Swish-e comes ready to index PDF, MS Word, MP3 ID3 tags, and MS Excel file

2762

types. But these filters need extra modules or tools to do the actual

2763

conversion.

2764

2765

<P>

2766

For example, the Swish-e distribution includes a module called

2767

SWISH::Filter::Pdf2HTML that uses the pdftotext and pdfinfo utilities

2768

provided by the Xpdf package.

2769

2770

<P>

2771

This means that if you are using spider.pl to spider your web site and you

2772

wish to index PDF documents, all that is needed is to install the Xpdf

2773

package and Swish-e (with the help of spider.pl) will begin indexing your

2774

PDF files.

2775

2776

<P>

2777

Ok, so what does all that mean? For a very simple site you should be able

2778

to run this:

2779

2780

<P>

2781

2782

<table>

2783

<tr>

2784

2785

2786

2787

</td>

2788

2789

<td>

2790

<pre> $ /usr/local/lib/swish-e/spider.pl default <A HREF="http://localhost/">http://localhost/</A> | swish-e -S prog -i stdin</pre>

2791

</td>

2792

2793

</tr>

2794

</table>

2795

<P>

2796

which is running the spider with default spider settings, indexing the Web

2797

server on localhost and piping its output into Swish-e using default

2798

indexing settings. Documents will be filtered automatically if you have the

2799

required helper applications installed.

2800

2801

<P>

2802

Most people will not want to just use the default settings (for one thing

2803

the spider will take a while because its default is to delay a few seconds

2804

between every request). So, read the documentation for spider.pl to learn

2805

how to use a spider config file. And also read <A HREF="././SWISH-CONFIG.html">SWISH-CONFIG</A> to learn about what configuration options can be used with Swish-e.

2806

2807

<P>

2808

The SWISH::Filter documentation provides more details on filtering and

2809

hints for debugging problems when filtering.

2810

2811

<P>

2812

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

2813

<HR>

2814

<H1><A NAME="Document_Info">Document Info</A></H1>

2815

<P>

2816

$Id: INSTALL.pod,v 1.36.2.2 2003/12/18 00:10:06 whmoseley Exp $

2817

2818

<P>

2819

2820

2821

[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]

2822

<HR>

2823

2824

2825

2826

<p>

2827

2828

<a href="./README.html">Prev</a> |

2829

<a href="./index.html">Contents</a> |

2830

2831

</div>

2832

<p>

2833

2834

2835

2836

2837

2838

2839

<BR>SWISH-E is distributed with <B>no warranty</B> under the terms of the

2840

<A HREF="http://www.fsf.org/copyleft/gpl.html">GNU Public License</A>,<BR>

2841

Free Software Foundation, Inc.,

2842

59 Temple Place - Suite 330, Boston, MA 02111-1307, USA<BR>

2843

Public questions may be posted to

2844

the <A HREF="http://swish-e.org/Discussion/">SWISH-E Discussion</A>.

2845

</div>

2846

2847

</body>

2848

</html>

Older »