1
<?xml version="1.0" encoding="UTF-8"?>
3
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
5
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
8
<meta http-equiv="Content-Language" content="en" />
9
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
10
<meta http-equiv="Content-Style-Type" content="text/css" />
11
<meta name="author" content="Mikio Hirabayashi" />
12
<meta name="keywords" content="Hyper Estraier, Estraier, full-text search, API" />
13
<meta name="description" content="Introduction of Hyper Estraier" />
14
<link rel="contents" href="./" />
15
<link rel="alternate" href="intro-en.html" hreflang="en" title="the English version" />
16
<link rel="stylesheet" href="common.css" />
17
<link rel="icon" href="icon16.png" />
18
<link rev="made" href="mailto:mikio@users.sourceforge.net" />
19
<title>Introduction of Hyper Estraier Version 1</title>
26
<div class="note">Copyright (C) 2004-2005 Mikio Hirabayashi</div>
27
<div class="note">Last Update: Tue, 01 Nov 2005 04:27:26 +0900</div>
28
<div class="navi">[<a href="intro-ja.html" hreflang="ja">Japanese</a>] [<a href="index.html">HOME</a>]</div>
32
<h2 id="tableofcontents">Table of Contents</h2>
35
<li><a href="#introduction">Introduction</a></li>
36
<li><a href="#installation">Installation</a></li>
37
<li><a href="#deployment">Deployment</a></li>
38
<li><a href="#complement">Complement</a></li>
43
<h2 id="introduction">Introduction</h2>
45
<p>Hyper Estraier is a full-text search system. You can search lots of documents for some documents including specified words. If you run a web site, it is useful as your own search engine for pages in your site. Also, it is useful as search utilities of mail boxes and file servers.</p>
47
<p>The characteristic of Hyper Estraier is the following.</p>
50
<li>High performance of search</li>
51
<li>High scalability of target documents</li>
52
<li>Perfect recall ratio by N-gram method</li>
53
<li>Phrase search, regular expressions, attribute search, and similarity search</li>
54
<li>Multilingualism with Unicode</li>
55
<li>Independent of file format and repository</li>
56
<li>Simple and powerful API</li>
57
<li>Supporting P2P architecture</li>
60
<p>To begin with, please try <a href="http://rbbs.sourceforge.jp/cgi-bin/estdemo-en/estseek.cgi?clip=7">the demonstration site of Hyper Estraier</a> so that you will grasp overview of Hyper Estraier.</p>
62
<p>Hyper Estraier has two aspects. One is as a library to construct a full-text search system. That is, API (application programming interface) is provided for programmers. It enables for you to embed advanced functions of full-text search into your applications.</p>
64
<p>The other is as an application of the API described above. A command and a CGI script are provided. Using them, you can construct a typical full-text search system without any programming.</p>
66
<p>This document describes how to construct a full-text search system with the command and the CGI script, seeing a subject matter of a search system of a web site. Let's start with learning of the command and then step to the API.</p>
70
<h2 id="installation">Installation</h2>
72
<p>This section describes how to install Hyper Estraier with the source package. As for a binary package, see its installation manual.</p>
76
<p>Hyper Estraier is available on UNIX-like systems and Windows NT series. At least, the following environment are supported.</p>
79
<li>Linux 2.2 and later (IA32/IA64/AMD64/SPARC/Alpha)</li>
80
<li>FreeBSD 4.9 and later (IA32/Alpha)</li>
81
<li>Solaris 8 and later (IA32/SPARC)</li>
82
<li>Mac OS X 10.2 and later (PowerPC)</li>
83
<li>HP-UX 11.11 and later (IA64/PA-RISC)</li>
84
<li>Windows 2000 and later (IA32)</li>
87
<p><code>gcc</code> 2.95 or later and <code>make</code> are required to install Hyper Estraier with the source package. They are installed by default on Linux, FreeBSD and so on.</p>
89
<p>As Hyper Estraier depends on the following libraries, install them beforehand.</p>
92
<li><a href="http://www.gnu.org/software/libiconv/">libiconv</a> : for conversion of character encodings. 1.9.1 or later is suggested.</li>
93
<li><a href="http://www.gzip.org/zlib/">zlib</a> : for loss-less data compression. 1.2.1 or later is suggested.</li>
94
<li><a href="http://qdbm.sourceforge.net">QDBM</a> : for embedded database. 1.8.33 or later is required.</li>
97
<p>As well, it is suggested to build QDBM with enabling zlib (<code>./configure --enable-zlib</code>) so that the index of Hyper Estraier becomes smaller. Note that QDBM 1.8.32 or earlier is not supported.</p>
101
<p>When an archive file of Hyper Estraier is extracted, change the current working directory to the generated directory and perform installation.</p>
103
<p>Run the configuration script.</p>
108
<p>Build programs.</p>
113
<p>Perform self-diagnostic test.</p>
118
<p>Install programs. This operation must be carried out by the <code>root</code> user.</p>
125
<p>When a series of work finishes, the following files will be installed.</p>
127
<pre>/usr/local/include/estraier.h
128
/usr/local/include/estmtdb.h
129
/usr/local/lib/libestraier.a
130
/usr/local/lib/libestraier.so.6.6.0
131
/usr/local/lib/libestraier.so.6
132
/usr/local/lib/libestraier.so
133
/usr/local/lib/pkgconfig/hyperestraier.pc
134
/usr/local/bin/estcmd
135
/usr/local/bin/estmttest
136
/usr/local/bin/estmaster
137
/usr/local/bin/estcall
138
/usr/local/bin/estwaver
139
/usr/local/bin/estload
140
/usr/local/bin/estconfig
141
/usr/local/bin/estwolefind
142
/usr/local/libexec/estseek.cgi
143
/usr/local/share/hyperestraier/estseek.conf
144
/usr/local/share/hyperestraier/estseek.tmpl
145
/usr/local/share/hyperestraier/estseek.top
146
/usr/local/share/hyperestraier/estresult.dtd
147
/usr/local/share/hyperestraier/estraier.idl
148
/usr/local/share/hyperestraier/locale/...
149
/usr/local/share/hyperestraier/filter/...
150
/usr/local/share/hyperestraier/increm/...
151
/usr/local/share/hyperestraier/doc/...
152
/usr/local/man/man1/...
153
/usr/local/man/man3/...
156
<h3>Mac OS X, HP-UX, and Windows</h3>
158
<p>On Mac OS X, perform `<code>make mac</code>' instead of `<code>make</code>', and `<code>make check-mac</code>' instead of `<code>make check</code>', and `<code>make install-mac</code>' instead of `<code>make install</code>'. As well, `<code>libqdbm.dylib</code>' and so on are created instead of `<code>libestraier.so</code>' and so on.</p>
160
<p>On HP-UX, perform `<code>make hpux</code>' instead of `<code>make</code>', and `<code>make check-hpux</code>' instead of `<code>make check</code>', and `<code>make install-hpux</code>' instead of `<code>make install</code>'. As well, `<code>libqdbm.sl</code>' is created instead of `<code>libestraier.so</code>' and so on.</p>
162
<p>On Windows, the Cygwin environment is required for building. Moreover, MinGW versions of zlib, libiconv, QDBM, and Pthreads are required. On that basis, perform `<code>make win</code>'. No installation command is provided for Windows.</p>
166
<h2 id="deployment">Deployment</h2>
168
<p>This section describes how to create an index, and deploy the CGI script.</p>
170
<h3>Administration Command</h3>
172
<p>A database called inverted index is used in order to search for documents quickly. That is, you should make the index containing target documents before you search some of them.</p>
174
<p><code>estcmd</code> is provided to administrate indexes. <code>estcmd</code> handles each file on the file system of the local host, as each document. <code>estcmd</code> can register documents to the index and remove them from the index. Moreover, <code>estcmd</code> can gather documents under a directory and register them as a job lot. Supported file formats are plain-text, HTML, and e-mail (MIME).</p>
176
<p>As other formats are also supported by using filters, the method is mentioned later.</p>
180
<p>It is presupposed that you run a web site and its contents are under `<code>/home/www/public_html</code>'. Then, let's register them into the index as `<code>/home/www/casket</code>'.</p>
183
estcmd gather -sd casket /home/www/public_html
186
<p>Files under `<code>/home/www/public_html</code>' are gathered and registered into a new index named as `<code>casket</code>'. That's all for indexing.</p>
188
<h3>Deployment of the CGI Script</h3>
190
<p>It is presupposed that the URL of a directory for CGI scripts is `<code>http://www.estraier.ad.jp/cgi-bin/</code>' and its local path is `<code>/home/www/cgi-bin</code>'. Then, let's deploy requisite files into there.</p>
192
<pre>cd /home/www/cgi-bin/
193
cp /usr/local/libexec/estseek.cgi .
194
cp /usr/local/share/hyperestraier/estseek.* .
197
<p>`<code>/usr/local/libexec/estseek.cgi</code>', `<code>estseek.(conf|tmpl|top)</code>' in `<code>/usr/local/share/hyperestraier/</code>' are copied into `<code>/home/www/cgi-bin/</code>'. <code>estseek.cgi</code> is the CGI script. <code>estseek.conf</code> is the configuration file. <code>estseek.tmpl</code> is the template file. <code>estseek.top</code> is for the message of the top page.</p>
199
<p>Open <code>estseek.conf</code> with a text editor and modify it. Most items are not needed to be modified, except for `<code>indexname</code>' and `<code>replace</code>'. Do as the following.</p>
201
<pre>indexname: /home/www/casket
203
replace: file:///home/www/public_html/{{!}}http://www.estraier.ad.jp/
207
<p>`<code>indexname</code>' specifies the path of the index. `<code>replace</code>' specifies regular expressions and replacement strings to convert the local URL of each document for the web server.</p>
209
<h3>Let's Try It</h3>
211
<p>All set? Let's access the URL `<code>http://www.estraier.ad.jp/cgi-bin/estseek.cgi</code>' with your favorite web browser. How to use is described on the page.</p>
213
<h3>Updating the Index</h3>
215
<p>When some documents in your site are modified or new documents are added, please update the index at regular intervals. Though it is okay to delete the index and remake it, incremental registration is useful.</p>
217
<p>The `<code>-sd</code>' option added when indexing is to record modification time of each document. And it is useful for incremental registration. Let's perform the following command.</p>
220
estcmd gather -cl -sd -cm casket /home/www/public_html
223
<p>The option `<code>-cm</code>' is to ignore files which are not modified. The option `<code>-cl</code>' is to clean up data of overwritten documents.</p>
225
<h3>Reflection of Deleted Documents</h3>
227
<p>If some documents in your site are deleted, please reflect them to the index. Let's perform the following command.</p>
230
estcmd purge -cl casket
233
<p>All records in the index are scanned and records of deleted documents are removed. The option `<code>-cl</code>' is to clean up data of overwritten documents.</p>
235
<h3>Optimization</h3>
237
<p>Iteration of `<code>gather</code>' and `<code>purge</code>' makes the index fat gradually. Optimization is to eliminate the dispensable regions and keeps the index small.</p>
240
estcmd optimize casket
243
<p>If `<code>gather</code>' or `<code>purge</code>' is performed without the `<code>-cl</code>' option, records of deleted documents are not deleted though deletion marks was applied to them. `<code>optimize</code>' is useful to delete such void regions.</p>
245
<h3>Automated Administration</h3>
247
<p>`<code>cron</code>' enables you to automate operations for administration. Register the following script to `<code>crontab</code>'.</p>
249
<pre>/usr/local/bin/estcmd gather -cl -sd -cm /home/www/casket /home/www/public_html
250
/usr/local/bin/estcmd purge -cl /home/www/casket
253
<h3>For more detail</h3>
255
<p>Detail information of the command and the CGI script is described in <a href="uguide-en.html">the user's guide</a>. Moreover, for information of the API, see <a href="pguide-en.html">the programming guide</a>.</p>
259
<h2 id="complement">Complement</h2>
261
<p>This section describes how to contact the author and the license of Hyper Estraier.</p>
265
<p>Hyper Estraier was written and is maintained by <a href="http://qdbm.sourceforge.net/mikio/">Mikio Hirabayashi</a>. You can contact the author by e-mail to `<code>mikio@users.sourceforge.net</code>'. However, as for topics which can be shared among other users, please send it to one of the mailing lists. To join the mailing list, refer to `<code><a href="http://lists.sourceforge.net/lists/listinfo/hyperestraier-users">http://lists.sourceforge.net/lists/listinfo/hyperestraier-users</a></code>'.</p>
269
<p>Hyper Estraier is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License or any later version.</p>
271
<p>Hyper Estraier is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.</p>
273
<p>You should have received a copy of the GNU Lesser General Public License along with Hyper Estraier (See the file `<code>COPYING</code>'); if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.</p>
275
<h3>Acknowledgment</h3>
277
<p>Hyper Estraier was developed under management by Fumitoshi Ukai and supports by Exploratory Software Project of <a href="http://www.ipa.go.jp/">Information-technology Promotion Agency, Japan (IPA)</a>.</p>
added
removed
doc/intro-en.html~
