1
<?xml version="1.0" encoding="UTF-8"?>
3
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
5
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
8
<meta http-equiv="Content-Language" content="en" />
9
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
10
<meta http-equiv="Content-Style-Type" content="text/css" />
11
<meta name="author" content="Mikio Hirabayashi" />
12
<meta name="keywords" content="Hyper Estraier, Estraier, full-text search, web crawler" />
13
<meta name="description" content="Crawler Guide of Hyper Estraier" />
14
<link rel="contents" href="./" />
15
<link rel="alternate" href="cguide-ja.html" hreflang="ja" title="the Japanese version" />
16
<link rel="stylesheet" href="common.css" />
17
<link rel="icon" href="icon16.png" />
18
<link rev="made" href="mailto:mikio@users.sourceforge.net" />
19
<title>Crawler Guide of Hyper Estraier Version 1</title>
24
<h1>Crawler Guide</h1>
26
<div class="note">Copyright (C) 2004-2006 Mikio Hirabayashi</div>
27
<div class="note">Last Update: Mon, 11 Sep 2006 21:41:45 +0900</div>
28
<div class="navi">[<span class="void">English</span>/<a href="cguide-ja.html" hreflang="ja">Japanese</a>] [<a href="index.html">HOME</a>]</div>
32
<h2 id="tableofcontents">Table of Contents</h2>
35
<li><a href="#introduction">Introduction</a></li>
36
<li><a href="#tutorial">Tutorial</a></li>
37
<li><a href="#estwaver">Crawler Command</a></li>
42
<h2 id="introduction">Introduction</h2>
44
<p>This guide describes usage of Hyper Estraier's web crawler. If you haven't read <a href="uguide-en.html">user's guide</a> and <a href="nguide-en.html">P2P guide</a> yet, now is a good moment to do so.</p>
46
<p><code>estcmd</code> can index files on local file system only. Though files on remote hosts can be indexed by using NFS or SMB remote mount mechanism, unspecified number of web sites on Internet can not be mounted by them. Though such web crawlers as <code>wget</code> can do prefetch of those files, it involves high overhead and wastes much disk space.</p>
48
<p>The command <code>estwaver</code> is useful to crawl arbitrary web sites and to index their documents directly. <code>estwaver</code> is so intelligent that it supports not only depth first order and width first but also similarity oriented order. It crawls documents similar to specified seed documents preferentially.</p>
52
<h2 id="tutorial">Tutorial</h2>
54
<p>First step is creation of the crawler root directory which contains a configuration file and some databases. Following command will create <code>casket</code>, the crawler root directory:</p>
56
<pre>estwaver init casket
59
<p>By default, the configuration is to start crawling at the project page of Hyper Estraier. Let's try it as it is:</p>
61
<pre>estwaver crawl casket
64
<p>Then, documents are fetched one after another and they are indexed into the index. To stop the operation, you can press <code>Ctrl-C</code> on terminal.</p>
66
<p>When the operation finishes, there is a directory <code>_index</code> in the crawler root directory. It is an index which can be treated with <code>estcmd</code> and so on. Let's try to search the index as with the following command:</p>
68
<pre>estcmd search -vs casket/_index "hyper estraier"
71
<p>If you want to resume the crawling operation, perform <code>estwaver crawl</code> again.</p>
75
<h2 id="estwaver">Crawler Command</h2>
77
<p>This section describes specification of <code>estwaver</code>, whose purpose is to index documents on the Web.</p>
79
<h3>Synopsis and Description</h3>
81
<p><code>estwaver</code> is an aggregation of sub commands. The name of a sub command is specified by the first argument. Other arguments are parsed according to each sub command. The argument <code>rootdir</code> specifies the crawler root directory which contains configuration file and so on.</p>
84
<dt><kbd>estwaver init <var>rootdir</var></kbd></dt>
85
<dd>Create the crawler root directory.</dd>
86
<dd>If -xs is specified, the index is tuned to register less than 50000 documents.</dd>
87
<dd>If -xl is specified, the index is tuned to register more than 300000 documents.</dd>
88
<dd>If -xh is specified, the index is tuned to register more than 1000000 documents.</dd>
92
<dt><kbd>estwaver crawl [-restart|-revisit|-revcont] <var>rootdir</var></kbd></dt>
93
<dd>Start crawling.</dd>
94
<dd>If -restart is specified, crawling is restarted from the seed documents.</dd>
95
<dd>If -revisit is specified, collected documents are revisited.</dd>
96
<dd>If -revcont is specified, collected documents are revisited and then crawling is continued.</dd>
100
<dt><kbd>estwaver unittest <var>rootdir</var></kbd></dt>
101
<dd>Perform unit tests.</dd>
105
<dt><kbd>estwaver fetch [-proxy <var>host</var> <var>port</var>] [-tout <var>num</var>] [-il <var>lang</var>] <var>url</var></kbd></dt>
106
<dd>Fetch a document.</dd>
107
<dd><var>url</var> specifies the URL of a document.</dd>
108
<dd>-proxy specifies the host name and the port number of the proxy server.</dd>
109
<dd>-tout specifies timeout in seconds.</dd>
110
<dd>-il specifies the preferred language. By default, it is English.</dd>
113
<p>All sub commands return 0 if the operation is success, else return 1. A running crawler finishes with closing the database when it catches the signal 1 (SIGHUP), 2 (SIGINT), 3 (SIGQUIT), or 15 (SIGTERM).</p>
115
<p>When crawling finishes, there is a directory <code>_index</code> in the crawler root directory. It is an index available by <code>estcmd</code> and so on.</p>
117
<h3>Constitution of the Crawler Root Directory</h3>
119
<p>The crawler root directory contains the following files and directories.</p>
122
<li><kbd>_conf</kbd> : configuration file.</li>
123
<li><kbd>_log</kbd> : log file.</li>
124
<li><kbd>_meta</kbd> : database file for meta data.</li>
125
<li><kbd>_queue</kbd> : priority queue of URLs to be crawled.</li>
126
<li><kbd>_trace/</kbd> : tracking records of crawled URLs.</li>
127
<li><kbd>_index/</kbd> : index directory.</li>
128
<li><kbd>_tmp/</kbd> : directory for temporary files.</li>
131
<h3>Configuration File</h3>
133
<p>The configuration file is composed of lines and the name of an variable and the value separated by "<code>:</code>" are in each line. By default, the following configuration is there.</p>
135
<pre>seed: 1.5|http://hyperestraier.sourceforge.net/uguide-en.html
136
seed: 1.0|http://hyperestraier.sourceforge.net/pguide-en.html
137
seed: 1.0|http://hyperestraier.sourceforge.net/nguide-en.html
138
seed: 0.0|http://qdbm.sourceforge.net/
149
replace: ^http://127.0.0.1/{{!}}http://localhost/
151
denyrx: \.(css|js|csv|tsv|log|md5|crc|conf|ini|inf|lnk|sys|tmp|bak)$
152
denyrx: \.(zip|tar|tgz|gz|bz2|tbz2|z|lha|lzh)(\?.*)?$
153
denyrx: ://(localhost|[a-z]*\.localdomain|127\.0\.0\.1)/
154
noidxrx: /\?[a-z]=[a-z](;|$)
155
urlrule: \.est${{!}}text/x-estraier-draft
156
urlrule: \.(eml|mime|mht|mhtml)${{!}}message/rfc822
157
typerule: ^text/x-estraier-draft${{!}}[DRAFT]
158
typerule: ^text/plain${{!}}[TEXT]
159
typerule: ^(text/html|application/xhtml+xml)${{!}}[HTML]
160
typerule: ^message/rfc822${{!}}[MIME]
170
#nodeserv: 1|http://admin:admin@localhost:1978/node/node1
171
#nodeserv: 2|http://admin:admin@localhost:1978/node/node2
172
#nodeserv: 3|http://admin:admin@localhost:1978/node/node3
180
<p>Meaning of each variable is the following.</p>
183
<li><kbd>seed</kbd> : specifies the weight and the URL of a seed document, separated by "<code>|</code>". This can be more than once.</li>
184
<li><kbd>proxyhost</kbd> : specifies the host name of the proxy server.</li>
185
<li><kbd>proxyport</kbd> : specifies the port number of the proxy server.</li>
186
<li><kbd>interval</kbd> : specifies waiting interval of each request (in milliseconds).</li>
187
<li><kbd>timeout</kbd> : specifies timeout of each request (in seconds).</li>
188
<li><kbd>strategy</kbd> : specifies strategy of crawling path (0:balanced, 1:similarity, 2:depth, 3:width, 4:random).</li>
189
<li><kbd>inherit</kbd> : specifies inheritance ratio of similarity from the parent.</li>
190
<li><kbd>seeddepth</kbd> : specifies maximum depth of seed documents.</li>
191
<li><kbd>maxdepth</kbd> : specifies maximum depth of recursion.</li>
192
<li><kbd>masscheck</kbd> : specifies standard value for checking mass sites.</li>
193
<li><kbd>queuesize</kbd> : specifies maximum number of records of the priority queue.</li>
194
<li><kbd>replace</kbd> : specifies regular expressions and replacement strings to normalize URLs. This can be more than once.</li>
195
<li><kbd>allowrx</kbd> : specifies allowing regular expressions of URLs to be visited. This can be more than once.</li>
196
<li><kbd>denyrx</kbd> : specifies denying regular expressions of URLs to be visited. This can be more than once.</li>
197
<li><kbd>noidxrx</kbd> : specifies denying regular expressions of URLs to be indexed. This can be more than once.</li>
198
<li><kbd>urlrule</kbd> : specifies URL rules (regular expressions and media types). This can be more than once.</li>
199
<li><kbd>typerule</kbd> : specifies media type rules (regular expressions and filter commands). This can be more than once.</li>
200
<li><kbd>language</kbd> : specifies the preferred language (0:English, 1:Japanese, 2:Chinese, 3:Korean, 4:misc).</li>
201
<li><kbd>textlimit</kbd> : specifies text size limitation (in kilobytes).</li>
202
<li><kbd>seedkeynum</kbd> : specifies the total number of keywords for seed documents.</li>
203
<li><kbd>savekeynum</kbd> : specifies the number of keywords saved for each document.</li>
204
<li><kbd>threadnum</kbd> : specifies the number of threads running in parallel.</li>
205
<li><kbd>docnum</kbd> : specifies the number of documents to collect.</li>
206
<li><kbd>period</kbd> : specifies running time period (in s:seconds, m:minutes, h:hours, d:days).</li>
207
<li><kbd>revisit</kbd> : specifies revisit span (in s:seconds, m:minutes, h:hours, d:days).</li>
208
<li><kbd>cachesize</kbd> : specifies the maximum size of the index cache (in megabytes).</li>
209
<li><kbd>nodeserv</kbd> : specifies the ID number and the URL of a node server, separated by "<code>|</code>". This can be more than once.</li>
210
<li><kbd>logfile</kbd> : specifies the path of the log file (relative path or absolute path).</li>
211
<li><kbd>loglevel</kbd> : specifies logging level (1:debug, 2:information, 3:warning, 4:error, 5:none).</li>
212
<li><kbd>draftdir</kbd> : specifies the path of the draft directory (relative path or absolute path).</li>
213
<li><kbd>entitydir</kbd> : specifies the path of the entity directory (relative path or absolute path).</li>
214
<li><kbd>postproc</kbd> : the postprocessor for retrieved files.</li>
217
<p><code>allowrx</code>, <code>denyrx</code>, and <code>noidxrx</code> are evaluated in the order of description. Alphabetical characters are case-insensitive.</p>
219
<p>Arbitrary filter commands can be specified with <code>typerule</code>. The interface of filter command is same as with <code>-fx</code> option of <code>estcmd gather</code>. For example, the following specifies to process PDF documents.</p>
221
<pre>typerule: ^application/pdf${{!}H@/usr/local/share/hyperestraier/filter/estfxpdftohtml
added
removed
doc/cguide-en.html
