« back to all changes in this revision
Viewing changes to lib/Lingua/Stem.pm
- browse files at revision 3
- compare with another revision
- download diff
- download tarball
- view history from revision 3
- Committer: Bazaar Package Importer
- Author(s): Dominic Hargreaves
- Date: 2006-08-23 20:10:09 UTC
- mfrom: (1.1.1 upstream)
- Revision ID: james.westby@ubuntu.com-20060823201009-a35rbiz7d292nlko
Tags: 0.82-1
* New upstream release
* Update Standards-Version (no changes)
* Update Standards-Version (no changes)
- files added:
- Artistic_License.txt
- files removed:
- t/import-all.t
- files modified:
- Changes
- examples/voc.txt *
- lib/Lingua/Stem/Da.pm *
- lib/Lingua/Stem/De.pm *
- lib/Lingua/Stem/EnBroken.pm *
- lib/Lingua/Stem/Gl.pm *
- lib/Lingua/Stem/No.pm *
- lib/Lingua/Stem/Pt.pm *
- lib/Lingua/Stem/Sv.pm *
added
removed
lib/Lingua/Stem.pm
2
2
3
3
# $RCSfile: Stem.pm,v $ $Revision: 1.2 $ $Date: 1999/06/16 17:45:28 $ $Author: snowhare $
4
4
5
#######################################################################
6
# Initial POD Documentation
7
#######################################################################
8
9
=head1 NAME
10
11
Lingua::Stem - Stemming of words
12
13
=head1 SYNOPSIS
14
15
use Lingua::Stem qw(stem);
16
my $stemmmed_words_anon_array = stem(@words);
17
18
or for the OO inclined,
19
20
use Lingua::Stem;
21
my $stemmer = Lingua::Stem->new(-locale => 'EN-UK');
22
$stemmer->stem_caching({ -level => 2 });
23
my $stemmmed_words_anon_array = $stemmer->stem(@words);
24
25
=head1 DESCRIPTION
26
27
This routine applies stemming algorithms to its parameters,
28
returning the stemmed words as appropriate to the selected
29
locale.
30
31
You can import some or all of the class methods.
32
33
use Lingua::Stem qw (stem clear_stem_cache stem_caching
34
add_exceptions delete_exceptions
35
get_exceptions set_locale get_locale
36
:all :locale :exceptions :stem :caching);
37
38
:all - imports stem add_exceptions delete_exceptions get_exceptions
39
set_locale get_locale
40
:stem - imports stem
41
:caching - imports stem_caching clear_stem_cache
42
:locale - imports set_locale get_locale
43
:exceptions - imports add_exceptions delete_exceptions get_exceptions
44
45
Currently supported locales are:
46
47
DA - Danish
48
DE - German
49
EN - English (also EN-US and EN-UK)
50
FR - French
51
GL - Galician
52
IT - Italian
53
NO - Norwegian
54
PT - Portuguese
55
RU - Russian (also RU-RU and RU-RU.KOI8-R)
56
SV - Swedish
57
58
If you have the memory and lots of stemming to do,
59
I B<strongly> suggest using cache level 2 and processing
60
lists in 'big chunks' (long lists) for best performance.
61
62
Some benchmarks using Lingua::Stem 0.80 and Lingua::Stem::Snowball 0.7
63
give an idea of how batching and caching affect performance. The dataset was
64
3000 randomly selected words from the snowball english voc.txt list repeated
65
100 times (300000 words total, with 3000 unique words). The tests were performed
66
on a Linux machine with a 1.7 Ghz Athlon processor running Perl 5.8.5. There
67
are no tests listed for Lingua::Stem::Snowball using caching because it doesn't have
68
a caching mode.
69
70
Lingua::Stem::Snowball, one word at a time, no caching: 26741 words/second
71
Lingua::Stem::Snowball, 3000 word batches, no caching: 169710 words/second
72
Lingua::Stem::Snowball, one batch, no caching: 155797 words/second
73
74
Lingua::Stem, one word at a time, no caching: 13509 words/second
75
Lingua::Stem, 3000 word batches, no caching: 32765 words/second
76
Lingua::Stem, one batch, no caching: 34847 words/second
77
78
Lingua::Stem, one word at a time, cache level 2: 25736 words/second
79
Lingua::Stem, 3000 word batches, cache level 2: 194384 words/second
80
Lingua::Stem, one batch, cache level 2: 216602 words/second
81
82
=head1 CHANGES
83
84
0.81 2004.07.26 - Minor documentation tweak. No functional change.
85
86
0.80 2004.07.25 - Added 'RU', 'RU_RU', 'RU_RU.KOI-8' locale.
87
Added support for Lingua::Stem::Ru to
88
Makefile.PL and autoloader.
89
90
Added documentation stressing use of caching
91
and batches for performance. Added support
92
for '_' as a seperator in the locale strings.
93
Added example benchmark script. Expanded copyright
94
credits.
95
96
0.70 2004.04.26 - Added FR locale and documentation fixes
97
to Lingua::Stem::Gl
98
99
0.61 2003.09.28 - Documentation fixes. No functional changes.
100
101
0.60 2003.04.05 - Added more locales by wrappering various stemming
102
implementations. Documented currently supported
103
list of locales.
104
105
0.50 2000.09.14 - Fixed major implementation error. Starting with
106
version 0.30 I forgot to include rulesets 2,3 and 4
107
for Porter's algorithm. The resulting stemming results
108
were very poor. Thanks go to <csyap@netfision.com>
109
for bringing the problem to my attention.
110
111
Unfortunately, the fix inherently generates *different*
112
stemming results than 0.30 and 0.40 did. If you
113
need identically broken output - use locale 'en-broken'.
114
115
0.40 2000.08.25 - Added stem caching support as an option. This
116
can provide a large speedup to the operation
117
of the stemmer. Caching is default turned off
118
to maximize compatibility with previous versions.
119
120
0.30 1999.06.24 - Replaced core of 'En' stemmers with code from
121
Jim Richardson <jimr@maths.usyd.edu.au>
122
Aliased 'en-us' and 'en-uk' to 'en'
123
Fixed 'SYNOPSIS' to correct return value
124
type for stemmed words (SYNOPIS error spotted
125
by <Arved_37@chebucto.ns.ca>)
126
127
0.20 1999.06.15 - Changed to '.pm' module, moved into Lingua:: namespace,
128
added OO interface, optionalized the export of routines
129
into the caller's namespace, added named parameter
130
initialization, stemming exceptions, autoloaded
131
locale support and isolated case flattening to
132
localized stemmers prevent i18n problems later.
133
134
Input and output text are assumed to be in UTF8
135
encoding (no operational impact right now, but
136
will be important when extending the module to
137
non-English).
138
139
=cut
140
141
#######################################################################
142
# Initialization
143
#######################################################################
144
145
5
use strict;
146
use Exporter;
147
use Carp;
6
require Exporter;
148
7
use Lingua::Stem::AutoLoader;
149
use vars qw (@ISA @EXPORT_OK %EXPORT_TAGS @EXPORT $VERSION);
150
8
151
9
BEGIN {
152
$VERSION = '0.81';
153
@ISA = qw (Exporter);
154
@EXPORT = ();
155
@EXPORT_OK = qw (stem clear_stem_cache stem_caching add_exceptions delete_exceptions get_exceptions set_locale get_locale);
156
%EXPORT_TAGS = ( 'all' => [qw (stem stem_caching clear_stem_cache add_exceptions delete_exceptions get_exceptions set_locale get_locale)],
10
$Lingua::Stem::VERSION = '0.82';
11
@Lingua::Stem::ISA = qw (Exporter);
12
@Lingua::Stem::EXPORT = ();
13
@Lingua::Stem::EXPORT_OK = qw (stem stem_in_place clear_stem_cache stem_caching add_exceptions delete_exceptions get_exceptions set_locale get_locale);
14
%Lingua::Stem::EXPORT_TAGS = ( 'all' => [qw (stem stem_in_place stem_caching clear_stem_cache add_exceptions delete_exceptions get_exceptions set_locale get_locale)],
157
15
'stem' => [qw (stem)],
16
'stem_in_place' => [qw (stem_in_place)],
158
17
'caching' => [qw (stem_caching clear_stem_cache)],
159
18
'locale' => [qw (set_locale get_locale)],
160
19
'exceptions' => [qw (add_exceptions delete_exceptions get_exceptions)],
162
21
}
163
22
164
23
my $defaults = {
165
-locale => 'en',
166
-stemmer => \&Lingua::Stem::En::stem,
167
-stem_caching => \&Lingua::Stem::En::stem_caching,
168
-clear_stem_cache => \&Lingua::Stem::En::clear_stem_cache,
169
-exceptions => {},
24
-locale => 'en',
25
-stemmer => \&Lingua::Stem::En::stem,
26
-stem_in_place => \&Lingua::Stem::En::stem,
27
-stem_caching => \&Lingua::Stem::En::stem_caching,
28
-clear_stem_cache => \&Lingua::Stem::En::clear_stem_cache,
29
-exceptions => {},
170
30
-known_locales => {
171
31
'da' => { -stemmer => \&Lingua::Stem::Da::stem,
172
32
-stem_caching => \&Lingua::Stem::Da::stem_caching,
173
33
-clear_stem_cache => \&Lingua::Stem::Da::clear_stem_cache,
34
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'da' locale"); },
174
35
},
175
36
'de' => { -stemmer => \&Lingua::Stem::De::stem,
176
37
-stem_caching => \&Lingua::Stem::De::stem_caching,
177
38
-clear_stem_cache => \&Lingua::Stem::De::clear_stem_cache,
39
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'de' locale"); },
178
40
},
179
41
'en' => { -stemmer => \&Lingua::Stem::En::stem,
180
42
-stem_caching => \&Lingua::Stem::En::stem_caching,
181
43
-clear_stem_cache => \&Lingua::Stem::En::clear_stem_cache,
44
-stem_in_place => \&Lingua::Stem::En::stem,
182
45
},
183
46
'en_us' => { -stemmer => \&Lingua::Stem::En::stem,
184
47
-stem_caching => \&Lingua::Stem::En::stem_caching,
185
48
-clear_stem_cache => \&Lingua::Stem::En::clear_stem_cache,
49
-stem_in_place => \&Lingua::Stem::En::stem,
186
50
},
187
51
'en-us' => { -stemmer => \&Lingua::Stem::En::stem,
188
52
-stem_caching => \&Lingua::Stem::En::stem_caching,
189
53
-clear_stem_cache => \&Lingua::Stem::En::clear_stem_cache,
54
-stem_in_place => \&Lingua::Stem::En::stem,
190
55
},
191
56
'en_uk' => { -stemmer => \&Lingua::Stem::En::stem,
192
57
-stem_caching => \&Lingua::Stem::En::stem_caching,
193
58
-clear_stem_cache => \&Lingua::Stem::En::clear_stem_cache,
59
-stem_in_place => \&Lingua::Stem::En::stem,
194
60
},
195
61
'en-uk' => { -stemmer => \&Lingua::Stem::En::stem,
196
62
-stem_caching => \&Lingua::Stem::En::stem_caching,
197
63
-clear_stem_cache => \&Lingua::Stem::En::clear_stem_cache,
64
-stem_in_place => \&Lingua::Stem::En::stem,
198
65
},
199
66
'en-broken' => { -stemmer => \&Lingua::Stem::En_Broken::stem,
200
67
-stem_caching => \&Lingua::Stem::En_Broken::stem_caching,
201
68
-clear_stem_cache => \&Lingua::Stem::En_Broken::clear_stem_cache,
69
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'en-broken' locale"); },
202
70
},
203
71
'fr' => { -stemmer => \&Lingua::Stem::Fr::stem,
204
72
-stem_caching => \&Lingua::Stem::Fr::stem_caching,
205
73
-clear_stem_cache => \&Lingua::Stem::Fr::clear_stem_cache,
74
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'fr' locale"); },
206
75
},
207
76
'gl' => { -stemmer => \&Lingua::Stem::Gl::stem,
208
77
-stem_caching => \&Lingua::Stem::Gl::stem_caching,
209
78
-clear_stem_cache => \&Lingua::Stem::Gl::clear_stem_cache,
79
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'gl' locale"); },
210
80
},
211
81
'it' => { -stemmer => \&Lingua::Stem::It::stem,
212
82
-stem_caching => \&Lingua::Stem::It::stem_caching,
213
83
-clear_stem_cache => \&Lingua::Stem::It::clear_stem_cache,
84
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'it' locale"); },
214
85
},
215
86
'no' => { -stemmer => \&Lingua::Stem::No::stem,
216
87
-stem_caching => \&Lingua::Stem::No::stem_caching,
217
88
-clear_stem_cache => \&Lingua::Stem::No::clear_stem_cache,
89
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'no' locale"); },
218
90
},
219
91
'pt' => { -stemmer => \&Lingua::Stem::Pt::stem,
220
92
-stem_caching => \&Lingua::Stem::Pt::stem_caching,
221
93
-clear_stem_cache => \&Lingua::Stem::Pt::clear_stem_cache,
94
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'pt' locale"); },
222
95
},
223
96
'sv' => { -stemmer => \&Lingua::Stem::Sv::stem,
224
97
-stem_caching => \&Lingua::Stem::Sv::stem_caching,
225
98
-clear_stem_cache => \&Lingua::Stem::Sv::clear_stem_cache,
99
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'sv' locale"); },
226
100
},
227
101
'ru' => { -stemmer => \&Lingua::Stem::Ru::stem,
228
102
-stem_caching => \&Lingua::Stem::Ru::stem_caching,
229
103
-clear_stem_cache => \&Lingua::Stem::Ru::clear_stem_cache,
104
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'ru' locale"); },
230
105
},
231
106
'ru_ru' => {
232
107
-stemmer => \&Lingua::Stem::Ru::stem,
233
108
-stem_caching => \&Lingua::Stem::Ru::stem_caching,
234
109
-clear_stem_cache => \&Lingua::Stem::Ru::clear_stem_cache,
110
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'ru_ru' locale"); },
235
111
},
236
112
'ru-ru' => {
237
113
-stemmer => \&Lingua::Stem::Ru::stem,
238
114
-stem_caching => \&Lingua::Stem::Ru::stem_caching,
239
115
-clear_stem_cache => \&Lingua::Stem::Ru::clear_stem_cache,
116
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'ru-ru' locale"); },
240
117
},
241
118
'ru-ru.koi8-r' => {
242
119
-stemmer => \&Lingua::Stem::Ru::stem,
243
120
-stem_caching => \&Lingua::Stem::Ru::stem_caching,
244
121
-clear_stem_cache => \&Lingua::Stem::Ru::clear_stem_cache,
122
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'ru-ru.koi8-r' locale"); },
245
123
},
246
124
'ru_ru.koi8-r' => {
247
125
-stemmer => \&Lingua::Stem::Ru::stem,
248
126
-stem_caching => \&Lingua::Stem::Ru::stem_caching,
249
127
-clear_stem_cache => \&Lingua::Stem::Ru::clear_stem_cache,
128
-stem_in_place => sub { require Carp; Carp::croak("'stem_in_place' not available for 'ru_ru.koi8-r' locale"); },
250
129
},
251
130
},
252
};
253
254
#######################################################################
255
# Methods
256
#######################################################################
257
258
=head1 METHODS
259
260
=cut
261
262
#######################################################################
263
264
=over 4
265
266
=item new(...);
267
268
Returns a new instance of a Lingua::Stem object and, optionally, selection
269
of the locale to be used for stemming.
270
271
Examples:
272
273
# By default the locale is en
274
$us_stemmer = Lingua::Stem->new;
275
276
# Turn on the cache
277
$us_stemmer->stem_caching({ -level => 2 });
278
279
# Overriding the default for a specific instance
280
$uk_stemmer = Lingua::Stem->new({ -locale => 'en-uk' });
281
282
# Overriding the default for a specific instance and changing the default
283
$uk_stemmer = Lingua::Stem->new({ -default_locale => 'en-uk' });
284
285
=back
286
287
=cut
131
};
132
133
###
288
134
289
135
sub new {
290
136
my $proto = shift;
291
my $class = ref ($proto) || $proto || __PACKAGE__;
137
my $package = __PACKAGE__;
138
my $proto_ref = ref($proto);
139
my $class;
140
if ($proto_ref) {
141
$class = $proto_ref;
142
} elsif ($proto) {
143
$class = $proto;
144
} else {
145
$class = $package;
146
}
292
147
my $self = bless {},$class;
293
148
294
149
# Set the defaults
295
150
%{$self->{'Lingua::Stem'}->{-exceptions}} = %{$defaults->{-exceptions}};
296
151
$self->{'Lingua::Stem'}->{-locale} = $defaults->{-locale};
297
152
$self->{'Lingua::Stem'}->{-stemmer} = $defaults->{-stemmer};
153
$self->{'Lingua::Stem'}->{-stem_in_place} = $defaults->{-stem_in_place};
298
154
$self->{'Lingua::Stem'}->{-stem_caching} = $defaults->{-stem_caching};
299
155
$self->{'Lingua::Stem'}->{-clear_stem_cache} = $defaults->{-clear_stem_cache};
300
156
307
163
}
308
164
foreach my $key (keys %$parm_ref) {
309
165
my $lc_key = lc ($key);
310
if ($lc_key eq '-locale') { $self->set_locale($parm_ref->{$key}); }
311
elsif ($lc_key eq '-default_locale') { set_locale($parm_ref->{$key}); }
312
else { push (@errors," '$key' => '$parm_ref->{$key}'"); }
166
if ($lc_key eq '-locale') { $self->set_locale($parm_ref->{$key}); }
167
elsif ($lc_key eq '-default_locale') { set_locale($parm_ref->{$key}); }
168
else { push (@errors," '$key' => '$parm_ref->{$key}'"); }
313
169
}
314
170
}
315
171
if ($#errors > -1) {
316
croak (__PACKAGE__ . "::new() - unrecognized parameters passed:" . join(', ',@errors));
172
require Carp;
173
Carp::croak ($package . "::new() - unrecognized parameters passed:" . join(', ',@errors));
317
174
}
318
175
319
$self;
176
return $self;
320
177
}
321
178
322
#######################################################################
323
324
=over 4
325
326
=item set_locale($locale);
327
328
Sets the locale to one of the recognized locales.
329
locale identifiers are converted to lowercase.
330
331
Called as a class method, it changes the default locale for all
332
subseqently generated object instances.
333
334
Called as an instance method, it only changes the locale for
335
that particular instance.
336
337
'croaks' if passed an unknown locale.
338
339
Examples:
340
341
# Change default locale
342
Lingua::Stem::set_locale('en-uk'); # UK's spellings
343
344
# Change instance locale
345
$self->set_locale('en-us'); # US's spellings
346
347
=back
348
349
=cut
179
###
350
180
351
181
sub set_locale {
352
182
my ($self) = shift;
356
186
($locale) = @_;
357
187
$locale = lc $locale;
358
188
if (not exists $defaults->{-known_locales}->{$locale}) {
359
croak (__PACKAGE__ . "::set_locale() - Unknown locale '$locale'");
189
require Carp;
190
Carp::croak (__PACKAGE__ . "::set_locale() - Unknown locale '$locale'");
360
191
}
361
192
$self->{'Lingua::Stem'}->{-locale} = $locale;
362
193
$self->{'Lingua::Stem'}->{-stemmer} = $defaults->{-known_locales}->{$locale}->{-stemmer};
194
$self->{'Lingua::Stem'}->{-stem_in_place} = $defaults->{-known_locales}->{$locale}->{-stem_in_place};
363
195
$self->{'Lingua::Stem'}->{-stem_caching} = $defaults->{-known_locales}->{$locale}->{-stem_caching};
364
196
$self->{'Lingua::Stem'}->{-clear_stem_cache} = $defaults->{-known_locales}->{$locale}->{-clear_stem_cache};
365
197
} else {
366
198
$locale = lc $self;
367
199
if (not exists $defaults->{-known_locales}->{$locale}) {
368
croak (__PACKAGE__ . "::set_locale() - Unknown locale '$locale'");
200
require Carp;
201
Carp::croak (__PACKAGE__ . "::set_locale() - Unknown locale '$locale'");
369
202
}
370
203
$defaults->{-locale} = $locale;
371
204
$defaults->{-stemmer} = $defaults->{-known_locales}->{$locale}->{-stemmer};
205
$defaults->{-stem_in_place} = $defaults->{-known_locales}->{$locale}->{-stem_in_place};
372
206
$defaults->{-stem_caching} = $defaults->{-known_locales}->{$locale}->{-stem_caching};
373
207
$defaults->{-clear_stem_cache} = $defaults->{-known_locales}->{$locale}->{-clear_stem_cache};
374
208
}
209
return;
375
210
}
376
211
377
#######################################################################
378
379
=over 4
380
381
=item get_locale;
382
383
Called as a class method, returns the current default locale.
384
385
Example:
386
387
$default_locale = Lingua::Stem::get_locale;
388
389
Called as an instance method, returns the locale for the instance
390
391
$instance_locale = $stemmer->get_locale;
392
393
=back
394
395
=cut
212
###
396
213
397
214
sub get_locale {
398
my ($self) = shift;
215
my $self = shift;
399
216
400
217
if (ref $self) {
401
218
return $self->{'Lingua::Stem'}->{-locale};
404
221
}
405
222
}
406
223
407
#######################################################################
408
409
=over 4
410
411
=item add_exceptions($exceptions_hash_ref);
412
413
Exceptions allow overriding the stemming algorithm on a case by case
414
basis. It is done on an exact match and substitution basis: If a passed
415
word is identical to the exception it will be replaced by the specified
416
value. No case adjustments are performed.
417
418
Called as a class method, adds exceptions to the default exceptions list
419
used for subsequently instantations of Lingua::Stem objects.
420
421
Example:
422
423
# adding default exceptions
424
Lingua::Stem::add_exceptions({ 'emily' => 'emily',
425
'driven' => 'driven',
426
});
427
428
Called as an instance method, adds exceptions only to the specific
429
instance.
430
431
# adding instance exceptions
432
$stemmer->add_exceptions({ 'steely' => 'steely' });
433
434
The exceptions shortcut the normal stemming - if an exception matches
435
no further stemming is performed after the substitution.
436
437
Adding an exception with the same key value as an already defined
438
exception replaces the pre-existing exception with the new value.
439
440
=back
441
442
=cut
224
###
443
225
444
226
sub add_exceptions {
445
my ($self);
227
my $self;
446
228
447
my ($exceptions,$exception_list);
448
my $reference =ref $_[0];
229
my ($exceptions, $exception_list);
230
my $reference = ref $_[0];
449
231
if ($reference eq 'HASH') {
450
232
($exceptions) = @_;
451
233
$exception_list = $defaults->{-exceptions};
460
242
while (my ($exception,$replace_with) = each %$exceptions) {
461
243
$exception_list->{$exception} = $replace_with;
462
244
}
245
return;
463
246
}
464
247
465
#######################################################################
466
467
=over 4
468
469
=item delete_exceptions(@exceptions_list);
470
471
The mirror of add_exceptions, this allows the _removal_ of exceptions
472
from either the defaults for the class or from the instance.
473
474
# Deletion of exceptions from class default exceptions
475
Lingua::Stem::delete_exceptions('aragorn','frodo','samwise');
476
477
# Deletion of exceptions from instance
478
$stemmer->delete_exceptions('smaug','sauron','gollum');
479
480
# Deletion of all class default exceptions
481
delete_exceptions;
482
483
# Deletion of all exceptions from instance
484
$stemmer->delete_exceptions;
485
486
=back
487
488
=cut
248
###
489
249
490
250
sub delete_exceptions {
491
251
my $self;
519
279
}
520
280
521
281
foreach (@$exceptions) { delete $exception_list->{$_}; }
282
return;
522
283
}
523
284
524
#######################################################################
525
526
=over 4
527
528
=item get_exceptions;
529
530
As a class method with no parameters it returns all the default exceptions
531
as an anonymous hash of 'exception' => 'replace with' pairs.
532
533
Example:
534
535
# Returns all class default exceptions
536
$exceptions = Lingua::Stem::get_exceptions;
537
538
As a class method with parameters, it returns the default exceptions listed
539
in the parameters as an anonymous hash of 'exception' => 'replace with' pairs.
540
If a parameter specifies an undefined 'exception', the value is set to undef.
541
542
# Returns class default exceptions for 'emily' and 'george'
543
$exceptions = Lingua::Stem::get_exceptions('emily','george');
544
545
As an instance method, with no parameters it returns the currently active
546
exceptions for the instance.
547
548
# Returns all instance exceptions
549
$exceptions = $stemmer->get_exceptions;
550
551
As an instance method with parameters, it returns the instance exceptions listed
552
in the parameters as an anonymous hash of 'exception' => 'replace with' pairs.
553
If a parameter specifies an undefined 'exception', the value is set to undef.
554
555
# Returns instance exceptions for 'lisa' and 'bart'
556
$exceptions = $stemmer->get_exceptions('lisa','bart');
557
558
=back
559
560
=cut
285
###
561
286
562
287
sub get_exceptions {
563
288
583
308
$exception_list->{$_} = $_;
584
309
}
585
310
}
586
$exception_list;
311
return $exception_list;
587
312
}
588
313
589
#######################################################################
590
591
=over 4
592
593
=item stem(@list);
594
595
Called as a class method, it applies the default settings
596
and stems the list of passed words, returning an anonymous
597
array with the stemmed words in the same order as the passed
598
list of words.
599
600
Example:
601
602
# Default settings applied
603
my $anon_array_of_stemmed_words = Lingua::Stem::stem(@words);
604
605
Called as an instance method, it applies the instance's settings
606
and stems the list of passed words, returning an anonymous
607
array with the stemmed words in the same order as the passed
608
list of words.
609
610
# Instance's settings applied
611
my $stemmed_words = $stemmer->stem(@words);
612
613
The stemmer performs best when handed long lists of words
614
rather than one word at a time. The cache also provides
615
a huge speed up if you are processing lots of text.
616
617
=back
618
619
=cut
314
####
620
315
621
316
sub stem {
622
317
my $self;
637
332
-exceptions => $exceptions });
638
333
}
639
334
640
#######################################################################
641
642
=over 4
643
644
=item clear_stem_cache;
645
646
Clears the stemming cache for the current locale. Can be called as either
647
a class method or an instance method.
648
649
$stemmer->clear_stem_cache;
650
651
clear_stem_cache;
652
653
=back
654
655
=cut
335
###
336
337
sub stem_in_place {
338
my $self;
339
return [] if ($#_ == -1);
340
my ($exceptions,$locale,$stemmer);
341
if (ref $_[0]) {
342
my $self = shift;
343
$exceptions = $self->{'Lingua::Stem'}->{-exceptions};
344
$stemmer = $self->{'Lingua::Stem'}->{-stem_in_place};
345
$locale = $self->{'Lingua::Stem'}->{-locale};
346
} else {
347
$exceptions = $defaults->{-exceptions};
348
$stemmer = $defaults->{-stem_in_place};
349
$locale = $defaults->{-locale};
350
}
351
&$stemmer({ -words => [\@_],
352
-locale => $locale,
353
-exceptions => $exceptions });
354
}
355
356
###
656
357
657
358
sub clear_stem_cache {
658
359
my $clear_stem_cache_sub;
665
366
&$clear_stem_cache_sub;
666
367
}
667
368
668
#######################################################################
669
670
=over 4
671
672
=item stem_caching ({ -level => 0|1|2 });
673
674
Sets stemming cache level for the current locale. Can be called as either
675
a class method or an instance method.
676
677
$stemmer->stem_caching({ -level => 1 });
678
679
stem_caching({ -level => 1 });
680
681
For the sake of maximum compatibility with previous versions,
682
stem caching is set to '-level => 0' by default.
683
684
'-level' definitions
685
686
'0' means 'no caching'. This is the default level.
687
688
'1' means 'cache per run'. This caches stemming results during each
689
call to 'stem'.
690
691
'2' means 'cache indefinitely'. This caches stemming results until
692
either the process exits or the 'clear_stem_cache' method is called.
693
694
stem caching is global to the locale. If you turn on stem caching for one
695
instance of a locale stemmer, all instances using the same locale will have it
696
turned on as well.
697
698
I B<STRONGLY> suggest turning caching on if you have enough memory and
699
are processing a lot of data.
700
701
=back
702
703
=cut
369
###
704
370
705
371
sub stem_caching {
706
372
my $stem_caching_sub;
714
380
&$stem_caching_sub(@_);
715
381
}
716
382
717
#######################################################################
718
# Terminal POD Documentation
719
#######################################################################
720
721
=head1 VERSION
722
723
0.81 2004.07.26
724
725
=head1 NOTES
726
727
It started with the 'Text::Stem' module which has been adapted into
728
a more general framework and moved into the more
729
language oriented 'Lingua' namespace and re-organized to support a OOP
730
interface as well as switch core 'En' locale stemmers.
731
732
Version 0.40 added a cache for stemmed words. This can provide up
733
to a several fold performance improvement.
734
735
Organization is such that extending this module to any number
736
of languages should be direct and simple.
737
738
Case flattening is a function of the language, so the 'exceptions'
739
methods have to be used appropriately to the language. For 'En'
740
family stemming, use lower case words, only, for exceptions.
741
742
=head1 AUTHORS
743
744
Benjamin Franz <snowhare@nihongo.org>
745
Jim Richardson <imr@maths.usyd.edu.au>
746
747
=head1 CREDITS
748
749
Jim Richardson <imr@maths.usyd.edu.au>
750
Ulrich Pfeifer <pfeifer@ls6.informatik.uni-dortmund.de>
751
Aldo Calpini <dada@perl.it>
752
xern <xern@cpan.org>
753
Ask Solem Hoel <ask@unixmonks.net>
754
Dennis Haney <davh@davh.dk>
755
S�bastien Darribere-Pleyt <sebastien.darribere@lefute.com>
756
Aleksandr Guidrevitch <pillgrim@mail.ru>
757
758
=head1 SEE ALSO
759
760
Lingua::Stem::En Lingua::Stem::En Lingua::Stem::Da
761
Lingua::Stem::De Lingua::Stem::Gl Lingua::Stem::No
762
Lingua::Stem::Pt Lingua::Stem::Sv Lingua::Stem::It
763
Lingua::Stem::Fr Lingua::Stem::Ru Text::German
764
Lingua::PT::Stemmer Lingua::GL::Stemmer Lingua::Stem::Snowball::No
765
Lingua::Stem::Snowball::Se Lingua::Stem::Snowball::Da Lingua::Stem::Snowball::Sv
766
Lingua::Stemmer::GL Lingua::Stem::Snowball
767
768
http://snowball.tartarus.org
769
770
=head1 COPYRIGHT
771
772
Copyright 1999-2004
773
774
Freerun Technologies, Inc (Freerun),
775
Jim Richardson, University of Sydney <imr@maths.usyd.edu.au>
776
and Benjamin Franz <snowhare@nihongo.org>. All rights reserved.
777
778
Text::German was written and is copyrighted by Ulrich Pfeifer.
779
780
Lingua::Stem::Snowball::Da was written and is copyrighted by
781
Dennis Haney and Ask Solem Hoel.
782
783
Lingua::Stem::It was written and is copyrighted by Aldo Calpini.
784
785
Lingua::Stem::Snowball::No, Lingua::Stem::Snowball::Se, Lingua::Stem::Snowball::Sv were
786
written and are copyrighted by Ask Solem Hoel.
787
788
Lingua::Stemmer::GL and Lingua::PT::Stemmer were written and are copyrighted by Xern.
789
790
Lingua::Stem::Fr was written and is copyrighted by Aldo Calpini and Sébastien Darribere-Pley.
791
792
Lingua::Stem::Ru was written and is copyrighted by Aleksandr Guidrevitch.
793
794
This software may be freely copied and distributed under the same
795
terms and conditions as Perl.
796
797
=head1 BUGS
798
799
None known.
800
801
=head1 TODO
802
803
Add more languages. Extend regression tests. Add support for the
804
Lingua::Stem::Snowball family of stemmers as an alternative core stemming
805
engine.
806
807
=cut
808
809
383
1;
Loggerhead is a web-based interface for Breezy
Version: 2.0.1