← Back to branch summary

~jtv/corpusfiltergraph/cross-python

« back to all changes in this revision

Viewing changes to trunk/lib/corpusfg/graphs/sa-champollion/bin/eos_tokenize.pl

  • Committer: tahoar
  • Date: 2012-05-02 15:46:23 UTC
  • Revision ID: svn-v4:bc069b21-dff4-4e29-a776-06a4e04bad4e::266
new layout. need to update code to use the new layout

Show diffs side-by-side

added added

removed removed

Lines of Context:
trunk/lib/corpusfg/graphs/sa-champollion/bin/eos_tokenize.pl
 
1
#!/usr/bin/perl -n
 
2
#
 
3
# Purpose: split input into one token per line,
 
4
#          put <EOS> marker at the end of sentence
 
5
# Author: Xiaoyi Ma, LDC
 
6
# Date:   September 17, 2003
 
7
# Input: text
 
8
# Output: text, one token per line
 
9
#         <EOS> marker at the end of each sentence
 
10
#
 
11
$| = 1; # disable Perl output buffering
 
12
 
 
13
split ' ', $_;
 
14
 
 
15
foreach (@_) {
 
16
    print "$_\n";
 
17
}
 
18
 
 
19
print "<EOS>\n";