~ubuntu-branches/ubuntu/hardy/dbacl/hardy

\" t

.TH MAILFOOT 1 "Bayesian Text Classification Tools" "Version @VERSION@" ""

.SH NAME

mailfoot \- a full-online-ordered-training simulator for use with dbacl.

.SH SYNOPSIS

.HP

.B mailfoot

.I command 

[

.I command_arguments 

]

.SH DESCRIPTION

.PP

.B mailfoot

automates the task of testing email filtering and classification

programs such as 

.BR dbacl (1).

Given a set of categorized documents, mailfoot initiates test runs 

to estimate the classification errors and thereby permit fine tuning 

of the parameters of the classifier. 

.PP

Full Online Ordered Training is a learning method for email classifiers where

each incoming email is learned as soon as it arrives, thereby always keeping category 

descriptions up to date for the next classification.

This directly models the way that some email classifiers are used in practice.

.PP

FOOT's error rates depend directly on the order in which emails are seen.

A small change in ordering, as might happen due to networking delays,

can have an impact on the number of misclassifications. 

Consequently, 

.B mailfoot

does not give meaningful results, unless the sample emails are chosen carefully.

However, as this method is commonly used by spam filters, it is still worth 

computing to foster comparisons. Other methods  (see

.BR mailcross (1), mailtoe (1))

attempt to capture the behaviour of classification errors in other ways.

.PP

To improve and stabilize the error rate calculation,

.B mailfoot

performs the FOOT simulations several times on slightly reordered email streams, and 

averages the results. The reorderings occur by multiplexing the emails from each

category mailbox in random order. Thus if there are three categories, the first email

classified is chosen randomly from the front of the sample email streams of each type. 

The second email is also chosen randomly among the three types, from the front of the

 streams after the first email was removed. Simulation stops when all sample streams 

are exhausted.

.PP

.B mailfoot

uses the environment variable MAILFOOT_FILTER when

executing, which permits the simulation of arbitrary filters, provided

these satisfy the compatibility conditions stated in the  

ENVIRONMENT section below.

.PP

For convenience, 

.B mailfoot

implements a 

.B testsuite 

framework with predefined wrappers for several open

source classifiers. This permits the direct comparison of 

.BR dbacl (1) 

with competing classifiers on the same set of email samples. See the USAGE section below.

.PP

During preparation, 

.B mailfoot

builds a subdirectory named mailfoot.d in the current working directory. 

All needed calculations are performed inside this subdirectory.

.SH EXIT STATUS

.B mailfoot

returns 0 on success, 1 if a problem occurred.

.SH COMMANDS

.PP

.PP

.IP "\fBprepare\fR \fIsize\fR"

Prepares a subdirectory named mailfoot.d in the current working directory, and

populates it with empty subdirectories for exactly 

.I size

subsets.

.IP "\fBadd\fR \fIcategory\fR [ \fIFILE\fR ]..."

Takes a set of emails from either FILE if specified, or STDIN, and 

associates them with 

.IR category .

The ordering of emails within \fIFILE\fR is preserved, and subsequent \fIFILE\fRs are appended

to the first in each category.

This command can be repeated several times, 

but should be executed at least once.

.IP "\fBclean\fR"

Deletes the directory mailfoot.d and all its contents.

.IP "\fBrun\fR"

Multiplexes randomly from the email streams added earlier, and relearns

categories only when a misclassification occurs. The simulation is repeated

.I size

times.

.IP "\fBsummarize\fR"

Prints average error rates for the simulations.

.IP "\fBplot\fR [ \fIps\fR | \fIlogscale\fR ]..."

Plots the number of errors over simulation time. The "ps" option, if present,

writes the plot to a postscript file in the directory mailfoot/plots, instead of 

being shown on-screen. The "logscale" option, if present, causes the plot to

be on the log scale for both ordinates.

.IP "\fBreview\fR \fItruecat\fR \fIpredcat\fR"

Scans the last run statistics and extracts all the messages which belong to category

.I truecat

but have been classified into category

.IR predcat .

The extracted messages are copied to the directory 

.I mailfoot.d/review 

for perusal.

.PP

.IP "\fBtestsuite list\fR"

Shows a list of available filters/wrapper scripts which can 

be selected.

.IP "\fBtestsuite select\fR [ \fIFILTER\fR ]..."

Prepares the filter(s) named 

.I FILTER

to be used for simulation. The filter name is the name of 

a wrapper script located in the directory 

.IR @PKGDATADIR@/testsuite .

Each filter has a rigid interface documented below, and the act of selecting

it copies it to the 

.I mailfoot.d/filters 

directory. Only filters located there

are used in the simulations.

.IP "\fBtestsuite deselect\fR [ \fIFILTER\fR ]..."

Removes the named filter(s) from the directory

.I mailfoot.d/filters

so that they are not used in the simulation.

.IP "\fBtestsuite run\fR [ \fIplots\fR ]" 

Invokes every selected filter on the datasets added previously, and 

calculates misclassification rates. If the "plots" option is present,

each filter simulation is plotted as a postscript file in the directory 

.IR mailfoot.d/plots .

.IP "\fBtestsuite status\fR"

Describes the scheduled simulations.

.IP "\fBtestsuite summarize\fR"

Shows the cross validation results for all filters. Only makes sense

after the 

.I run 

command.

.SH USAGE

.PP

The normal usage pattern is the following: first, you should separate your email

collection into several categories (manually or otherwise). Each category should

be associated with one or more folders, but each folder should not contain 

more than one category. Next, you should decide how many runs to use, say 10. 

The more runs you use, the better the predicted error rates. However, more runs take more time. 

Now you can type

.HP

.na

% mailfoot prepare 10

.ad

.PP

Next, for every category, you must add every folder associated with this

category. Suppose you have three categories named 

.IR spam , 

.IR work , 

and 

.IR play ,

which are associated with the mbox files 

.IR spam.mbox , 

.IR work.mbox , 

and 

.IR play.mbox 

respectively. You would type

.PP

.na

% mailfoot add spam spam.mbox

.br

% mailfoot add work work.mbox

.br

% mailfoot add play play.mbox

.ad

.PP

You should aim for a similar number of emails in each category, as the random 

multiplexing will be unbalanced otherwise. The ordering of the email messages

in each 

.I *.mbox

file is important, and is preserved during each simulation. If you repeatedly

add to the same category, the later mailboxes will be appended to the first, preserving

the implied ordering. 

.PP

You can now perform as many FOOT simulations as desired. The multiplexed emails

are classified and learned one at a time, by executing the command given in the 

environment variable MAILFOOT_FILTER. If not set, a default value is used. 

.PP

.na

% mailfoot run

.br

% mailfoot summarize

.ad

.PP

The testsuite commands are designed to simplify the above steps and allow comparison

of a wide range of email classifiers, including but not limited to 

.BR dbacl .

Classifiers are supported through wrapper scripts, which are located in the 

.I @PKGDATADIR@/testsuite 

directory. 

.PP

The first stage when using the testsuite is deciding which classifiers to compare.

You can view a list of available wrappers by typing:

.PP

.na

% mailfoot testsuite list

.ad

.PP

Note that the wrapper scripts are NOT the actual email classifiers, which must 

be installed separately by your system administrator or otherwise.

Once this is done, you can select one or more wrappers for the simulation

by typing, for example:

.PP

.na 

% mailfoot testsuite select dbaclA ifile

.ad

.PP

If some of the selected classifiers cannot be found on the system, they

are not selected. Note also that some wrappers

can have hard-coded category names, e.g. if the classifier only supports binary

classification. Heed the warning messages. 

.PP

It remains only to run the simulation. Beware, this can take a long time 

(several hours depending on the classifier). 

.PP

.na

% mailfoot testsuite run

.br

% mailfoot testsuite summarize

.ad

.PP

Once you are all done, you can delete the working files, log

files etc. by typing

.PP

.na

% mailfoot clean

.ad

.SH SCRIPT INTERFACE

.PP

.B mailfoot testsuite 

takes care of learning and classifying your prepared email corpora for each

selected classifier. Since classifiers have widely varying interfaces, this

is only possible by wrapping those interfaces individually into a standard 

form which can be used by 

.BR "mailfoot testsuite" .

.PP

Each wrapper script is a command line tool which accepts a single command 

followed by zero or more optional arguments, in the standard form:

.PP

.na

wrapper command [argument]...

.ad

.PP

Each wrapper script also makes use of STDIN and STDOUT in a well defined 

way. If no behaviour is described, then no output or input should be used.

The possible commands are described below:

.IP filter

In this case, a single email is expected on STDIN, and a list of 

category filenames is expected in $2, $3, etc. The script writes the 

category name corresponding to the input email on STDOUT. No trailing newline

is required or expected.

.IP learn

In this case, a standard mbox stream is expected on STDIN, while a

suitable category file name is expected in $2. No output is written to

STDOUT.

.IP clean

In this case, a directory is expected in $2, which is examined for old

database information. If any old databases are found, they are purged or

reset. No output is written to STDOUT.

.IP describe

IN this case, a single line of text is written to STDOUT, describing the filter's

functionality. The line should be kept short to prevent line wrapping on a terminal.

.IP bootstrap

In this case, a directory is expected in $2. The wrapper script first checks for

the existence of its associated classifier, and other prerequisites. If the 

check is successful, then the wrapper is cloned into the supplied directory.

A courtesy notification should be given on STDOUT to express success or failure.

It is also permissible to give longer descriptions caveats.

.IP toe

Used by 

.BR mailtoe (1).

.IP foot

In this case, a list of categories is expected in $3, $4, etc. Every possible

category must be listed. Preceding this list, the true category is given in $2.

.SH ENVIRONMENT

.PP

Right after loading, 

.B mailfoot 

reads the hidden file .mailfootrc in the $HOME directory, if it exists, so

this would be a good place to define custom values for environment variables.

.IP MAILFOOT_FILTER

This variable contains a shell command to be executed repeatedly

during the running stage.

The command should accept an email message on STDIN and output a

resulting category name. On the command line, it should also accept

first the true category name, then a list of all possible category

file names.  If the output category does not match the true category,

then the relevant categories are assumed to have been silently

updated/relearned.

If MAILFOOT_FILTER is undefined, 

.B mailfoot

uses a default value.

.IP TEMPDIR

This directory is exported for the benefit of wrapper scripts. Scripts which

need to create temporary files should place them a the location given in TEMPDIR.

.SH NOTES

.PP

The subdirectory mailfoot.d can grow quite large. It 

contains a full copy of the training corpora, as well as learning files for 

.I size 

times all the added categories, and various log files. 

.PP

FOOT simulations for 

.BR dbacl (1)

are very, very slow (order n squared) and will take all night to perform. This is not easy to improve.

.SH WARNING

.PP

Because the ordering of emails within the added mailboxes matters, the estimated 

error rates are not well defined or even meaningful in an objective sense. 

However, if the sample emails represent an actual snapshot of a user's incoming email,

then the error rates are somewhat meaningful. The simulations can then be interpreted 

as alternate realities where a given classifier would have intercepted the incoming mail.  

.SH SOURCE

.PP

The source code for the latest version of this program is available at the

following locations: 

.PP

.na

http://www.lbreyer.com/gpl.html

.br

http://dbacl.sourceforge.net

.ad

.SH AUTHOR

.PP

Laird A. Breyer <laird@lbreyer.com>

.SH SEE ALSO

.PP

.BR bayesol (1)

.BR dbacl (1), 

.BR mailcross (1),

.BR mailinspect (1),

.BR mailtoe (1),

.BR regex (7)