~ubuntu-branches/ubuntu/wily/calamaris/wily

Calamaris
Version 2

What is it?
-----------

Calamaris is a Perl script, which was first intended as demo for a statistical
software for Squid.  I started it at 13 January 1997 (Version 1.1) as a
rewrite of my old Squid-Analysis-Script weekly.pl (which was in German
language). I announced it (Version 1.16) to the public at 28 Feb 1997.  (see
http://www.squid-cache.org/mail-archive/squid-users/199702/0551.html for the
Original-Announcement) Since then it is used by people all around the world,
and i decided to build a new improved version of it.  Calamaris V2 is a nearly
complete rewrite, with changed and more reports.


Which software can produce Calamaris-parseable Log-files?
---------------------------------------------------------

* Squid V1.1.alpha1-V2.x (http://www.squid-cache.org/)
* NetCache V??? (http://www.netapp.com/products/netcache/)
* Inktomi Traffic Server V???
  (http://www.inktomi.com/products/network/products/)
* Oops! proxy server V??? (http://zipper.paco.net/~igor/oops.eng/)
* Compaq Tasksmart (http://www.compaq.com/tasksmart/)
* Novell Internet Caching System (http://www.novell.com/products/ics/)
* Netscape/iPlanet/SunONE Web Proxy Server
  (http://www.iplanet.com/downloads/download/detail_14_13.html)
* Squid with the SmartFilter-patch
* Cisco Content Engines
  (http://www.cisco.com/en/US/products/hw/contnetw/index.html)

Where to get Calamaris?
-----------------------

The Calamaris-Home-page is located at http://Calamaris.Cord.de/

There is also an Announcement-Mailing-list. To subscribe send mail with
'subscribe your@mail.adr.ess' in the Mail-Body to
<Calamaris-announce-request@Cord.de>.  Subscribers will get a mail on every new
release, including a list of the changes. --> low traffic.

Philipp Frauenfelder <pfrauenf@debian.org> has build a Debian Package, which
can be found via http://packages.debian.org/calamaris .

There is a port for FreeBSD, which can be found at
http://www.freebsd.org/ports/www.html

A port for NetBSD is here:
ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-current/pkgsrc/www/calamaris/

Henri Gomez build Red-Hat rpm's which can be obtained from
ftp://ftp.falsehope.com/home/gomez/calamaris/

rpm's are also available from various people. You can search for them
via http://rpmfind.net/linux/rpm2html/search.php?query=calamaris, for
SuSE-Linux search for 'calamari'. (Yes, without the tailing 's', they use for
some reason a 8.3-scheme for their packages.)


How to use it?
--------------

* You'll need Perl Version 5 (see http://www.Perl.com/). Calamaris is reported
  to work with Perl 5.001 (maybe you have to remove the '-w' from the first
  line and comment out the 'use vars'-line), but it is highly recommended
  (especially for security of your computer) that you use a recent version
  (>=5.005_03) of it.

* You'll also need one of the noted log-files:

  + Squid V1.1.alpha1-V1.1.beta25 Native log-files
  + Squid V1.1.beta26-V2.x Native log-files
  + Squid V1.1.beta26-V2.x Native log-files with log_mime_hdrs enabled
  + NetCache V??? Squid-style Log-files
  + NetCache V5.x Default Log-file-Format (Extended Log-file-Format)
  + Inktomi Traffic Server V??? Log-files
  + OOPS V??? Native log-files
  + Extended Log-file-Format
  + NetApp Default Log-file-Format (some kind of Extended Log-file-Format)
  + NetApps understanding of Squid Native log-files
  + Squid with SmartFilter-Patch Log-files
  + Cisco Content Engines

  If Calamaris can't parse the input, check your log-file format.
  + Squid-Log-files: http://www.squid-cache.org/Doc/FAQ/FAQ-6.html
  + Extended Log-file-Format: http://www.w3.org/TR/WD-logfile

* Put Calamaris itself into a warm, dry place on your computer (i.e. into the
  Squid-bin-directory, or /usr/local/bin/). Maybe (if your Perl isn't located
  at /usr/bin/perl) you'll have to change the first line of Calamaris to point
  to your copy of Perl.

* There is also a man-page for Calamaris. You should copy it to an appropriate
  place like /usr/local/man/man1, where your man(1) can find it.

* Use it!

  'cat access.log.1 access.log.0 | calamaris'

  Calamaris by default generates by a brief ASCII report of incoming and
  outgoing requests.

  NOTE: If you pipe more than one log-file into Calamaris, make sure that they
    are chronologically ordered (oldest file first), else some reports can
    return wrong values.

  You can alter Calamaris' behaviour with switches. Start Calamaris with '-h'
  or check the man-page.

  You should also take a look at the EXAMPLES-File, for
  'Real-Life'-usage-examples of Calamaris.


Are there known bugs or other problems?
---------------------------------------

* RedHat 8.0 has set the LANG-Variable to en_US.UTF-8, which caused Calamaris
  to crash with 'Split loop at (eval 1) line 21, <> line 1', maybe due to a
  perl-bug. (Investigation needed). You can workaround this problem by
  unsetting LANG. Please see
  https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=77437 .

* There is a problem if you parse Logfiles from accelerating proxies. In
  iPlanet Web Proxy Server there is only an 'unqualified' URL in the Logfiles,
  which confuses all reports that rely on that field. (reported by Pawel
  Worach <pawel.worach@nordea.com>)

* Calamaris can't resolve IPv6-IPs to DNS-Names yet. If you can tell me how i
  can do it in perl, let me know ;-)

* the byte-histogram sometimes displays a 0-0-byte line. this is correct. the
  requests added there are really logged as 0-byte-sized in the Logfile. Note
  that empty byte-ranges as 1-9 (which is impossible because of
  protocol-overhead) are skipped and not displayed in the report, so you can
  end up with 0-0 followed by 10-99 in the report. (noted by Reagan Blundell
  <reagan@muppet.whatever.net.au> through Debian-Bug-Tracking)

* there were many requests to add something that enables Calamaris to track
  down who is using the Cache to get what. I added these with stomach-ache,
  because it breaks the privacy of the users. So please read the following
  and think about it, before using Calamaris to be the 'Big Brother':

  - If you don't trust your users than there is something more wrong than
    the loss of productivity.
  - Squid has some nice acl-mechanisms. If you think that your users don't
    use the net properly, don't let them use it. (You can also open the net
    at specific times or to specific sites, if you want.)

  If you still want to use Calamaris that way, let your vict^Wusers know
  that they'll be monitored. (in Germany you have to let them know!)

  After some discussion in a newsgroup i was 'accused' to not announce the
  spy-features of Calamaris. THIS IS INTENTIONAL. This is MY program, it can
  spy on users, because there are many people who want to have that feature,
  but what i write on the feature-sheet is MY decision. Don't annoy me, you
  might damage my motivation to maintain this FREE (as in speech) piece of
  software.

* I fixed a bug regarding caching of Performance-Data. This breaks old
  Calamaris-Cache-Files... I put a workaround in, which allows to parse old
  and new cache-files. But you will loose the 'Cache-Hits'-value from old
  files in the Performance-Report.  It is set to '-' in the output.

* If you reuse a cache-file, which is not created with
  '-d -1 -r -1 -t -1 -R -1' the number of 'others' is likely wrong everywhere.
  (reported by Clare Lahiff <clare@tarboosh.anu.edu.au>)

  If i store the number of 'others' somewhere i still don't know which data is
  ment there, and in the next run (if i sum up) the number of others is to
  high (if the number of occurrences is below the threshold) or the summed up
  data misses the occurrences of the last run (if the number of occurances is
  above the threshold). i think i can't fix this...

* If you want to parse more than one logfile (i.e. from the 'logfilerotate')
  or want to use more than one input-cache-file you have to put them in
  chronological sorted order (oldest first), else you get wrong peak values.

  However: If you use the caching function the peak-values can be wrong,
  because peaks occurring during log-rotate-time can't be detected.

  Calamaris will add a warning to the report if it recognises unsorted input.

* Squid with SmartFilter-Patch and Cisco Content Engines have the ability to
  block or allow requests by checking against a database, and write this to
  the Logfiles. I will not add a report to give an overview about the usage
  of the Categories. (see first point of this chapter.)

* Squid doesn't log outgoing UDP-Requests, so i can't put them into the
  statistics without parsing squid.conf and the cache.log-file. (Javier Puche
  <Javier.Puche@rediris.es> asked for this), but i don't think that i should
  put this into Calamaris...

* Squid and NetCache also support some kind of 'Common Logfile-format'.  I
  won't support that, because Common Log is missing some very important data
  i.e. the request-time and the hierarchie-information. If you're still stuck
  with that format, i recommend the 'analog'-software by Steven Turner. Other
  way round:  change logging to 'native' and convert it to 'common'. There is
  software for that available, i.e. my shrimp.pl. This also applies for the
  Common-style Log-files which NetCache produces.

* If you use Calamaris at UNIX-epoch-date 2147483648 or later (~19.Jan 2038)
  you might get wrong dates on 32bit-systems.  (I just added this to delight
  the people who really read this ;-) and to make a statement on this... on
  Y2K they found many systems which wasn't expected to run in that year. If
  you read this while checking if this package is Y2K038-compatible, then
  this is probably a really old system ;-)
  Y2038-Statement: Calamaris is as buggy as the used perl-version.

* It is written in Perl. Yea, Perl is a great language for something like this
  (also it is the only one I'm able to write something like this in ;-).
  Calamaris was first intended as demo for what i expect from a statistical
  software. (OK, it is fun to write it, and it is even more fun to recognise
  that many people use the script). For my Caches with about 150MB logfile per
  week it is OK, but for those people on a heavy loaded Parent-cache it is
  possibly to slow.  How does it perform with the perlcc coming in Perl5.6 or
  Perl6?


What will happen next?
----------------------

I think that Calamaris v2 is now finished. (except for bugs, that maybe were
not found yet.)

But if you have an idea what is still missing in a software for parsing proxy
log-files, let me know. --> <Calamaris@Cord.de>. I'll will build it in, or add
it to the wish-list below :-)

* adding an option to limit domain (and related) reports to a minimum number
  of transactions. (suggested by Franck Bourdonnec <fbourdonnec@chez.com>)

* I suggest that the '-P' option has a <number of lines> option, like all
  other analyses. So the time step would be something like = <analysis time> /
  <number of lines>. (Ahmad Kamal <eng_ak@link.net>)

* rewrite peak-measurement. The new calculation method is very time intensive
  and slows down Calamaris by 30 or more percent, but it is faster than the
  old way. However: i'm not really satisfied with it, so i put it out of the
  -a -option. You'll have to add a '-p (old|new)' -option to get old or new
  peak-statistic.

  HELP REQUEST: If someone has an idea how to build an efficient AND fast
  method to work it out... let me know!

* try 'use integer'. This can result in a less memory-hungry, but faster
  version of Calamaris. (idea by Gerold Meerkoetter)

* build graphics (hope i remember who suggested this first, the mail must be
  somewhere in my work-mailbox ;-) (This is a thing for Calamaris v3, if i
  ever going to write it. there are nice gd-libs in Perl ;-)

* make Calamaris faster. see above. If someone wants to rewrite Calamaris in a
  faster language: Feel Free! (But respect the GNU-License) It would be nice
  if you drop me a line about it, I'll mention it below. And please please
  please don't use the name 'Calamaris' for it without asking me!


Is there anything else?
-----------------------

Ernst Heiri has build a spin-off of my Calamaris V1, which can be found
*where?*

There is also a C++-port of Ernst Heiri's Calamaris available. It is
(according to the author Jens-S. Voeckler <voeckler@rvs.uni-hannover.de>) five
times faster than the Perl-variant.  check
http://www.cache.dfn.de/DFN-Cache/Development/seafood.html for this.

more Squid-logfile-Analysers can be found via the Squid-Home-page at
http://www.squid-cache.org/Scripts/


Thank You!
----------

* The developers and contributors of Squid.
* The developers and contributors of Perl.
* The contributors, feature requesters and bug-reporters of Calamaris.
* Gerold 'Nimm Perl' Meerkoetter.
* Massimo Carnevali <lec3748@iperbole.bologna.it>


Not happy yet?
--------------

Drop me a line to <Calamaris@Cord.de> and tell me what is missing or wrong or
not clear or whatever. You are welcome (especially if you read this file that
far :-)


Version of the README
---------------------

$Id: README,v 2.43 2004/06/06 16:29:02 cord Exp $