~bronger/+junk/package-refdb-svn

UPGRADING REFDB
===============

This file contains instructions how to upgrade RefDB without losing
your precious data. There is one section for each version that
introduced backwards-incompatible changes requiring a rebuild of your
databases:

0.9.9 -> 1.0.0-preX
0.9.8 -> 0.9.9
0.9.7 -> 0.9.8
0.9.6 -> 0.9.7
0.9.5 -> 0.9.6
0.9.4 -> 0.9.5
0.9.3 -> 0.9.4
0.9/0.9.1/0.9.2 -> 0.9.3
<0.9 -> 0.9

Please visit the refdb-users mailing list archives for further hints
on how to apply this information to pre-releases.


Migrating your existing RefDB installation from 0.9.9 to 1.0.0
==============================================================

__________
Background

The content model of UNPB (unpublished) references is not very well
defined in the RIS "spec". RefDB so far treated this as a monograph,
i.e. like a book or a report. However, in practical use it seems to be
more convenient to treat UNPB as a maybe-submitted, but
not-yet-accepted journal article, thus complementing INPR (in press)
and JOUR (published). 1.0.0 therefore treats UNPB as an
analytic/monographic reference just like JOUR and INPR. Updating from
an older version to 1.0.0 may corrupt UNPB datasets.

_________
Procedure

- save all UNPB references as RIS or risx:
  refdbc -C getref :TY:=UNPB -t ris > unpb.ris

- upgrade to 1.0.0

- update the saved references:
  refdbc -C updateref unpb.ris



Migrating your existing RefDB installation from 0.9.8 to 0.9.9
==============================================================

__________
Background

The RIS format is not very consistent in where it hides the volume,
issue, doi, and edition information of various reference types. They
seem to be scattered randomly among VL, IS, M1-M3 and a few other
tags. RefDB versions until 0.9.8 imported the data straight. RIS data
exported from the database were consistent with the input data, but
other formats often showed inconsistent results. Both the import and
the export of RIS data has been fixed accordingly. However, if you now
export RIS data that were added to the database with an older RefDB
version, the data of the above fields may end up in the wrong tags. It
is therefore recommended to export the data using the older RefDB
version and importing them again using the new version.

The new version makes better use of the L1-L4 fields. The AV field
should no longer be used for electronic offprints, but for physical
copies only. Use L1 instead to provide paths to electronic
offprints. All AV fields using the PATH: kludge (denoting a local PDF
copy) need to be migrated to appropriately encoded L1 fields. To this
end, use the following sed command, or perform a similar substitution
using your favourite text editor:

sed 's%AV  - PATH:%L1  - file://' < oldversion.ris > newversion.ris

The citestylex.dtd was changed to take the improved internal
representation of authors and titles into account. This required a
change in the main database. The upgrading procedure outlined below
should automatically upgrade the main database for you. If you use the
styles shipped with RefDB, you just have to add them again. If you use
custom styles, you'll have to make a few minor adjustments to your
custom styles and add them again.

_________
Procedure

- back up your reference data. Run these commands for each of your
  reference databases from refdbc:
  getref -t risx -o dbname.xml :ID:>0
  getnote -o dbname.notes.xml :NID:>0

- migrate AV fields to L1 where appropriate (see above)

- delete the reference databases
  deletedb dbname

- if you have custom styles and you don't have the XML files handy,
  retrieve the styles now using refdba:
  getstyle -o stylename.xml stylename

- stop refdbd

- now install 0.9.9. Make sure to restart refdbd.

- update the main database
  refdbd -a -u <admin-username> -w <admin-password>

- create the reference databases
  createdb dbname

- if necessary, add the required user accounts
  adduser -d dbname username

- reload your reference and notes data:
  addref -t risx dbname.xml
  addnote dbname.notes.xml

- add the styles shipped with RefDB:
  addstyle styles/*.xml

- if you have custom styles, make the following substitutions in the
  AUTHORLIST ROLE and TITLE ROLE attributes:

  * for monographic items like books, reports, theses:
	PRIMARY -> PUB
	TERTIARY -> SERIES

  * for analytic items like journal or newspaper articles, book chapters:
  	PRIMARY -> PART
  	SECONDARY -> PUB
  	TERTIARY -> SERIES

  If in doubt, consult the styles shipped with RefDB as it is strictly
  necessary to adhere to the analytic/monograpic/series distinction in
  the styles since 0.9.9-preX

  Also, take care of the ALTERNATETEXT attribute of AUTHORLIST. To
  pick a particular title as a replacement for missing author names,
  TITLEPART, TITLEPUB, TITLESERIES, and TITLEALL are now
  supported. The author lists inside CITSTYLE should use TITLEALL.
  The AUTHORLIST and TITLE entries inside CITSTYLE are supposed to
  have the role attribute set to "ALL". Finally, the styles now
  support different formatting of authorlists depending on their
  role. In addition to the TEXT element, which is used to format
  authors, you can define an additional TEXTED element which is used to
  format editors. This way a style can e.g. append a "(ed.)" after the
  names of book editors but not after the name of book authors.



Migrating your existing RefDB installation from 0.9.7 to 0.9.8
==============================================================

__________
Background

Support for citation keys in bibliographies and citations required an
extension of the citestylex.dtd, which in turn required a couple of
additional columns in the refdb database.

_________
Procedure (0.9.8-pre4 and later)

These versions have a still experimental feature to ease upgrading the
main database. If you're sitting in front of your production system,
please back up your main database before proceeding, or at least make
sure you have citestylex copies of your citation styles. If anything
goes wrong, you can still re-create the main database the hard way
using the procedure described in the manual.

MySQL: run as root:
       ~# refdbd -a -u username -w password
PostgreSQL: run as root (or from a special postgres account):
       ~# refdbd -a -u username -w password
SQLite, SQLite3: run as root:
       ~# refdbd -a

"username" and "password" are the ones of your database administrator
account.

If no error message pops up, your main database should be up to date
now.

_________
Procedure (0.9.8-pre3 and older)

As only one of the tables needs three additional columns, MySQL and
PostgreSQL users may directly alter their existing databases like
this:

MySQL:

- start the command line client using your database administrator
  account:
	mysql -u root -p refdb

- run the following two SQL commands (each one in one line):
	mysql> alter table REFSTYLE add CITEKEYPRECEEDING tinytext,
        add CITEKEYFOLLOWING tinytext, add CITEKEYSTYLE varchar(12);

 	mysql> update t_meta set meta_dbversion='2';

PostgreSQL:

- start the command line client using your database administrator
  account:
	psql -U root refdb

- run the following two SQL commands (each one in one line):
	refdb=> alter table REFSTYLE add CITEKEYPRECEEDING text, add
        CITEKEYFOLLOWING text, add CITEKEYSTYLE text;

 	refdb=> update t_meta set meta_dbversion='2';

As SQLite and SQLite3 do not support the ALTER TABLE command, users of
these database engines will have to recreate their system databases as
described in the manual, using the new dump files shipped with RefDB
version 0.9.8.

Existing styles are not affected by the upgrade. They are still valid
against the new version of the DTD, and they can be used right away as
they do not make use of the newly added columns.


Migrating your existing RefDB installation from 0.9.6 to 0.9.7
==============================================================

__________
Background

Only MySQL users need to fix the following problem: The table
definitions in refdb.dump.mysql and refdb.dump.mysql41 used field
sizes too short to hold all available *NAMEORDER strings. The other
databases use a text field with unlimited sizes here, so they are not
affected, and upgrading is a piece of cake.

_________
Procedure

- pray that you have the XML files for all your styles (the handbook
  told you so!!), because RefDB may have truncated the *NAMEORDER
  fields. If not, export them using the refdba:getstyle command and
  fix the contents of the *NAMEORDER fields manually if needed

- stop refdbd

- Follow the instructions to generate the system database
  "refdb". Please be aware that there are two different dump files for
  MySQL: refdb.dump.mysql replaces the previous refdb.dump and is
  meant for MySQL versions prior to 4.1. For versions 4.1 and later,
  please use refdb.dump.mysql41. The SQL scripts remove the existing
  tables before re-creating them, so there is no need to delete the
  database before commencing this step.

- Now install 0.9.7. Make sure to restart refdbd.

- Add your styles using the refdba command addstyle:
  "addstyle stylename.xml"



Migrating your existing RefDB installation from 0.9.5 to 0.9.6
==============================================================

__________
Background

As of 0.9.6-pre1, both reference and the main database are
versioned. The version numbers can be used by the server to avoid
compatibility problems instead of getting stuck in obscure
errors. Needless to say, this change once again requires updating your
databases.

To add some additional hassle, the libdbi drivers were renamed as of
version 0.8.1. If you upgrade from libdbi-drivers-0.7.x, the old
drivers may still be around and lead to confusing problems.

_________
Procedure

- While still running your existing installation, use the refdba
  command getstyle to retrieve existing styles:
  "getstyle -o stylename.xml stylename"
  (you may skip this step if you have the XML files for all your
  styles)

- back up your reference data. Run these commands for each of your
  reference databases:
  getref -t risx -o dbname.xml :ID:>0
  getnote -o dbname.notes.xml :NID:>0

- stop refdbd

- use the command-line client of your database engine to make backups
  of your system and reference databases.

- Use the command-line client of your database engine to remove your
  existing system database "refdb".

- Follow the instructions to generate the system database
  "refdb". Please be aware that there are two different dump files for
  MySQL: refdb.dump.mysql replaces the previous refdb.dump and is
  meant for MySQL versions prior to 4.1. For versions 4.1 and later,
  please use refdb.dump.mysql41.

- deinstall libdbi-drivers 0.7.x. If you built the drivers from
  source, remove all existing driver files in /usr/local/lib/dbd (or
  wherever you installed them).

- install libdbi-0.8.1 and libdbi-drivers-0.8.1

- Now install 0.9.6. Make sure to restart refdbd.

- Add your styles using the refdba command addstyle:
  "addstyle stylename.xml"

- reload your reference and notes data:
  addref -t risx dbname.xml
  addnote dbname.notes.xml


Migrating your existing RefDB installation from 0.9.4 to 0.9.5
==============================================================

__________
Background

The extended bibliography formatting capabilities necessitated another
change in the system database schema. Upgrading requires you to
generate a new system database "refdb".

The in-text citation format had to be changed in order to avoid
ambiguities. This affects only documents that use the long citation
style.


_________
Procedure

- While still running your existing installation, use the refdba
  command getstyle to retrieve existing styles:
  "getstyle -o stylename.xml stylename"
  (you may skip this step if you have the XML files for all your
  styles)

- stop refdbd

- Use the command-line client of your database to remove your existing
  system database "refdb".

- Follow the instructions to generate the system database "refdb".

- Now install 0.9.5. Make sure to restart refdbd.

- Add your styles using the refdba command addstyle:
  "addstyle stylename.xml"

  NB Existing styles will import ok even if they use the older DTD
  from version 0.9.4. There is no need to rewrite the XML style files
  unless you want to use the new features.

- If you're using MySQL as your database engine, you also have to
  migrate your reference databases. To this end, run the following
  command on each of your reference databases:

  mysql -p dbname < scripts/migrate_mysql_0.9.5

  (use -u admin_name if you do not have admin permissions by default)  

- If you maintain your documents using the short citation style,
  e.g. by using refdbnd-generated Makefiles, the new in-text citation
  format will automatically be applied the next time the document is
  edited and re-processed. If in doubt, run "make clean" before
  transforming the document again. If you maintain your documents
  using the long citation format, you'll have to fix the citations by
  inserting a dash before the format specifier. For example, the
  old-style "ID442Y" has to be changed to "ID442-Y".


Migrating your existing RefDB installation from 0.9.3 to 0.9.4
===============================================================

__________
Background

The citation and bibliography styles now support the full range of
fields in the reference databases. The system database uses a
different schema to accommodate the new data.

RefDB now supports extended notes which can be attached to any number
of references, author names, keywords, or periodical names. To store
these data, the reference database schema was changed as well.

In order to use your existing data, you'll have to migrate your
databases to the new version.

_________
Procedure

- Make all sorts of backups of your existing databases. Use the native
  tools of your database server to create SQL dumps. In case of MySQL or
  SQLite databases, you can also make copies of the subdirectories and
  files, respectively.

- While still running your existing installation, use the refdbc
  command "getref -t ris -o dbname.ris :ID:>0" to dump your reference
  databases to RIS files. Please note that this will export the
  personal information of the current user only. If you share your
  database with other users, you should re-run this command for each
  user separately:
  "set username otheruser"
  "set passwd *"
  "getref -t ris -o dbname-otheruser.ris -P :ID:>0". 
  Using risx output is not recommended as the output
  was not bug-free in 0.9.3.

- Unless you kept copies of your bibliography styles in XML files,
  export these as well, using the refdba command getstyle:
  "getstyle -o stylename.xml stylename"

- Now install 0.9.4. Make sure to restart refdbd if it is running.

- Follow the instructions to generate the system database and make
  sure it is called "refdb".

- Add any of your custom styles using the refdba command addstyle:
  "addstyle stylename.xml"

- Create new reference databases and read back your existing data
  with the refdbc command "addref -t ris dbname.ris". If you have
  dumps for other users as well, run "updateref -t ris -P -U otheruser
  dbname-otheruser.ris" in turn for each user.



Migrating your existing RefDB databases from 0.9/0.9.1/0.9.2 to 0.9.3
=====================================================================

__________
Background

The automatic citation key generation code was not exactly brilliant
in versions prior to 0.9.3. The bibliography scripts caused case
mismatches that led to nasty problems. Part of this problem has been
fixed by using xsltproc instead of Jade for extracting the citation
keys from XML documents. SGML documents still have to rely on the
citation keys being uppercase (this is a limitation imposed by the DocBook
SGML declaration). The import routine of version 0.9.3 can be configured
to ensure that the citation keys are all uppercase. In any case,
refdbd makes sure that citation keys use only the 7-bit ASCII range to
avoid character encoding hassles.

_________
Procedure

- Make all sorts of backups of your existing databases. Use the native
  tools of your database server to create SQL dumps. In case of MySQL or
  SQLite databases, you can also make copies of the subdirectories and
  files, respectively.

- After upgrading to 0.9.3, use the refdbc command "getref -t risx -o
  dbname.xml :ID:>0" to dump your databases to XML files.

- If you want to use RefDB for creating SGML bibliographies, make sure
  to restart refdbd with the -U command line option or set the
  upper_citekey configuration variable to "t".

- Create a new database and read back your existing data with the refdbc
  command "addref -t risx dbname.xml"


Upgrading your existing RefDB installation from <0.9 to 0.9
=============================================================

RefDB underwent a couple of modifications which are
backwards-incompatible. The file README lists these changes. This file
explains your options how to migrate existing reference databases to
the new version.

__________
Background

The schema of the main table of the reference databases was changed to
accommodate citation keys. This additional column has to be created
and filled with suitable values before you can use existing databases
with RefDB 0.9.

__________
Procedures

There's two ways to update your reference databases:

Procedure 1: Dump all entries to a file and re-create the database
from scratch using the new RefDB version.

Procedure 2: Alter the table and fill in default values using a SQL
script.


Procedure 1:
------------

Pros:
- Re-creating the database from scratch will fill in citation keys
automatically, using the pattern AuthorYearSuffix, where Author is the
first author, Year is the publication year, and Suffix is a sequential
string to make the citation key unique if necessary.

- If you feel like switching from MySQL to PostgreSQL, this is the
easiest way to move on.

Cons:
- If you ever deleted references from your database, this procedure
will not re-create the same numeric ID for all references after the
"holes". If you rely on the association between the numeric ID and your
paper copies or if you use documents with RefDB bibliographies, you'll
get into trouble. BTW the "hole" problem was fixed in 0.9 so once you
get there you're safe.

Steps:

1) Dump your data
Use the refdbc command "getref -t ris -o all.ris :ID:>0"
to retrieve all references and save them to a file. For backup
purposes you should also create a SQL dump with the mysqldump tool,
just in case.
To get around the "holes" problem, you can manually edit the resulting
RIS file and insert new references from your queue wherever the ID
sequence is not consecutive. However, this is advisable only for small
collections and soon becomes a PITA.

2) Remove the database
Use the refdba command deletedb. 

3) Install 0.9

4) Create the database
Use the refdba command createdb to create a fresh database.

5) Restore your data
Use the refdbc command addref to add your reference data to the new
database.


Procedure 2:
------------

Pros:
- A single command will fix your existing database

- No problem with "holes" whatsoever

Cons:
- The automatically created citation keys follow the somewhat less
flashy pattern ID-Year, where ID is the existing unique ID and Year is
the publication year. 

Steps:

1) Backup your data
Use mysqldump to backup your reference database, just in case (you did
that on a regular schedule anyway, didn't you?).

2) Install 0.9

3) Run "mysql -u root -p dbname < refdb-upgrade.sql" where root is the
username of the database administrator and dbname is the name of your
reference database. The SQL script refdb-upgrade.sql is installed in
/usr/local/share/refdb/sql or an equivalent location.

Send questions about these procedures to refdb-users@lists.sourceforge.net