1
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
4
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
5
<META NAME="GENERATOR" CONTENT="Mozilla/4.08 [en] (X11; I; FreeBSD 3.3-RELEASE i386) [Netscape]">
11
Client/Server Interface for Berkeley DB</H1></CENTER>
13
<CENTER><I>Susan LoVerso</I>
14
<BR><I>sue@sleepycat.com</I>
16
<BR><I>1999 Nov 29</I></CENTER>
18
<P>We provide an interface allowing client/server access to Berkeley DB.
19
Our goal is to provide a client and server library to allow users to separate
20
the functionality of their applications yet still have access to the full
21
benefits of Berkeley DB. The goal is to provide a totally seamless
22
interface with minimal modification to existing applications as well.
23
<P>The client/server interface for Berkeley DB can be broken up into several
24
layers. At the lowest level there is the transport mechanism to send
25
out the messages over the network. Above that layer is the messaging
26
layer to interpret what comes over the wire, and bundle/unbundle message
27
contents. The next layer is Berkeley DB itself.
28
<P>The transport layer uses ONC RPC (RFC 1831) and XDR (RFC 1832).
29
We declare our message types and operations supported by our program and
30
the RPC library and utilities pretty much take care of the rest.
32
<I>rpcgen</I> program generates all of the low level code needed.
33
We need to define both sides of the RPC.
36
<A NAME="DB Modifications"></A>DB Modifications</H2>
37
To achieve the goal of a seamless interface, it is necessary to impose
38
a constraint on the application. That constraint is simply that all database
39
access must be done through an open environment. I.e. this model
40
does not support standalone databases. The reason for this constraint
41
is so that we have an environment structure internally to store our connection
42
to the server. Imposing this constraint means that we can provide
43
the seamless interface just by adding a single environment method: <A HREF="../docs/api_c/env_set_server.html">DBENV->set_server()</A>.
44
<P>The planned interface for this method is:
45
<PRE>DBENV->set_server(dbenv, /* DB_ENV structure */
46
hostname /* Host of server */
47
cl_timeout, /* Client timeout (sec) */
48
srv_timeout,/* Server timeout (sec) */
49
flags); /* Flags: unused */</PRE>
50
This new method takes the hostname of the server, establishes our connection
51
and an environment on the server. If a server timeout is specified,
52
then we send that to the server as well (and the server may or may not
53
choose to use that value). This timeout is how long the server will
54
allow the environment to remain idle before declaring it dead and releasing
55
resources on the server. The pointer to the connection is stored
56
on the client in the DBENV structure and is used by all other methods to
57
figure out with whom to communicate. If a client timeout is specified,
58
it indicates how long the client is willing to wait for a reply from the
59
server. If the values are 0, then defaults are used. Flags
60
is currently unused, but exists because we always need to have a placeholder
61
for flags and it would be used for specifying authentication desired (were
62
we to provide an authentication scheme at some point) or other uses not
64
<P>This client code is part of the monolithic DB library. The user
65
accesses the client functions via a new flag to <A HREF="../docs/api_c/db_env_create.html">db_env_create()</A>.
66
That flag is DB_CLIENT. By using this flag the user indicates they
67
want to have the client methods rather than the standard methods for the
68
environment. Also by issuing this flag, the user needs to connect
69
to the server via the <A HREF="../docs/api_c/env_set_server.html">DBENV->set_server()</A>
71
<P>We need two new fields in the <I>DB_ENV </I>structure. One is
72
the socket descriptor to communicate to the server, the other field is
73
the client identifier the server gives to us. The <I>DB, </I>and<I>
74
DBC </I>only need one additional field, the client identifier. The
76
structure does not need modification, we are overloading the <I>txn_id
80
We need to figure out what to do in case of client and server crashes.
81
Both the client library and the server program are stateful. They
82
both consume local resources during the lifetime of the connection.
83
Should one end drop that connection, the other side needs to release those
85
<P>If the server crashes, then the client will get an error back.
86
I have chosen to implement time-outs on the client side, using a default
87
or allowing the application to specify one through the <A HREF="../docs/api_c/env_set_server.html">DBENV->set_server()</A>
88
method. Either the current operation will time-out waiting for the
89
reply or the next operation called will time out (or get back some other
90
kind of error regarding the server's non-existence). In any case,
91
if the client application gets back such an error, it should abort any
92
open transactions locally, close any databases, and close its environment.
93
It may then decide to retry to connect to the server periodically or whenever
94
it comes back. If the last operation a client did was a transaction
95
commit that did not return or timed out from the server, the client cannot
96
determine if the transaction was committed or not but must release the
97
local transaction resources. Once the server is back up, recovery must
98
be run on the server. If the transaction commit completed on
99
the server before the crash, then the operation is redone, if the transaction
100
commit did not get to the server, the pieces of the transaction are undone
101
on recover. The client can then re-establish its connection and begin
102
again. This is effectively like beginning over. The client
103
cannot use ID's from its previous connection to the server. However,
104
if recovery is run, then consistency is assured.
105
<P>If the client crashes, the server needs to somehow figure this out.
106
The server is just sitting there waiting for a request to come in.
107
A server must be able to time-out a client. Similar to ftpd, if a
108
connection is idle for N seconds, then the server decides the client is
109
dead and releases that client's resources, aborting any open transactions,
110
closing any open databases and environments. The server timing
111
out a client is not a trivial issue however. The generated function
112
for the server just calls <I>svc_run()</I>. The server code I write
113
contains procedures to do specific things. We do not have access
114
to the code calling <I>select()</I>. Timing out the select is not
115
good enough even if we could do so. We want to time-out idle environments,
116
not simply cause a time-out if the server is idle a while. See the
117
discussion of the <A HREF="#The Server Program">server program</A> for
118
a description of how we accomplish this.
119
<P>Since rpcgen generates the main() function of the server, I do not yet
120
know how we are going to have the server multi-threaded or multi-process
121
without changing the generated code. The RPC book indicates that
122
the only way to accomplish this is through modifying the generated code
123
in the server. <B>For the moment we will ignore this issue while
124
we get the core server working, as it is only a performance issue.</B>
125
<P>We do not do any security or authentication. Someone could get
126
the code and modify it to spoof messages, trick the server, etc.
127
RPC has some amount of authentication built into it. I haven't yet
128
looked into it much to know if we want to use it or just point a user at
129
it. The changes to the client code are fairly minor, the changes
130
to our server procs are fairly minor. We would have to add code to
131
a <I>sed</I> script or <I>awk</I> script to change the generated server
132
code (yet again) in the dispatch routine to perform authentication.
133
<P>We will need to get an official program number from Sun. We can
134
get this by sending mail to <I>rpc@sun.com</I> and presumably at some point
135
they will send us back a program number that we will encode into our XDR
136
description file. Until we release this we can use a program number
137
in the "user defined" number space.
140
<A NAME="The Server Program"></A>The Server Program</H2>
141
The server is a standalone program that the user builds and runs, probably
142
as a daemon like process. This program is linked against the Berkeley
143
DB library and the RPC library (which is part of the C library on my FreeBSD
144
machine, others may have/need <I>-lrpclib</I>). The server basically
145
is a slave to the client process. All messages from the client are
146
synchronous and two-way. The server handles messages one at a time,
147
and sends a reply back before getting another message. There are
148
no asynchronous messages generated by the server to the client.
149
<P>We have made a choice to modify the generated code for the server.
150
The changes will be minimal, generally calling functions we write, that
151
are in other source files. The first change is adding a call to our
152
time-out function as described below. The second change is changing
153
the name of the generated <I>main()</I> function to <I>__dbsrv_main()</I>,
154
and adding our own <I>main()</I> function so that we can parse options,
155
and set up other initialization we require. I have a <I>sed</I> script
156
that is run from the distribution scripts that massages the generated code
157
to make these minor changes.
158
<P>Primarily the code needed for the server is the collection of the specified
159
RPC functions. Each function receives the structure indicated, and
160
our code takes out what it needs and passes the information into DB itself.
161
The server needs to maintain a translation table for identifiers that we
162
pass back to the client for the environment, transaction and database handles.
163
<P>The table that the server maintains, assuming one client per server
164
process/thread, should contain the handle to the environment, database
165
or transaction, a link to maintain parent/child relationships between transactions,
166
or databases and cursors, this handle's identifier, a type so that we can
167
error if the client passes us a bad id for this call, and a link to this
168
handle's environment entry (for time out/activity purposes). The
169
table contains, in entries used by environments, a time-out value and an
170
activity time stamp. Its use is described below for timing out idle
172
<P>Here is how we time out clients in the server. We have to modify
173
the generated server code, but only to add one line during the dispatch
174
function to run the time-out function. The call is made right before
175
the return of the dispatch function, after the reply is sent to the client,
176
so that client's aren't kept waiting for server bookkeeping activities.
177
This time-out function then runs every time the server processes a request.
178
In the time-out function we maintain a time-out hint that is the youngest
179
environment to time-out. If the current time is less than the hint
180
we know we do not need to run through the list of open handles. If
181
the hint is expired, then we go through the list of open environment handles,
182
and if they are past their expiration, then we close them and clean up.
183
If they are not, we set up the hint for the next time.
184
<P>Each entry in the open handle table has a pointer back to its environment's
185
entry. Every operation within this environment can then update the
186
single environment activity record. Every environment can have a
187
different time-out. The <A HREF="../docs/api_c/env_set_server.html">DBENV->set_server
189
takes a server time-out value. If this value is 0 then a default
190
(currently 5 minutes) is used. This time-out value is only a hint
191
to the server. It may choose to disregard this value or set the time-out
192
based on its own implementation.
193
<P>For completeness, the flaws of this time-out implementation should be
194
pointed out. First, it is possible that a client could crash with
195
open handles, and no other requests come in to the server. Therefore
196
the time-out function never gets run and those resources are not released
197
(until a request does come in). Similarly, this time-out is not exact.
198
The time-out function uses its hint and if it computes a hint on one run,
199
an earlier time-out might be created before that time-out expires.
200
This issue simply yields a handle that doesn't get released until that
201
original hint expires. To illustrate, consider that at the time that
202
the time-out function is run, the youngest time-out is 5 minutes in the
203
future. Soon after, a new environment is opened that has a time-out
204
of 1 minute. If this environment becomes idle (and other operations
205
are going on), the time-out function will not release that environment
206
until the original 5 minute hint expires. This is not a problem since
207
the resources will eventually be released.
208
<P>On a similar note, if a client crashes during an RPC, our reply generates
209
a SIGPIPE, and our server crashes unless we catch it. Using <I>signal(SIGPIPE,
210
SIG_IGN) </I>we can ignore it, and the server will go on. This is
211
a call in our <I>main()</I> function that we write. Eventually
212
this client's handles would be timed out as described above. We need
213
this only for the unfortunate window of a client crashing during the RPC.
214
<P>The options below are primarily for control of the program itself,.
215
Details relating to databases and environments should be passed from the
216
client to the server, since the server can serve many clients, many environments
217
and many databases. Therefore it makes more sense for the client
218
to set the cache size of its own environment, rather than setting a default
219
cachesize on the server that applies as a blanket to any environment it
220
may be called upon to open. Options are:
223
<B>-t </B> to set the default time-out given to an environment.</LI>
226
<B>-T</B> to set the maximum time-out allowed for the server.</LI>
229
<B>-L</B> to log the execution of the server process to a specified file.</LI>
232
<B>-v</B> to run in verbose mode.</LI>
235
<B>-M</B> to specify the maximum number of outstanding child server
236
processes/threads we can have at any given time. The default is 10.
238
are not yet doing multiple threads/processes.]</B></LI>
243
The client code contains all of the supported functions and methods used
244
in this model. There are several methods in the <I>__db_env
247
structures that currently do not apply, such as the callbacks. Those
248
fields that are not applicable to the client model point to NULL to notify
249
the user of their error. Some method functions remain unchanged,
250
as well such as the error calls.
251
<P>The client code contains each method function that goes along with the
252
<A HREF="#Remote Procedure Calls">RPC
253
calls</A> described elsewhere. The client library also contains its
254
own version of <A HREF="../docs/api_c/env_create.html">db_env_create()</A>,
255
which does not result in any messages going over to the server (since we
256
do not yet know what server we are talking to). This function sets
257
up the pointers to the correct client functions.
258
<P>All of the method functions that handle the messaging have a basic flow
262
Local arg parsing that may be needed</LI>
265
Marshalling the message header and the arguments we need to send to the
269
Sending the message</LI>
272
Receiving a reply</LI>
275
Unmarshalling the reply</LI>
278
Local results processing that may be needed</LI>
283
Almost all of the code is generated from a source file describing the interface
284
and an <I>awk</I> script. This awk script generates six (6)
285
files for us. It also modifies one. The files are:
288
Client file - The C source file created containing the client code.</LI>
291
Client template file - The C template source file created containing interfaces
292
for handling client-local issues such as resource allocation, but with
293
a consistent interface with the client code generated.</LI>
296
Server file - The C source file created containing the server code.</LI>
299
Server template file - The C template source file created containing interfaces
300
for handling server-local issues such as resource allocation, calling into
301
the DB library but with a consistent interface with the server code generated.</LI>
304
XDR file - The XDR message description file created.</LI>
307
Server sed file - A sed script that contains commands to apply to the server
308
procedure file (i.e. the real source file that the server template file
309
becomes) so that minor interface changes can be consistently and easily
310
applied to the real code.</LI>
313
Server procedure file - This is the file that is modified by the sed script
314
generated. It originated from the server template file.</LI>
316
The awk script reads a source file, <I>db_server/rpc.src </I>that describes
317
each operation and what sorts of arguments it takes and what it returns
318
from the server. The syntax of the source file describes the interface
319
to that operation. There are four (4) parts to the syntax:
322
<B>BEGIN</B> <B><I>function version# codetype</I></B> - begins a new functional
323
interface for the given <B><I>function</I></B>. Each function has
324
a <B><I>version number</I></B>, currently all of them are at version number
325
one (1). The <B><I>code type</I></B> indicates to the awk script
326
what kind of code to generate. The choices are:</LI>
330
<B>CODE </B>- Generate all code, and return a status value. If specified,
331
the client code will simply return the status to the user upon completion
332
of the RPC call.</LI>
335
<B>RETCODE </B>- Generate all code and call a return function in the client
336
template file to deal with client issues or with other returned items.
337
If specified, the client code generated will call a function of the form
338
<I>__dbcl_<name>_ret()
340
<name> is replaced with the function name given here. This function
341
is placed in the template file because this indicates that something special
342
must occur on return. The arguments to this function are the same
343
as those for the client function, with the addition of the reply message
347
<B>NOCLNTCODE - </B>Generate XDR and server code, but no corresponding
348
client code. (This is used for functions that are not named the same thing
349
on both sides. The only use of this at the moment is db_env_create
350
and db_create. The environment create call to the server is actually
351
called from the <A HREF="../docs/api_c/env_set_server.html">DBENV->set_server()</A>
352
method. The db_create code exists elsewhere in the library and we
353
modify that code for the client call.)</LI>
357
<B>ARG <I>RPC-type C-type varname [list-type]</I></B>- each line of this
358
describes an argument to the function. The argument is called <B><I>varname</I></B>.
359
The <B><I>C-type</I></B> given is what it should look like in the C code
360
generated, such as <B>DB *, u_int32_t, const char *</B>. The
361
<B><I>RPC-type</I></B>
362
is an indication about how the RPC request message should be constructed.
363
The RPC-types allowed are described below.</LI>
366
<B>RET <I>RPC-type C-type varname [list-type]</I></B>- each line of this
367
describes what the server should return from this procedure call (in addition
368
to a status, which is always returned and should not be specified).
369
The argument is called <B><I>varname</I></B>. The <B><I>C-type</I></B>
370
given is what it should look like in the C code generated, such as <B>DB
371
*, u_int32_t, const char *</B>. The <B><I>RPC-type</I></B> is an
372
indication about how the RPC reply message should be constructed.
373
The RPC-types are described below.</LI>
376
<B>END </B>- End the description of this function. The result is
377
that when the awk script encounters the <B>END</B> tag, it now has all
378
the information it needs to construct the generated code for this function.</LI>
380
The <B><I>RPC-type</I></B> must be one of the following:
383
<B>IGNORE </B>- This argument is not passed to the server and should be
384
ignored when constructing the XDR code. <B>Only allowed for an ARG
385
specfication.</B></LI>
388
<B>STRING</B> - This argument is a string.</LI>
391
<B>INT </B>- This argument is an integer of some sort.</LI>
394
<B>DBT </B>- This argument is a DBT, resulting in its decomposition into
395
the request message.</LI>
398
<B>LIST</B> - This argument is an opaque list passed to the server (NULL-terminated).
399
If an argument of this type is given, it must have a <B><I>list-type</I></B>
400
specified that is one of:</LI>
414
<B>ID</B> - This argument is an identifier.</LI>
416
So, for example, the source for the DB->join RPC call looks like:
417
<PRE>BEGIN dbjoin 1 RETCODE
418
ARG ID DB * dbp
419
ARG LIST DBC ** curs ID
420
ARG IGNORE DBC ** dbcpp
421
ARG INT u_int32_t flags
422
RET ID long dbcid
424
Our first line tells us we are writing the dbjoin function. It requires
425
special code on the client so we indicate that with the RETCODE.
426
This method takes four arguments. For the RPC request we need the
427
database ID from the dbp, we construct a NULL-terminated list of IDs for
428
the cursor list, we ignore the argument to return the cursor handle to
429
the user, and we pass along the flags. On the return, the reply contains
430
a status, by default, and additionally, it contains the ID of the newly
433
Building and Installing</H2>
434
I need to verify with Don Anderson, but I believe we should just build
435
the server program, just like we do for db_stat, db_checkpoint, etc.
436
Basically it can be treated as a utility program from the building and
437
installation perspective.
438
<P>As mentioned early on, in the section on <A HREF="#DB Modifications">DB
439
Modifications</A>, we have a single library, but allowing the user to access
440
the client portion by sending a flag to <A HREF="../docs/api_c/env_create.html">db_env_create()</A>.
441
The Makefile is modified to include the new files.
442
<P>Testing is performed in two ways. First I have a new example program,
443
that should become part of the example directory. It is basically
444
a merging of ex_access.c and ex_env.c. This example is adequate to
445
test basic functionality, as it does just does database put/get calls and
446
appropriate open and close calls. However, in order to test the full
447
set of functions a more generalized scheme is required. For the moment,
448
I am going to modify the Tcl interface to accept the server information.
449
Nothing else should need to change in Tcl. Then we can either write
450
our own test modules or use a subset of the existing ones to test functionality
added
removed
db/rpc_server/clsrv.html
