1
DISK RESIDENT ARRAYS PRELIMINARY DOCUMENTATION
2
Jarek Nieplocha, 05.10.95
5
Disk Resident Arrays extend Global Arrays NUMA programming model with
6
disk I/O operations. Either whole or sections of global arrays
7
can be transferred between GA memory and the disk.
9
At present time, all operations are declared to be collective.
10
This simplifies implementation on machines where only some processors
11
are connected to I/O devices.
13
Except where stated otherwise, all operations are synchronous (blocking)
14
which means that control is returned to the calling process only after
15
the requested operation completes.
17
All operations return error code.
24
A program that uses Disk Resident Arrays should look like:
31
call pbeginf() ! initialize TCGMSG
32
if(.not. ma_init(...)) ERROR ! initialize MA
33
call ga_initialize() ! initialize Global Arrays
34
if(dra_init(....).ne.0) ERROR ! initialize Disk Arrays
38
if(dra_terminate().ne.0)ERROR ! destroy DRA internal data structures
39
call ga_terminate ! terminate Global Arrays
40
call pend() ! terminate TCGMSG
45
List of DRA operations:
48
status = dra_init(max_arrays, max_array_size, total_disk_space, max_memory)
49
integer max_arrays [input]
50
double precision max_array_size [input]
51
double precision total_disk_space [input]
52
double precision max_memory [input]
56
Initializes disk resident array I/O subsystem.
58
"max_array_size", "total_disk_space" and "max_memory" are given
61
"max_memory" specifies how much local memory per processor the
62
application is willing to provide to the DRA I/O subsystem for
65
The value of "-1" for any of input arguments means:
66
"don't care", "don't know", or "use defaults"
70
status = dra_terminate()
74
Close all open disk resident arrays and shut down
79
status = dra_create(type, dim1, dim2, name, filename, mode,rdim1,rdim2,d_a)
80
integer type [input] ! MA type identifier
83
character*(*) name [input]
84
character*(*) filename [input]
88
integer d_a [output] ! DRA handle
92
Creates new disk resident array with specified dimensions
95
String "filename" specifies name of an abstract
96
meta-file that will store the data on the disk. The
97
meta-file might be implemented as multiple disk files
98
that will contain parts of the disk resident array. The component
99
files will have names derived from the string "filename"
100
according to some established scheme(s).
102
Only one DRA object can be stored in DRA meta-file identified by
105
DRA objects persist on the disk after calling dra_close().
106
dra_delete() should be used instead of dra_close() to delete disk
107
array and associated meta-file on the disk.
109
A set of UNIX commands that understand "filename" identifier
110
will be provided to copy, rename, move, change access attributes,
111
and delete DRA meta-files.
113
String "name" can be used for more informative (longer)names.
115
Disk array is implicitly initialized to "0".
117
Access permissions (read, write, read&write) are set in "mode".
118
These are set using defined in DRA.fh (Fortran) and DRA.h (C)
119
preprocessor constants: DRA_R, DRA_W, DRA_RW.
121
The pair [rdim1, rdim2] specifies dimensions of a
122
"typical" request. The value of "-1" for either of them
123
means "unspecified". The layout of the data on the
124
disk(s) is determined based on the values of these
125
arguments. Performance of the DRA operations will depend
126
on the dimensions (section shape) of the requests. If
127
data layout is optimized for "column-like" sections,
128
performance of DRA operations for "row-like" sections
129
might be seriously degraded. This is analogous to the
130
effect of wrong loop order yielding frequent cache misses
131
on RISC processors in the example below.
134
double precision a(1000, 1000)
151
status = dra_open(filename, mode, d_a)
152
character*(*) filename [input]
154
integer d_a [output] ! DRA handle
158
Open and assign DRA handle to disk resident array stored in DRA
159
meta-file "filename". Disk arrays that are created
160
with 'dra_create' and saved by calling 'dra_close' can be
161
later opened and accessed by the same or different
164
Attributes of disk resident array can be found by calling
169
status = dra_write(g_a, d_a, request)
170
integer g_a [input] ! GA handle
171
integer d_a [input] ! DRA handle
172
integer request [output] ! request id
176
Write asynchronously specified global array to specified
179
Dimensions and type of g_a and d_a must match. If dimensions
180
don't match, 'dra_write_section' should be used instead.
182
The operation is by definition asynchronous but it might
183
be implemented as synchronous i.e., it would return only
186
"request" can be used to 'dra_probe' or 'dra_wait' for completion.
190
status = dra_write_section(transp, g_a, gilo, gihi, gjlo, gjhi,
191
d_a, dilo, dihi, djlo, djhi, request)
192
logical transp [input] ! transpose operator
193
integer g_a [input] ! GA handle
194
integer d_a [input] ! DRA handle
203
integer request [output] ! request id
208
Write asynchronously specified global array section to
209
specified disk resident array section:
210
OP(g_a[ gilo:gihi, gjlo:gjhi]) --> d_a[ dilo:dihi, djlo:djhi]
211
where OP is the transpose operator (.true./.false.).
213
Return error if the two section's types or sizes mismatch.
215
See 'dra_write' specs for discussion of "request".
218
Section reshaping and transpose operation not implemented yet.
221
status = dra_read(g_a, d_a, request)
222
integer g_a [input] ! GA handle
223
integer d_a [input] ! DRA handle
224
integer request [output] ! request id
228
Read asynchronously specified global array from specified
231
Dimensions and type of g_a and d_a must match. If dimensions
232
don't match, 'dra_read_section' could be used instead.
234
See 'dra_write' specs for discussion of "request".
237
status = dra_read_section(transp, g_a, gilo, gihi, gjlo, gjhi,
238
d_a, dilo, dihi, djlo, djhi, request)
239
logical transp [input] ! transpose operator
240
integer g_a [input] ! GA handle
241
integer d_a [input] ! DRA handle
250
integer request [output] ! request id
254
Read asynchronously specified global array section from
255
specified disk resident array section:
256
OP(d_a[ dilo:dihi, djlo:djhi]) --> g_a[ gilo:gihi, gjlo:gjhi]
257
where OP is the transpose operator (.true./.false.).
259
See 'dra_write' specs for discussion of "request".
262
Section reshaping and transpose operation not implemented yet.
265
status = dra_probe(request, compl_status)
266
integer request [input] ! request id
267
integer compl_status [output] ! completion status
271
Tests for completion of 'dra_write/read' or
272
'dra_write/read_section' operation which set the value
273
passed in "request" argument.
275
compl_status = 0 the operation has been completed
276
compl_status <> 0 not done yet
280
status = dra_wait(request)
281
integer request [input] ! request id
285
Blocks until completion of 'dra_write/read' or
286
'dra_write/read_section' operation which set the value
287
passed in "request" argument.
291
status = dra_inquire(d_a, type, dim1, dim2, name, filename)
292
integer d_a [input] ! DRA handle
293
integer type [output]
294
integer dim1 [output]
295
integer dim2 [output]
296
character*(*) name [output]
297
character*(*) filename [output]
301
Return dimensions, "type", "name" of disk resident array,
302
and "filename" of DRA meta-file associated with "d_a"
306
status = dra_delete(d_a)
307
integer d_a [input] ! DRA handle
311
Delete a disk resident array associated with "d_a" handle.
315
The corresponding DRA meta-file is destroyed.
319
status = dra_close(d_a)
320
integer d_a [input] ! DRA handle
324
Close DRA meta-file associated with "d_a" handle and
325
deallocate data structures corresponding to this disk
326
array. Invalidate "d_a" handle. The array on the disk is
330
subroutine dra_flick()
334
Returns control to DRA for a VERY short time to improve
335
progress of pending asynchronous operations.