3
PDL::IO::FastRaw -- A simple, fast and convenient io format for PerlDL.
10
writefraw($pdl,"fname"); # write a raw file
12
$pdl2 = readfraw("fname"); # read a raw file
13
$pdl2 = PDL->readfraw("fname");
15
$pdl3 = mapfraw("fname2",{ReadOnly => 1}); # mmap a file, don't read yet
17
$pdl4 = maptextfraw("fname3",{...}); # map a text file into a 1-D pdl.
22
This is a very simple and fast io format for PerlDL.
23
The disk data consists of two files, a header metadata file
24
in ASCII and a binary file consisting simply of consecutive
25
bytes, shorts or whatever.
27
It is hoped that this will not only make for a simple PerlDL module
28
for saving and retrieving these files but also make it easy
29
for other programs to use these files.
31
The format of the ASCII header is simply
37
The binary files are in general
38
NOT interchangeable between different architectures since the binary
39
file is simply dumped from the memory region of the piddle.
40
This is what makes the approach efficient.
42
It is also possible to mmap the file which can give a large
43
speedup in certain situations as well as save a lot of memory
44
by using a disk file as virtual memory. When a file is mapped,
45
parts of it are read only as they are accessed in the memory
46
(or as the kernel decides: if you are reading the pages in order,
47
it may well preread some for you).
49
Note that memory savings and copy-on-write are operating-system
50
dependent - see Core.xs and your operating system documentation
51
for exact semantics of whatever. Basically, if you write to a
52
mmapped file without C<ReadOnly>, the change will be reflected
53
in the file immediately. C<ReadOnly> doesn't really make it impossible
54
to write to the piddle but maps the memory privately so the file
55
will not be changed when you change the piddle. Be aware though
56
that mmapping a 40Mb file without C<ReadOnly> spends no virtual
57
memory but with C<ReadOnly> it does reserve 40Mb.
65
Read a raw format binary file
69
$pdl2 = readfraw("fname");
70
$pdl2 = PDL->readfraw("fname");
77
Write a raw format binary file
81
writefraw($pdl,"fname");
88
Memory map a raw format binary file (see the module docs also)
92
$pdl3 = mapfraw("fname2",{ReadOnly => 1});
96
The C<mapfraw> command
97
supports the following options (not all combinations make sense):
103
If creating a new file or if you want to specify your own header
104
data for the file, you can give an array reference and a scalar,
109
Create the file. Also writes out a header for the file.
113
Set the file size. Automatically enabled with C<Creat>. NOTE: This also
114
clears the file to all zeroes.
118
Disallow writing to the file.
126
Memory map a text file (see the module docs also).
128
Note that this function maps the raw format so if you are
129
using an operating system which does strange things to e.g.
130
line delimiters upon reading a text file, you get the raw (binary)
133
The file doesn't really need to be text but it is just mapped
134
as one large binary chunk.
136
This function is just a convenience wrapper which firsts C<stat>s
137
the file and sets the dimensions and datatype.
141
$pdl4 = maptextfraw("fname", {options}
145
The options other than Dims, Datatype of C<mapfraw> are
150
Should be documented better. C<writefraw> and C<readfraw> should
151
also have options (the author nowadays only uses C<mapfraw> ;)
155
Copyright (C) Tuomas J. Lukka 1997.
156
All rights reserved. There is no warranty. You are allowed
157
to redistribute this software / documentation under certain
158
conditions. For details, see the file COPYING in the PDL
159
distribution. If this file is separated from the PDL distribution,
160
the copyright notice should be included in the file.
165
package PDL::IO::FastRaw;
172
@PDL::IO::FastRaw::ISA = qw/PDL::Exporter/;
174
@EXPORT_OK = qw/writefraw readfraw mapfraw maptextfraw/;
175
%EXPORT_TAGS = (Func=>[@EXPORT_OK]);
179
*writefraw = \&PDL::writefraw;
180
sub readfraw {PDL->readfraw(@_)}
181
sub mapfraw {PDL->mapfraw(@_)}
182
sub maptextfraw {PDL->maptextfraw(@_)}
186
my $hname = "$name.hdr";
187
my $h = new FileHandle "$hname"
188
or barf "Couldn't open '$hname' for reading";
191
my $str = <$h>; if(!defined $str) {barf("Format error in '$hname'");}
192
my @dims = split ' ',$str;
193
if($#dims != $ndims-1) {
194
barf("Format error reading fraw header file '$hname'");
205
my $hname = "$name.hdr";
206
my $h = new FileHandle ">$hname"
207
or barf "Couldn't open '$hname' for writing";
208
print $h map {"$_\n"} ($pdl->get_datatype,
209
$pdl->getndims, (join ' ',$pdl->dims));
213
my($pdl,$name,$opts) = @_;
214
_writefrawhdr($pdl,$name);
215
my $d = new FileHandle ">$name"
216
or barf "Couldn't open '$name' for writing";
218
print $d ${$pdl->get_dataref};
223
my($name,$opts) = @_;
224
my $d = new FileHandle "$name"
225
or barf "Couldn't open '$name' for reading";
227
my $hdr = _read_frawhdr($name);
228
my $pdl = $class->zeroes ((new PDL::Type($hdr->{Type})), @{$hdr->{Dims}});
229
my $len = length ${$pdl->get_dataref};
231
# $d->sysread(${$pdl->get_dataref},$len) == $len
232
# or barf "Couldn't read enough data from '$name'";
236
while (($retlen = $d->sysread($data, $len)) != 0) {
237
substr(${$pdl->get_dataref},$index,$len) = $data;
247
my($name,$opts) = @_;
250
my $datatype = $opts->{Datatype};
251
if(!defined $datatype) {$datatype = $PDL_D;}
252
$hdr->{Type} = $datatype;
253
$hdr->{Dims} = $opts->{Dims};
254
$hdr->{NDims} = scalar(@{$opts->{Dims}});
256
$hdr = _read_frawhdr($name);
258
$s = PDL::Core::howbig($hdr->{Type});
259
for(@{$hdr->{Dims}}) {
262
my $pdl = $class->zeroes(new PDL::Type($hdr->{Type}));
264
$pdl->setdims($hdr->{Dims});
266
$pdl->set_data_by_mmap($name,$s,1,($opts->{ReadOnly}?0:1),
267
($opts->{Creat}?1:0),
269
($opts->{Creat} || $opts->{Trunc} ? 1:0));
272
_writefrawhdr($pdl,$name);
277
sub PDL::maptextfraw {
278
my($class, $name, $opts) = @_;
279
$opts = {%$opts}; # Copy just in case
281
$opts->{Dims} = [$s[7]];
282
$opts->{Datatype} = &PDL::byte;
283
return PDL::mapfraw($class, $name, $opts);
added
removed
IO/FastRaw/FastRaw.pm
