2
ULOOP -- Loopback block device in userspace
3
(and a sample for HTTP and generic block device)
6
# $Id: 00readme.txt,v 1.3 2007/11/26 01:35:40 sfjro Exp $
10
As you know, there is a Loopback block device in Linux, /dev/loop,
11
which enables you to mount a fs-image local file.
12
Also it can adopt a userspace program, such as cryptloop.
13
This sample ULOOP driver makes it generic, and enables to adopt any
15
You can give an empty or non-existing file to /dev/loop backend.
16
When a process reads from /dev/loop, this dirver wakes a user process
17
up and passes the I/O transaction to it. A user process makes the
18
required block ready and tells the driver. Then the driver completes
20
Also there is sample scripts or usage for diskless nodes working with
21
aufs. This driver may work with it well.
22
The name is unrelated to YouTube. :-)
26
- int ulo_init(struct ulo_init *init);
30
unsigned long long size;
32
enum {ULO_DEV, ULO_CACHE, ULO_BITMAP, ULO_Last};
34
Initializes ULOOP driver. All members in struct ulo_init must be set
35
before you call ulo_init().
37
pathname of loopback device such as "/dev/loopN".
39
pathname of a cache file. A userspace program stores the
40
real data to this file.
42
pathname of a bitmap file. The ULOOP driver sets the bit
43
which is corresponding the block number when the block is
44
filled by a userspace program. When the bit is not set,
45
ULOOP driver invokes the userspace program.
47
Flags for open(2) of path[ULO_DEV].
49
the size of real data. the ULOOP library set this size to
50
the cache file after creating it internally.
52
- int ulo_loop(int sig, ulo_cb_t store, void *arg);
53
typedef int (*ulo_cb_t)(unsigned long long start, int size, void *arg);
55
Waits for a I/O request from ULOOP driver. When a user accesses a
56
ULOOP device, ULOOP driver translates the request to the offset in
57
the cache file and the requested size, and invokes the user-defined
58
callback function which is specifed by `store.' The function `store'
59
must fill the data in the cache file following the given offset and
60
size. You can add an arugment `arg' for the callback function.
62
- extern const struct uloop *uloop;
66
unsigned long long tgt_size, cache_size;
69
A global variable in ULOOP library. Usually you will need
70
'ulo_cache_fd` only. See below.
71
#define ulo_dev_fd ({ uloop->fd[ULO_DEV]; })
72
#define ulo_cache_fd ({ uloop->fd[ULO_CACHE]; })
73
#define ulo_bitmap_fd ({ uloop->fd[ULO_BITMAP]; })
77
Simple 'make' will build ./drivers/block/uloop.ko and ./ulohttp.
78
Ulohttp application behaves like losetup(8). Additionally, ulohttp is
79
an actual daemon which handles I/O request.
82
ulohttp [-b bitmap] [-c cache] device URL
84
The device is /dev/loopN and the URL is a URL for fs-image file via
85
HTTP. The http server must support byte range (Range: header).
86
The bitmap is a new filename or previously specified as the bitmap for
87
the same URL. Its filesize will be 'the size of the specified fs-image
88
/ pagesize (usually 4k) / bits in a byte (8)', and round-up to
90
The cache is a new filename or previously specified as the cache for
91
the same URL. Its filesize will be 'the size of the specified
92
fs-image', and round-up to pagesize.
93
Note that both the bitmap and the cache are re-usable as long as you
94
don't change the filedata and URL.
96
When someone reads from the specified /dev/loopN, or accesses a file
97
on a filesystem after mounting /dev/loopN, ULOOP driver first checks
98
the corresponding bit in the bitmap file. When the bit is not set,
99
which means the block is not retreived yet, it passes the offset and
100
size of the I/O request to ulohttp daemon.
101
Ulohttp converts the offset and the size into HTTP GET request with
102
Range header and send it to the http server.
103
Retriving the data from the http server, ulohttp stores it to the
104
cache file, and tells ULOOP driver that the HTTP transfer completes.
105
Then the ULOOP driver sets the corresponding bit in the bitmap, and
106
finishes the I/O/request.
108
In other words, it is equivalent to this operation.
109
$ wget URL_for_fsimage
110
$ sudo mount -o loop retrieved_fsimage /mnt
111
But ULOOP driver and ulohttp retrives only the data (block) on-demand,
112
and stores into the cache file. The first access to a block is slow
113
since it involves HTTP GET, but the next access to the same block is
114
fast since it is in the local cache file. In this case, the behaviour
115
is equivalent to the simple /dev/loop device.
118
- ulohttp requires libcurl.
119
- ulohttp doesn't support HTTP PUT or POST, so the device rejects
121
- ulohttp doesn't have a smart exit routine.
122
- This sample is "proof-of-concepts", do not expect the maturity level
124
- This driver and the sample is developed and tested on linux-2.6.21.3.
125
- If you impelment other protocols such like nbd/enbd, iscsi, aoe or
126
somehting, instead of http, I guess it will be fantastic. :-)
131
$ sudo insmod ./drivers/block/uloop.ko
133
$ ./ulohttp -b /tmp/b -c /tmp/c $dev http://whatever/you/like
134
$ sudo mount -o ro $dev /mnt
139
$ sudo losetup -d $dev
142
3. sample for generic block device
143
The sample `ulohttp' (above) retrives data from a remote host via
144
HTTP, and stores it into a local file as a cache. It means you can
145
reduce the network traffic and the workload on a remote server.
146
As you can guess easily, this scheme is also effective to a local disk
147
device, espcially when you want to make your disk and spin down/off
148
it. Recent flash memory is getting larger and cheaper. You can cache
149
the whole contents of your harddirve into a file on your flash.
150
Here is a sample for it, `ulobdev.' The basic usage is very similar to
151
`ulohttp'. See above.
152
Of course, it is available for remote block devices too, such as
153
nbd/enbd, iscsi and aoe.
155
You should not mount the backend block device as readwrite, since it
156
modifies the superblock of the filesystem on the block device even if
157
youu don't write anything to it.
159
Currently this sample supports readonly mode only.
160
If someone is interested in this approach and sample, I will add some
161
features which will support read/write mode and write-back to the
162
harddrive periodically, and discard/re-create the cache file.
added
removed
sample/uloop/00readme.txt
