5
By default, the content of an object cannot be greater than 5 GB.
6
However, you can use a number of smaller objects to construct a large
7
object. The large object is comprised of two types of objects:
9
- **Segment objects** store the object content. You can divide your
10
content into segments, and upload each segment into its own segment
11
object. Segment objects do not have any special features. You create,
12
update, download, and delete segment objects just as you would normal
15
- A **manifest object** links the segment objects into one logical
16
large object. When you download a manifest object, Object Storage
17
concatenates and returns the contents of the segment objects in the
18
response body of the request. This behavior extends to the response
19
headers returned by **GET** and **HEAD** requests. The
20
``Content-Length`` response header value is the total size of all
21
segment objects. Object Storage calculates the ``ETag`` response
22
header value by taking the ``ETag`` value of each segment,
23
concatenating them together, and returning the MD5 checksum of the
24
result. The manifest object types are:
26
**Static large objects**
27
The manifest object content is an ordered list of the names of
28
the segment objects in JSON format.
30
**Dynamic large objects**
31
The manifest object has no content but it has a
32
``X-Object-Manifest`` metadata header. The value of this header
33
is ``{container}/{prefix}``, where ``{container}`` is the name of
34
the container where the segment objects are stored, and
35
``{prefix}`` is a string that all segment objects have in common.
40
If you make a **COPY** request by using a manifest object as the source,
41
the new object is a normal, and not a segment, object. If the total size
42
of the source segment objects exceeds 5 GB, the **COPY** request fails.
43
However, you can make a duplicate of the manifest object and this new
44
object can be larger than 5 GB.
49
To create a static large object, divide your content into pieces and
50
create (upload) a segment object to contain each piece.
52
You must record the ``ETag`` response header that the **PUT** operation
53
returns. Alternatively, you can calculate the MD5 checksum of the
54
segment prior to uploading and include this in the ``ETag`` request
55
header. This ensures that the upload cannot corrupt your data.
57
List the name of each segment object along with its size and MD5
60
Create a manifest object. Include the *``?multipart-manifest=put``*
61
query string at the end of the manifest object name to indicate that
62
this is a manifest object.
64
The body of the **PUT** request on the manifest object comprises a json
65
list, where each element contains the following attributes:
67
- ``path``. The container and object name in the format:
68
``{container-name}/{object-name}``
70
- ``etag``. The MD5 checksum of the content of the segment object. This
71
value must match the ``ETag`` of that object.
73
- ``size_bytes``. The size of the segment object. This value must match
74
the ``Content-Length`` of that object.
76
**Example Static large object manifest list**
78
This example shows three segment objects. You can use several containers
79
and the object names do not have to conform to a specific pattern, in
80
contrast to dynamic large objects.
86
"path": "mycontainer/objseg1",
87
"etag": "0228c7926b8b642dfb29554cd1f00963",
91
"path": "mycontainer/pseudodir/seg-obj2",
92
"etag": "5bfc9ea51a00b790717eeb934fb77b9b",
96
"path": "other-container/seg-final",
97
"etag": "b9c3da507d2557c1ddc51f27c54bae51",
104
The ``Content-Length`` request header must contain the length of the
105
json content—not the length of the segment objects. However, after the
106
**PUT** operation completes, the ``Content-Length`` metadata is set to
107
the total length of all the object segments. A similar situation applies
108
to the ``ETag``. If used in the **PUT** operation, it must contain the
109
MD5 checksum of the json content. The ``ETag`` metadata value is then
110
set to be the MD5 checksum of the concatenated ``ETag`` values of the
111
object segments. You can also set the ``Content-Type`` request header
112
and custom object metadata.
114
When the **PUT** operation sees the *``?multipart-manifest=put``* query
115
parameter, it reads the request body and verifies that each segment
116
object exists and that the sizes and ETags match. If there is a
117
mismatch, the **PUT**\ operation fails.
119
If everything matches, the manifest object is created. The
120
``X-Static-Large-Object`` metadata is set to ``true`` indicating that
121
this is a static object manifest.
123
Normally when you perform a **GET** operation on the manifest object,
124
the response body contains the concatenated content of the segment
125
objects. To download the manifest list, use the
126
*``?multipart-manifest=get``* query parameter. The resulting list is not
127
formatted the same as the manifest you originally used in the **PUT**
130
If you use the **DELETE** operation on a manifest object, the manifest
131
object is deleted. The segment objects are not affected. However, if you
132
add the *``?multipart-manifest=delete``* query parameter, the segment
133
objects are deleted and if all are successfully deleted, the manifest
134
object is also deleted.
136
To change the manifest, use a **PUT** operation with the
137
*``?multipart-manifest=put``* query parameter. This request creates a
138
manifest object. You can also update the object metadata in the usual
141
Dynamic large objects
142
~~~~~~~~~~~~~~~~~~~~~
144
You must segment objects that are larger than 5 GB before you can upload
145
them. You then upload the segment objects like you would any other
146
object and create a dynamic large manifest object. The manifest object
147
tells Object Storage how to find the segment objects that comprise the
148
large object. The segments remain individually addressable, but
149
retrieving the manifest object streams all the segments concatenated.
150
There is no limit to the number of segments that can be a part of a
153
To ensure the download works correctly, you must upload all the object
154
segments to the same container and ensure that each object name is
155
prefixed in such a way that it sorts in the order in which it should be
156
concatenated. You also create and upload a manifest file. The manifest
157
file is a zero-byte file with the extra ``X-Object-Manifest``
158
``{container}/{prefix}`` header, where ``{container}`` is the container
159
the object segments are in and ``{prefix}`` is the common prefix for all
160
the segments. You must UTF-8-encode and then URL-encode the container
161
and common prefix in the ``X-Object-Manifest`` header.
163
It is best to upload all the segments first and then create or update
164
the manifest. With this method, the full object is not available for
165
downloading until the upload is complete. Also, you can upload a new set
166
of segments to a second location and update the manifest to point to
167
this new location. During the upload of the new segments, the original
168
manifest is still available to download the first set of segments.
170
**Example Upload segment of large object request: HTTP**
174
PUT /{api_version}/{account}/{container}/{object} HTTP/1.1
175
Host: storage.clouddrive.com
176
X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
177
ETag: 8a964ee2a5e88be344f36c22562a6486
179
X-Object-Meta-PIN: 1234
182
No response body is returned. A status code of 2\ *``nn``* (between 200
183
and 299, inclusive) indicates a successful write; status 411 Length
184
Required denotes a missing ``Content-Length`` or ``Content-Type`` header
185
in the request. If the MD5 checksum of the data written to the storage
186
system does NOT match the (optionally) supplied ETag value, a 422
187
Unprocessable Entity response is returned.
189
You can continue uploading segments like this example shows, prior to
190
uploading the manifest.
192
**Example Upload next segment of large object request: HTTP**
196
PUT /{api_version}/{account}/{container}/{object} HTTP/1.1
197
Host: storage.clouddrive.com
198
X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
199
ETag: 8a964ee2a5e88be344f36c22562a6486
201
X-Object-Meta-PIN: 1234
204
Next, upload the manifest you created that indicates the container the
205
object segments reside within. Note that uploading additional segments
206
after the manifest is created causes the concatenated object to be that
207
much larger but you do not need to recreate the manifest file for
208
subsequent additional segments.
210
**Example Upload manifest request: HTTP**
214
PUT /{api_version}/{account}/{container}/{object} HTTP/1.1
215
Host: storage.clouddrive.com
216
X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
218
X-Object-Meta-PIN: 1234
219
X-Object-Manifest: {container}/{prefix}
222
**Example Upload manifest response: HTTP**
229
The ``Content-Type`` in the response for a **GET** or **HEAD** on the
230
manifest is the same as the ``Content-Type`` set during the **PUT**
231
request that created the manifest. You can easily change the
232
``Content-Type`` by reissuing the **PUT** request.
234
Comparison of static and dynamic large objects
235
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
237
While static and dynamic objects have similar behavior, here are
240
**Comparing static and dynamic large objects**
242
Static large object: Assured end-to-end integrity. The list of segments
243
includes the MD5 checksum (``ETag``) of each segment. You cannot upload the
244
manifest object if the ``ETag`` in the list differs from the uploaded segment
245
object. If a segment is somehow lost, an attempt to download the manifest
246
object results in an error. You must upload the segment objects before you
247
upload the manifest object. You cannot add or remove segment objects from the
248
manifest. However, you can create a completely new manifest object of the same
249
name with a different manifest list.
251
With static large objects, you can upload new segment objects or remove
252
existing segments. The names must simply match the ``{prefix}`` supplied
253
in ``X-Object-Manifest``. The segment objects must be at least 1 MB in size
254
(by default). The final segment object can be any size. At most, 1000 segments
255
are supported (by default). The manifest list includes the container name of
256
each object. Segment objects can be in different containers.
258
Dynamic large object: End-to-end integrity is not guaranteed. The eventual
259
consistency model means that although you have uploaded a segment object, it
260
might not appear in the container listing until later. If you download the
261
manifest before it appears in the container, it does not form part of the
262
content returned in response to a **GET** request.
264
With dynamic large objects, you can upload manifest and segment objects
265
in any order. In case a premature download of the manifest occurs, we
266
recommend users upload the manifest object after the segments. However,
267
the system does not enforce the order. Segment objects can be any size. All
268
segment objects must be in the same container.
270
Manifest object metadata
271
------------------------
273
For static large objects, the object has ``X-Static-Large-Object`` set to
274
``true``. You do not set this metadata directly. Instead the system sets
275
it when you **PUT** a static manifest object.
277
For dynamic object,s the ``X-Object-Manifest`` value is the
278
``{container}/{prefix}``, which indicates where the segment objects are
279
located. You supply this request header in the **PUT** operation.
281
Copying the manifest object
282
---------------------------
284
With static large objects, you include the *``?multipart-manifest=get``*
285
query string in the **COPY** request. The new object contains the same
286
manifest as the original. The segment objects are not copied. Instead,
287
both the original and new manifest objects share the same set of segment
290
When creating dynamic large objects, the **COPY** operation does not create
291
a manifest object. To duplicate a manifest object, use the **GET** operation
292
to read the value of ``X-Object-Manifest`` and use this value in the
293
``X-Object-Manifest`` request header in a **PUT** operation. This creates
294
a new manifest object that shares the same set of segment objects as the
295
original manifest object.
added
removed
doc/source/api/large_objects.rst
