~annegentle/openstack-devel/working

<title>Managing Large Objects (Greater than 5 GB)</title><meta name="generator" content="DocBook XSL Stylesheets V1.76.1"/><link rel="home" href="index.html" title="OpenStack Object Storage Admin Guide"/><link rel="up" href="ch05.html" title="Chapter 5. Running OpenStack Object Storage"/><link rel="prev" href="ch05s07.html" title="Replication"/><link rel="next" href="ch05s09.html" title="Throttling Resources by Setting Rate Limits"/><meta http-equiv="Content-Type" content="text/html; charset=utf-8"/><script type="text/javascript">

//The id for tree cookie

var treeCookieId = "treeview-4625";

var language = "en";

var w = new Object();

//Localization

txt_filesfound = 'Results';

txt_enter_at_least_1_char = "You must enter at least one character.";

txt_browser_not_supported = "Your browser is not supported. Use of Mozilla Firefox is recommended.";

txt_please_wait = "Please wait. Search in progress...";

txt_results_for = "Results for: ";

</script><style type="text/css">

input {

margin-bottom: 5px;

margin-top: 2px;

}

.folder {

display: block;

height: 22px;

padding-left: 20px;

background: transparent url(../common/jquery/treeview/images/folder.gif) 0 0px no-repeat;

}

</style><link rel="shortcut icon" href="../favicon.ico" type="image/x-icon"/><link rel="stylesheet" type="text/css" href="../common/css/positioning.css"/><link rel="stylesheet" type="text/css" href="../common/jquery/theme-redmond/jquery-ui-1.8.2.custom.css"/><link rel="stylesheet" type="text/css" href="../common/jquery/treeview/jquery.treeview.css"/><script type="text/javascript" src="../common/jquery/jquery-1.4.2.min.js"></script><script type="text/javascript" src="../common/jquery/jquery-ui-1.8.2.custom.min.js"></script><script type="text/javascript" src="../common/jquery/jquery.cookie.js"></script><script type="text/javascript" src="../common/jquery/treeview/jquery.treeview.min.js"></script><script type="text/javascript" src="search/htmlFileList.js"></script><script type="text/javascript" src="search/htmlFileInfoList.js"></script><script type="text/javascript" src="search/nwSearchFnt.js"></script><script type="text/javascript" src="search/stemmers/en_stemmer.js"/><script type="text/javascript" src="search/index-1.js"></script><script type="text/javascript" src="search/index-2.js"></script><script type="text/javascript" src="search/index-3.js"></script></head><body><div id="header"><img src="../common/images/openstack-logo.png" alt="OpenStack Documentation" width="157" height="47"/><p class="breadcrumbs"><a href="#">OpenStack Manuals</a><a href="#">Object Storage</a></p><h1>Managing Large Objects (Greater than 5 GB)<br/>Chapter 5. Running OpenStack Object Storage</h1><div id="navheader" align="right"><table class="navLinks"><tr><td><a id="showHideButton" onclick="showHideToc();" class="pointLeft" title="Hide TOC tree">Sidebar

</a></td><td><a accesskey="p" class="navLinkPrevious" href="ch05s07.html">Prev</a>

<a accesskey="n" class="navLinkNext" href="ch05s09.html">Next</a></td></tr></table></div></div><div id="content"><div class="section" title="Managing Large Objects (Greater than 5 GB)"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="d5e1367"/>Managing Large Objects (Greater than 5 GB)</h2></div></div></div>

<p>OpenStack Object Storage has a limit on the size of a single uploaded object; by default this is

5GB. However, the download size of a single object is virtually unlimited with

the concept of segmentation. Segments of the larger object are uploaded and a

special manifest file is created that, when downloaded, sends all the segments

concatenated as a single object. This also offers much greater upload speed

with the possibility of parallel uploads of the segments.</p>

<div class="section" title="Using st to Manage Segmented Objects"><div class="titlepage"><div><div><h3 class="title"><a id="d5e1370"/>Using st to Manage Segmented Objects</h3></div></div></div>

<p>The quickest way to try out this feature is use the included st OpenStack Object Storage Tool.

You can use the -S option to specify the segment size to use when splitting

a large file. For example:</p>

<div class="literallayout"><p>st upload test_container -S 1073741824 large_file</p></div>

<p>This would split the large_file into 1G segments and begin uploading those

segments in parallel. Once all the segments have been uploaded, st will

then create the manifest file so the segments can be downloaded as one.</p>

<p>So now, the following st command would download the entire large object:</p>

<div class="literallayout"><p>st download test_container large_file</p></div>

<p>st uses a strict convention for its segmented object support. In the above

example it will upload all the segments into a second container named

test_container_segments. These segments will have names like

large_file/1290206778.25/21474836480/00000000,

large_file/1290206778.25/21474836480/00000001, etc.</p>

<p>The main benefit for using a separate container is that the main container

listings will not be polluted with all the segment names. The reason for using

the segment name format of <name>/<timestamp>/<size>/<segment> is so that an

upload of a new file with the same name won't overwrite the contents of the

first until the last moment when the manifest file is updated.</p>

<p>st will manage these segment files for you, deleting old segments on

deletes and overwrites, etc. You can override this behavior with the

--leave-segments option if desired; this is useful if you want to have

multiple versions of the same large object available.</p>

</div>

<div class="section" title="Direct API Management of Large Objects"><div class="titlepage"><div><div><h3 class="title"><a id="d5e1380"/>Direct API Management of Large Objects</h3></div></div></div>

<p>You can also work with the segments and manifests directly with HTTP requests

instead of having st do that for you. You can just upload the segments like

you would any other object and the manifest is just a zero-byte file with an

extra X-Object-Manifest header.</p>

<p>All the object segments need to be in the same container, have a common object

name prefix, and their names sort in the order they should be concatenated.

They don't have to be in the same container as the manifest file will be, which

is useful to keep container listings clean as explained above with st.</p>

<p>The manifest file is simply a zero-byte file with the extra

X-Object-Manifest:<code class="code"><container>/<prefix> header</code>, where <container> is

the container the object segments are in and <prefix> is the common prefix

for all the segments.</p>

<p>It is best to upload all the segments first and then create or update the

manifest. In this way, the full object won't be available for downloading until

the upload is complete. Also, you can upload a new set of segments to a second

location and then update the manifest to point to this new location. During the

upload of the new segments, the original manifest will still be available to

download the first set of segments.</p>

<p>Here's an example using curl with tiny 1-byte segments:</p>

# First, upload the segments<br/>

100

curl -X PUT -H 'X-Auth-Token: <token>' \<br/>

101

http://<storage_url>/container/myobject/1 --data-binary '1'<br/>

102

curl -X PUT -H 'X-Auth-Token: <token>' \<br/>

103

http://<storage_url>/container/myobject/2 --data-binary '2'<br/>

104

curl -X PUT -H 'X-Auth-Token: <token>' \<br/>

105

http://<storage_url>/container/myobject/3 --data-binary '3'<br/>

106

<br/>

107

# Next, create the manifest file<br/>

108

curl -X PUT -H 'X-Auth-Token: <token>' \<br/>

109

-H 'X-Object-Manifest: container/myobject/' \<br/>

110

http://<storage_url>/container/myobject --data-binary ''<br/>

111

<br/>

112

# And now we can download the segments as a single object<br/>

113

curl -H 'X-Auth-Token: <token>' \<br/>

114

http://<storage_url>/container/myobject</p></div>

115

116

</div>

117

<div class="section" title="Additional Notes on Large Objects"><div class="titlepage"><div><div><h3 class="title"><a id="d5e1389"/>Additional Notes on Large Objects</h3></div></div></div>

118

119

<div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem"><p>With a GET or HEAD of a manifest file, the X-Object-Manifest:

120

<code class="code"><container>/<prefix></code> header will be returned with the concatenated object

121

so you can tell where it's getting its segments from.</p></li><li class="listitem"><p>The response's Content-Length for a GET or HEAD on the manifest

122

file will be the sum of all the segments in the <container>/<prefix>

123

listing, dynamically. So, uploading additional segments after the manifest is

124

created will cause the concatenated object to be that much larger; there's no

125

need to recreate the manifest file.</p></li><li class="listitem"><p>The response's Content-Type for a GET or HEAD on the manifest

126

will be the same as the Content-Type set during the PUT request that

127

created the manifest. You can easily change the Content-Type by reissuing

128

the PUT.</p></li><li class="listitem"><p>The response's ETag for a GET or HEAD on the manifest file will

129

be the MD5 sum of the concatenated string of ETags for each of the segments

130

in the <container>/<prefix> listing, dynamically. Usually in OpenStack Object Storage the

131

ETag is the MD5 sum of the contents of the object, and that holds true for

132

each segment independently. But, it's not feasible to generate such an ETag

133

for the manifest itself, so this method was chosen to at least offer change

134

detection.</p></li></ul></div>

135

</div>

136

<div class="section" title="Large Object Storage History and Background"><div class="titlepage"><div><div><h3 class="title"><a id="d5e1401"/>Large Object Storage History and Background</h3></div></div></div>

137

138

<p>Large object support has gone through various iterations before settling on

139

this implementation.</p>

140

<p>The primary factor driving the limitation of object size in OpenStack Object Storage is

141

maintaining balance among the partitions of the ring. To maintain an even

142

dispersion of disk usage throughout the cluster the obvious storage pattern

143

was to simply split larger objects into smaller segments, which could then be

144

glued together during a read.</p>

145

<p>Before the introduction of large object support some applications were already

146

splitting their uploads into segments and re-assembling them on the client

147

side after retrieving the individual pieces. This design allowed the client

148

to support backup and archiving of large data sets, but was also frequently

149

employed to improve performance or reduce errors due to network interruption.

150

The major disadvantage of this method is that knowledge of the original

151

partitioning scheme is required to properly reassemble the object, which is

152

not practical for some use cases, such as CDN origination.</p>

153

<p>In order to eliminate any barrier to entry for clients wanting to store

154

objects larger than 5GB, initially we also prototyped fully transparent

155

support for large object uploads. A fully transparent implementation would

156

support a larger max size by automatically splitting objects into segments

157

during upload within the proxy without any changes to the client API. All

158

segments were completely hidden from the client API.</p>

159

<p>This solution introduced a number of challenging failure conditions into the

160

cluster, wouldn't provide the client with any option to do parallel uploads,

161

and had no basis for a resume feature. The transparent implementation was

162

deemed just too complex for the benefit.</p>

163

164

<p>The current “user manifest” design was chosen in order to provide a

165

transparent download of large objects to the client and still provide the

166

uploading client a clean API to support segmented uploads.</p>

167

<p>Alternative “explicit” user manifest options were discussed which would have

168

required a pre-defined format for listing the segments to “finalize” the

169

segmented upload. While this may offer some potential advantages, it was

170

decided that pushing an added burden onto the client which could potentially

171

limit adoption should be avoided in favor of a simpler “API” (essentially just

172

the format of the ‘X-Object-Manifest' header).</p>

173

174

<p>During development it was noted that this “implicit” user manifest approach

175

which is based on the path prefix can be potentially affected by the eventual

176

consistency window of the container listings, which could theoretically cause

177

a GET on the manifest object to return an invalid whole object for that short

178

term. In reality you're unlikely to encounter this scenario unless you're

179

running very high concurrency uploads against a small testing environment

180

which isn't running the object-updaters or container-replicators.</p>

181

<p>Like all of OpenStack Object Storage, Large Object Support is living feature which will continue

182

to improve and may change over time.</p>

183

184

</div>

185

</div><script type="text/javascript" src="../common/main.js"></script><div class="navfooter"><hr/><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="ch05s07.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="ch05.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="ch05s09.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top"> </td><td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td><td width="40%" align="right" valign="top"> </td></tr></table></div></div><div><div id="leftnavigation" style="padding-top:3px; background-color:white;"><div id="tabs"><ul><li><a href="#treeDiv"><span class="contentsTab">Contents</span></a></li><li><a href="#searchDiv"><span class="searchTab">Search</span></a></li></ul><div id="treeDiv"><img src="../common/images/loading.gif" alt="loading table of contents..." id="tocLoading" style="display:block;"/><div id="ulTreeDiv" style="display:none"><ul id="tree" class="filetree"><li><span class="file"><a href="ch01.html">Getting Started with OpenStack</a></span><ul><li><span class="file"><a href="ch01s01.html">What is OpenStack?</a></span></li><li><span class="file"><a href="ch01s02.html">Components of OpenStack</a></span></li><li><span class="file"><a href="ch01s03.html">Why Cloud?</a></span></li></ul></li><li><span class="file"><a href="ch02.html">Introduction to OpenStack Object Storage</a></span><ul><li><span class="file"><a href="ch02s01.html">Accounts and Account Servers</a></span></li><li><span class="file"><a href="ch02s02.html">Authentication</a></span></li><li><span class="file"><a href="ch02s03.html">Permissions</a></span></li><li><span class="file"><a href="ch02s04.html">Containers</a></span></li><li><span class="file"><a href="ch02s05.html">Objects</a></span></li><li><span class="file"><a href="ch02s06.html">Operations</a></span></li><li><span class="file"><a href="ch02s07.html">Language-Specific API Bindings</a></span></li></ul></li><li><span class="file"><a href="ch03.html">OpenStack Object Storage Tutorials</a></span><ul><li><span class="file"><a href="ch03s01.html">Storing Large Photos or Videos on the Cloud</a></span><ul><li><span class="file"><a href="ch03s01.html#d5e133">Part I: Setting Up Secure Access</a></span></li><li><span class="file"><a href="ch03s01.html#d5e145">Part II: Configuring Cyberduck</a></span></li><li><span class="file"><a href="ch03s01.html#d5e148">Part III: Copying Files</a></span></li></ul></li></ul></li><li><span class="file"><a href="ch04.html">Installing and Configuring OpenStack Object Storage</a></span><ul><li><span class="file"><a href="ch04s01.html">System Requirements</a></span></li><li><span class="file"><a href="ch04s02.html">Installing OpenStack Object Storage on Ubuntu</a></span><ul><li><span class="file"><a href="ch04s02.html#d5e164">Before You Begin</a></span></li><li><span class="file"><a href="ch04s02.html#d5e177">Example Installation Architecture</a></span></li><li><span class="file"><a href="ch04s02.html#d5e196">Network Setup Notes</a></span></li><li><span class="file"><a href="ch04s02.html#d5e199">General Installation Steps </a></span></li></ul></li><li><span class="file"><a href="ch04s03.html">Configuring OpenStack Object Storage </a></span><ul><li><span class="file"><a href="ch04s03.html#d5e221">Installing and Configuring the Proxy Node</a></span></li><li><span class="file"><a href="ch04s03.html#d5e280">Installing and Configuring the Auth Node</a></span></li><li><span class="file"><a href="ch04s03.html#d5e297">Installing and Configuring the Storage Nodes</a></span><ul><li><span class="file"><a href="ch04s03.html#d5e342">Create OpenStack Object Storage admin Account and Verify the Installation</a></span></li><li><span class="file"><a href="ch04s03.html#d5e374">Adding a Proxy Server</a></span></li><li><span class="file"><a href="ch04s03.html#d5e395">Troubleshooting Notes</a></span></li></ul></li></ul></li></ul></li><li><span class="file"><a href="ch05.html">Running OpenStack Object Storage</a></span><ul><li><span class="file"><a href="ch05s01.html">Understanding How Object Storage Works</a></span><ul><li><span class="file"><a href="ch05s01.html#d5e405">The Ring</a></span></li><li><span class="file"><a href="ch05s01.html#d5e413">Proxy Server</a></span></li><li><span class="file"><a href="ch05s01.html#d5e418">Object Server</a></span></li><li><span class="file"><a href="ch05s01.html#d5e422">Container Server</a></span></li><li><span class="file"><a href="ch05s01.html#d5e425">Account Server</a></span></li><li><span class="file"><a href="ch05s01.html#d5e428">Replication</a></span></li><li><span class="file"><a href="ch05s01.html#d5e434">Updaters</a></span></li><li><span class="file"><a href="ch05s01.html#d5e438">Auditors</a></span></li></ul></li><li><span class="file"><a href="ch05s02.html">Configuring and Tuning an OpenStack Object Storage Deployment</a></span></li><li><span class="file"><a href="ch05s03.html">Deployment Options</a></span></li><li><span class="file"><a href="ch05s04.html">Preparing the Ring</a></span></li><li><span class="file"><a href="ch05s05.html">Server Configuration Reference</a></span><ul><li><span class="file"><a href="ch05s05.html#d5e483">Object Server Configuration</a></span></li><li><span class="file"><a href="ch05s05.html#d5e683">Container Server Configuration</a></span></li><li><span class="file"><a href="ch05s05.html#d5e859">Account Server Configuration</a></span></li><li><span class="file"><a href="ch05s05.html#d5e1024">Proxy Server Configuration</a></span></li></ul></li><li><span class="file"><a href="ch05s06.html">Considerations and Tuning</a></span><ul><li><span class="file"><a href="ch05s06.html#d5e1221">Memcached Considerations</a></span></li><li><span class="file"><a href="ch05s06.html#d5e1226">System Time</a></span></li><li><span class="file"><a href="ch05s06.html#d5e1229">General Service Tuning</a></span></li><li><span class="file"><a href="ch05s06.html#d5e1235">Filesystem Considerations</a></span></li><li><span class="file"><a href="ch05s06.html#d5e1247">General System Tuning</a></span></li><li><span class="file"><a href="ch05s06.html#d5e1256">Logging Considerations</a></span></li><li><span class="file"><a href="ch05s06.html#d5e1260">Working with Rings</a></span><ul><li><span class="file"><a href="ch05s06.html#d5e1265">Managing Rings with the Ring Builder</a></span><ul><li><span class="file"><a href="ch05s06.html#d5e1269">About the Ring Data Structure</a></span><ul><li><span class="file"><a href="ch05s06.html#d5e1272">List of Devices in the Ring</a></span></li></ul></li><li><span class="file"><a href="ch05s06.html#d5e1311">Partition Assignment List</a></span></li><li><span class="file"><a href="ch05s06.html#d5e1316">Partition Shift Value</a></span></li></ul></li><li><span class="file"><a href="ch05s06.html#d5e1319">Building the Ring</a></span></li><li><span class="file"><a href="ch05s06.html#d5e1326">History of the Ring Design</a></span></li></ul></li><li><span class="file"><a href="ch05s06.html#d5e1336">The Account Reaper</a></span><ul><li><span class="file"><a href="ch05s06.html#d5e1342">Account Reaper Background and History</a></span></li></ul></li></ul></li><li><span class="file"><a href="ch05s07.html">Replication</a></span><ul><li><span class="file"><a href="ch05s07.html#d5e1356">Database Replication</a></span></li><li><span class="file"><a href="ch05s07.html#d5e1362">Object Replication</a></span></li></ul></li><li id="webhelp-currentid"><span class="file"><a href="ch05s08.html">Managing Large Objects (Greater than 5 GB)</a></span><ul><li><span class="file"><a href="ch05s08.html#d5e1370">Using st to Manage Segmented Objects</a></span></li><li><span class="file"><a href="ch05s08.html#d5e1380">Direct API Management of Large Objects</a></span></li><li><span class="file"><a href="ch05s08.html#d5e1389">Additional Notes on Large Objects</a></span></li><li><span class="file"><a href="ch05s08.html#d5e1401">Large Object Storage History and Background</a></span></li></ul></li><li><span class="file"><a href="ch05s09.html">Throttling Resources by Setting Rate Limits</a></span><ul><li><span class="file"><a href="ch05s09.html#d5e1415">Configuration for Rate Limiting</a></span></li></ul></li><li><span class="file"><a href="ch05s10.html">Managing OpenStack Object Storage with ST</a></span><ul><li><span class="file"><a href="ch05s10.html#d5e1482">ST Basics</a></span><ul><li><span class="file"><a href="ch05s10.html#d5e1487">stat [container] [object]</a></span></li><li><span class="file"><a href="ch05s10.html#d5e1490">list [options] [container]</a></span></li><li><span class="file"><a href="ch05s10.html#d5e1493">upload [options] container file_or_directory [file_or_directory] […]</a></span></li><li><span class="file"><a href="ch05s10.html#d5e1496">post [options] [container] [object]</a></span></li><li><span class="file"><a href="ch05s10.html#d5e1500">download —all OR download container [object] [object] …</a></span></li><li><span class="file"><a href="ch05s10.html#d5e1503">delete —all OR delete container [object] [object] …</a></span></li><li><span class="file"><a href="ch05s10.html#d5e1507">Options for st</a></span></li></ul></li><li><span class="file"><a href="ch05s10.html#d5e1517">Analyzing Log Files with ST</a></span></li></ul></li></ul></li><li><span class="file"><a href="ch06.html">Support and Troubleshooting</a></span><ul><li><span class="file"><a href="ch06s01.html">Community Support</a></span><ul><li><span class="file"><a href="ch06s01.html#d5e1541">The Launchpad Answers area </a></span></li><li><span class="file"><a href="ch06s01.html#d5e1546">OpenStack mailing list</a></span></li><li><span class="file"><a href="ch06s01.html#d5e1551">The OpenStack Wiki search </a></span></li><li><span class="file"><a href="ch06s01.html#d5e1554">The Launchpad Bugs area </a></span></li><li><span class="file"><a href="ch06s01.html#d5e1570">The OpenStack IRC channel </a></span></li><li><span class="file"><a href="ch06s01.html#d5e1573">OpenStack Mailing Lists </a></span></li></ul></li><li><span class="file"><a href="ch06s02.html">Troubleshooting OpenStack Object Storage</a></span><ul><li><span class="file"><a href="ch06s02.html#d5e1579">Handling Drive Failure</a></span></li><li><span class="file"><a href="ch06s02.html#d5e1583">Handling Server Failure</a></span></li><li><span class="file"><a href="ch06s02.html#d5e1588">Detecting Failed Drives</a></span></li></ul></li><li><span class="file"><a href="ch06s03.html">Troubleshooting OpenStack Compute</a></span><ul><li><span class="file"><a href="ch06s03.html#d5e1595">Log files for OpenStack Compute</a></span></li><li><span class="file"><a href="ch06s03.html#d5e1599">Common Errors and Fixes for OpenStack Compute</a></span></li></ul></li></ul></li></ul></div></div><div id="searchDiv"><div id="search"><form onsubmit="Verifie(ditaSearch_Form);return false" name="ditaSearch_Form" class="searchForm"><fieldset class="searchFieldSet"><legend>Search</legend><center><input id="textToSearch" name="textToSearch" type="text" class="searchText"/>   <input onclick="Verifie(ditaSearch_Form)" type="button" class="searchButton" value="Go" id="doSearch"/></center></fieldset></form></div><div id="searchResults"><center/></div><p class="searchHighlight"><a href="#" onclick="toggleHighlight()">Search Highlighter (On/Off)</a></p></div></div></div></div></body></html>

b'\\ No newline at end of file'

Older »