« back to all changes in this revision
Viewing changes to swift/common/middleware/staticweb.py
- browse files at revision 255
- compare with another revision
- download diff
- download tarball
- view history from revision 255
- Committer: Tarmac
- Author(s): gholt
- Date: 2011-03-26 00:27:28 UTC
- mfrom: (222.1.19 staticweb)
- Revision ID: tarmac-20110326002728-ue5com09shohjvt3
Static web filter middleware.
- files added:
- swift/common/middleware/staticweb.py
- files modified:
- doc/source/misc.rst
added
removed
swift/common/middleware/staticweb.py
1
# Copyright (c) 2010-2011 OpenStack, LLC.
2
#
3
# Licensed under the Apache License, Version 2.0 (the "License");
4
# you may not use this file except in compliance with the License.
5
# You may obtain a copy of the License at
6
#
7
# http://www.apache.org/licenses/LICENSE-2.0
8
#
9
# Unless required by applicable law or agreed to in writing, software
10
# distributed under the License is distributed on an "AS IS" BASIS,
11
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
12
# implied.
13
# See the License for the specific language governing permissions and
14
# limitations under the License.
15
16
"""
17
This StaticWeb WSGI middleware will serve container data as a static web site
18
with index file and error file resolution and optional file listings. This mode
19
is normally only active for anonymous requests. If you want to use it with
20
authenticated requests, set the ``X-Web-Mode: true`` header on the request.
21
22
The ``staticweb`` filter should be added to the pipeline in your
23
``/etc/swift/proxy-server.conf`` file just after any auth middleware. Also, the
24
configuration section for the ``staticweb`` middleware itself needs to be
25
added. For example::
26
27
[DEFAULT]
28
...
29
30
[pipeline:main]
31
pipeline = healthcheck cache swauth staticweb proxy-server
32
33
...
34
35
[filter:staticweb]
36
use = egg:swift#staticweb
37
# Seconds to cache container x-container-meta-web-* header values.
38
# cache_timeout = 300
39
# You can override the default log routing for this filter here:
40
# set log_name = staticweb
41
# set log_facility = LOG_LOCAL0
42
# set log_level = INFO
43
# set access_log_name = staticweb
44
# set access_log_facility = LOG_LOCAL0
45
# set access_log_level = INFO
46
# set log_headers = False
47
48
Any publicly readable containers (for example, ``X-Container-Read: .r:*``, see
49
`acls`_ for more information on this) will be checked for
50
X-Container-Meta-Web-Index and X-Container-Meta-Web-Error header values::
51
52
X-Container-Meta-Web-Index <index.name>
53
X-Container-Meta-Web-Error <error.name.suffix>
54
55
If X-Container-Meta-Web-Index is set, any <index.name> files will be served
56
without having to specify the <index.name> part. For instance, setting
57
``X-Container-Meta-Web-Index: index.html`` will be able to serve the object
58
.../pseudo/path/index.html with just .../pseudo/path or .../pseudo/path/
59
60
If X-Container-Meta-Web-Error is set, any errors (currently just 401
61
Unauthorized and 404 Not Found) will instead serve the
62
.../<status.code><error.name.suffix> object. For instance, setting
63
``X-Container-Meta-Web-Error: error.html`` will serve .../404error.html for
64
requests for paths not found.
65
66
For psuedo paths that have no <index.name>, this middleware can serve HTML file
67
listings if you set the ``X-Container-Meta-Web-Listings: true`` metadata item
68
on the container.
69
70
If listings are enabled, the listings can have a custom style sheet by setting
71
the X-Container-Meta-Web-Listings-CSS header. For instance, setting
72
``X-Container-Meta-Web-Listings-CSS: listing.css`` will make listings link to
73
the .../listing.css style sheet. If you "view source" in your browser on a
74
listing page, you will see the well defined document structure that can be
75
styled.
76
77
Example usage of this middleware via ``st``:
78
79
Make the container publicly readable::
80
81
st post -r '.r:*' container
82
83
You should be able to get objects directly, but no index.html resolution or
84
listings.
85
86
Set an index file directive::
87
88
st post -m 'web-index:index.html' container
89
90
You should be able to hit paths that have an index.html without needing to
91
type the index.html part.
92
93
Turn on listings::
94
95
st post -m 'web-listings: true' container
96
97
Now you should see object listings for paths and pseudo paths that have no
98
index.html.
99
100
Enable a custom listings style sheet::
101
102
st post -m 'web-listings-css:listings.css' container
103
104
Set an error file::
105
106
st post -m 'web-error:error.html' container
107
108
Now 401's should load 401error.html, 404's should load 404error.html, etc.
109
"""
110
111
112
try:
113
import simplejson as json
114
except ImportError:
115
import json
116
117
import cgi
118
import time
119
from urllib import unquote, quote
120
121
from webob import Response, Request
122
from webob.exc import HTTPMovedPermanently, HTTPNotFound
123
124
from swift.common.utils import cache_from_env, get_logger, human_readable, \
125
split_path, TRUE_VALUES
126
127
128
class StaticWeb(object):
129
"""
130
The Static Web WSGI middleware filter; serves container data as a static
131
web site. See `staticweb`_ for an overview.
132
133
:param app: The next WSGI application/filter in the paste.deploy pipeline.
134
:param conf: The filter configuration dict.
135
"""
136
137
def __init__(self, app, conf):
138
#: The next WSGI application/filter in the paste.deploy pipeline.
139
self.app = app
140
#: The filter configuration dict.
141
self.conf = conf
142
#: The seconds to cache the x-container-meta-web-* headers.,
143
self.cache_timeout = int(conf.get('cache_timeout', 300))
144
#: Logger for this filter.
145
self.logger = get_logger(conf, log_route='staticweb')
146
access_log_conf = {}
147
for key in ('log_facility', 'log_name', 'log_level'):
148
value = conf.get('access_' + key, conf.get(key, None))
149
if value:
150
access_log_conf[key] = value
151
#: Web access logger for this filter.
152
self.access_logger = get_logger(access_log_conf,
153
log_route='staticweb-access')
154
#: Indicates whether full HTTP headers should be logged or not.
155
self.log_headers = conf.get('log_headers') == 'True'
156
# Results from the last call to self._start_response.
157
self._response_status = None
158
self._response_headers = None
159
self._response_exc_info = None
160
# Results from the last call to self._get_container_info.
161
self._index = self._error = self._listings = self._listings_css = None
162
163
def _start_response(self, status, headers, exc_info=None):
164
"""
165
Saves response info without sending it to the remote client.
166
Uses the same semantics as the usual WSGI start_response.
167
"""
168
self._response_status = status
169
self._response_headers = headers
170
self._response_exc_info = exc_info
171
172
def _error_response(self, response, env, start_response):
173
"""
174
Sends the error response to the remote client, possibly resolving a
175
custom error response body based on x-container-meta-web-error.
176
177
:param response: The error response we should default to sending.
178
:param env: The original request WSGI environment.
179
:param start_response: The WSGI start_response hook.
180
"""
181
self._log_response(env, self._get_status_int())
182
if not self._error:
183
start_response(self._response_status, self._response_headers,
184
self._response_exc_info)
185
return response
186
save_response_status = self._response_status
187
save_response_headers = self._response_headers
188
save_response_exc_info = self._response_exc_info
189
tmp_env = self._get_escalated_env(env)
190
tmp_env['REQUEST_METHOD'] = 'GET'
191
tmp_env['PATH_INFO'] = '/%s/%s/%s/%s%s' % (self.version, self.account,
192
self.container, self._get_status_int(), self._error)
193
resp = self.app(tmp_env, self._start_response)
194
if self._get_status_int() // 100 == 2:
195
start_response(save_response_status, self._response_headers,
196
self._response_exc_info)
197
return resp
198
start_response(save_response_status, save_response_headers,
199
save_response_exc_info)
200
return response
201
202
def _get_status_int(self):
203
"""
204
Returns the HTTP status int from the last called self._start_response
205
result.
206
"""
207
return int(self._response_status.split(' ', 1)[0])
208
209
def _get_escalated_env(self, env):
210
"""
211
Returns a new fresh WSGI environment with escalated privileges to do
212
backend checks, listings, etc. that the remote user wouldn't be able to
213
accomplish directly.
214
"""
215
new_env = {'REQUEST_METHOD': 'GET',
216
'HTTP_USER_AGENT': '%s StaticWeb' % env.get('HTTP_USER_AGENT')}
217
for name in ('eventlet.posthooks', 'HTTP_X_CF_TRANS_ID', 'REMOTE_USER',
218
'SCRIPT_NAME', 'SERVER_NAME', 'SERVER_PORT',
219
'SERVER_PROTOCOL', 'swift.cache'):
220
if name in env:
221
new_env[name] = env[name]
222
return new_env
223
224
def _get_container_info(self, env, start_response):
225
"""
226
Retrieves x-container-meta-web-index, x-container-meta-web-error,
227
x-container-meta-web-listings, and x-container-meta-web-listings-css
228
from memcache or from the cluster and stores the result in memcache and
229
in self._index, self._error, self._listings, and self._listings_css.
230
231
:param env: The WSGI environment dict.
232
:param start_response: The WSGI start_response hook.
233
"""
234
self._index = self._error = self._listings = self._listings_css = None
235
memcache_client = cache_from_env(env)
236
if memcache_client:
237
memcache_key = '/staticweb/%s/%s/%s' % (self.version, self.account,
238
self.container)
239
cached_data = memcache_client.get(memcache_key)
240
if cached_data:
241
(self._index, self._error, self._listings,
242
self._listings_css) = cached_data
243
return
244
tmp_env = self._get_escalated_env(env)
245
tmp_env['REQUEST_METHOD'] = 'HEAD'
246
req = Request.blank('/%s/%s/%s' % (self.version, self.account,
247
self.container), environ=tmp_env)
248
resp = req.get_response(self.app)
249
if resp.status_int // 100 == 2:
250
self._index = \
251
resp.headers.get('x-container-meta-web-index', '').strip()
252
self._error = \
253
resp.headers.get('x-container-meta-web-error', '').strip()
254
self._listings = \
255
resp.headers.get('x-container-meta-web-listings', '').strip()
256
self._listings_css = \
257
resp.headers.get('x-container-meta-web-listings-css',
258
'').strip()
259
if memcache_client:
260
memcache_client.set(memcache_key,
261
(self._index, self._error, self._listings,
262
self._listings_css),
263
timeout=self.cache_timeout)
264
265
def _listing(self, env, start_response, prefix=None):
266
"""
267
Sends an HTML object listing to the remote client.
268
269
:param env: The original WSGI environment dict.
270
:param start_response: The original WSGI start_response hook.
271
:param prefix: Any prefix desired for the container listing.
272
"""
273
if self._listings not in TRUE_VALUES:
274
resp = HTTPNotFound()(env, self._start_response)
275
return self._error_response(resp, env, start_response)
276
tmp_env = self._get_escalated_env(env)
277
tmp_env['REQUEST_METHOD'] = 'GET'
278
tmp_env['PATH_INFO'] = \
279
'/%s/%s/%s' % (self.version, self.account, self.container)
280
tmp_env['QUERY_STRING'] = 'delimiter=/&format=json'
281
if prefix:
282
tmp_env['QUERY_STRING'] += '&prefix=%s' % quote(prefix)
283
else:
284
prefix = ''
285
resp = self.app(tmp_env, self._start_response)
286
if self._get_status_int() // 100 != 2:
287
return self._error_response(resp, env, start_response)
288
listing = json.loads(''.join(resp))
289
if not listing:
290
resp = HTTPNotFound()(env, self._start_response)
291
return self._error_response(resp, env, start_response)
292
headers = {'Content-Type': 'text/html'}
293
body = '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 ' \
294
'Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n' \
295
'<html>\n' \
296
' <head>\n' \
297
' <title>Listing of %s</title>\n' % \
298
cgi.escape(env['PATH_INFO'])
299
if self._listings_css:
300
body += ' <link rel="stylesheet" type="text/css" ' \
301
'href="%s%s" />\n' % \
302
('../' * prefix.count('/'), quote(self._listings_css))
303
else:
304
body += ' <style type="text/css">\n' \
305
' h1 {font-size: 1em; font-weight: bold;}\n' \
306
' th {text-align: left; padding: 0px 1em 0px 1em;}\n' \
307
' td {padding: 0px 1em 0px 1em;}\n' \
308
' a {text-decoration: none;}\n' \
309
' </style>\n'
310
body += ' </head>\n' \
311
' <body>\n' \
312
' <h1 id="title">Listing of %s</h1>\n' \
313
' <table id="listing">\n' \
314
' <tr id="heading">\n' \
315
' <th class="colname">Name</th>\n' \
316
' <th class="colsize">Size</th>\n' \
317
' <th class="coldate">Date</th>\n' \
318
' </tr>\n' % \
319
cgi.escape(env['PATH_INFO'])
320
if prefix:
321
body += ' <tr id="parent" class="item">\n' \
322
' <td class="colname"><a href="../">../</a></td>\n' \
323
' <td class="colsize"> </td>\n' \
324
' <td class="coldate"> </td>\n' \
325
' </tr>\n'
326
for item in listing:
327
if 'subdir' in item:
328
subdir = item['subdir']
329
if prefix:
330
subdir = subdir[len(prefix):]
331
body += ' <tr class="item subdir">\n' \
332
' <td class="colname"><a href="%s">%s</a></td>\n' \
333
' <td class="colsize"> </td>\n' \
334
' <td class="coldate"> </td>\n' \
335
' </tr>\n' % \
336
(quote(subdir), cgi.escape(subdir))
337
for item in listing:
338
if 'name' in item:
339
name = item['name']
340
if prefix:
341
name = name[len(prefix):]
342
body += ' <tr class="item %s">\n' \
343
' <td class="colname"><a href="%s">%s</a></td>\n' \
344
' <td class="colsize">%s</td>\n' \
345
' <td class="coldate">%s</td>\n' \
346
' </tr>\n' % \
347
(' '.join('type-' + cgi.escape(t.lower(), quote=True)
348
for t in item['content_type'].split('/')),
349
quote(name), cgi.escape(name),
350
human_readable(item['bytes']),
351
cgi.escape(item['last_modified']).split('.')[0].
352
replace('T', ' '))
353
body += ' </table>\n' \
354
' </body>\n' \
355
'</html>\n'
356
resp = Response(headers=headers, body=body)
357
self._log_response(env, resp.status_int)
358
return resp(env, start_response)
359
360
def _handle_container(self, env, start_response):
361
"""
362
Handles a possible static web request for a container.
363
364
:param env: The original WSGI environment dict.
365
:param start_response: The original WSGI start_response hook.
366
"""
367
self._get_container_info(env, start_response)
368
if not self._listings and not self._index:
369
return self.app(env, start_response)
370
if env['PATH_INFO'][-1] != '/':
371
resp = HTTPMovedPermanently(
372
location=(env['PATH_INFO'] + '/'))
373
self._log_response(env, resp.status_int)
374
return resp(env, start_response)
375
if not self._index:
376
return self._listing(env, start_response)
377
tmp_env = dict(env)
378
tmp_env['HTTP_USER_AGENT'] = \
379
'%s StaticWeb' % env.get('HTTP_USER_AGENT')
380
tmp_env['PATH_INFO'] += self._index
381
resp = self.app(tmp_env, self._start_response)
382
status_int = self._get_status_int()
383
if status_int == 404:
384
return self._listing(env, start_response)
385
elif self._get_status_int() // 100 not in (2, 3):
386
return self._error_response(resp, env, start_response)
387
start_response(self._response_status, self._response_headers,
388
self._response_exc_info)
389
return resp
390
391
def _handle_object(self, env, start_response):
392
"""
393
Handles a possible static web request for an object. This object could
394
resolve into an index or listing request.
395
396
:param env: The original WSGI environment dict.
397
:param start_response: The original WSGI start_response hook.
398
"""
399
tmp_env = dict(env)
400
tmp_env['HTTP_USER_AGENT'] = \
401
'%s StaticWeb' % env.get('HTTP_USER_AGENT')
402
resp = self.app(tmp_env, self._start_response)
403
status_int = self._get_status_int()
404
if status_int // 100 in (2, 3):
405
return self.app(env, start_response)
406
if status_int != 404:
407
return self._error_response(resp, env, start_response)
408
self._get_container_info(env, start_response)
409
if not self._listings and not self._index:
410
return self.app(env, start_response)
411
status_int = 404
412
if self._index:
413
tmp_env = dict(env)
414
tmp_env['HTTP_USER_AGENT'] = \
415
'%s StaticWeb' % env.get('HTTP_USER_AGENT')
416
if tmp_env['PATH_INFO'][-1] != '/':
417
tmp_env['PATH_INFO'] += '/'
418
tmp_env['PATH_INFO'] += self._index
419
resp = self.app(tmp_env, self._start_response)
420
status_int = self._get_status_int()
421
if status_int // 100 in (2, 3):
422
if env['PATH_INFO'][-1] != '/':
423
resp = HTTPMovedPermanently(
424
location=env['PATH_INFO'] + '/')
425
self._log_response(env, resp.status_int)
426
return resp(env, start_response)
427
start_response(self._response_status, self._response_headers,
428
self._response_exc_info)
429
return resp
430
if status_int == 404:
431
if env['PATH_INFO'][-1] != '/':
432
tmp_env = self._get_escalated_env(env)
433
tmp_env['REQUEST_METHOD'] = 'GET'
434
tmp_env['PATH_INFO'] = '/%s/%s/%s' % (self.version,
435
self.account, self.container)
436
tmp_env['QUERY_STRING'] = 'limit=1&format=json&delimiter' \
437
'=/&limit=1&prefix=%s' % quote(self.obj + '/')
438
resp = self.app(tmp_env, self._start_response)
439
if self._get_status_int() // 100 != 2 or \
440
not json.loads(''.join(resp)):
441
resp = HTTPNotFound()(env, self._start_response)
442
return self._error_response(resp, env, start_response)
443
resp = HTTPMovedPermanently(location=env['PATH_INFO'] +
444
'/')
445
self._log_response(env, resp.status_int)
446
return resp(env, start_response)
447
return self._listing(env, start_response, self.obj)
448
449
def __call__(self, env, start_response):
450
"""
451
Main hook into the WSGI paste.deploy filter/app pipeline.
452
453
:param env: The WSGI environment dict.
454
:param start_response: The WSGI start_response hook.
455
"""
456
env['staticweb.start_time'] = time.time()
457
try:
458
(self.version, self.account, self.container, self.obj) = \
459
split_path(env['PATH_INFO'], 2, 4, True)
460
except ValueError:
461
return self.app(env, start_response)
462
memcache_client = cache_from_env(env)
463
if memcache_client:
464
if env['REQUEST_METHOD'] in ('PUT', 'POST'):
465
if not self.obj and self.container:
466
memcache_key = '/staticweb/%s/%s/%s' % \
467
(self.version, self.account, self.container)
468
memcache_client.delete(memcache_key)
469
return self.app(env, start_response)
470
if (env['REQUEST_METHOD'] not in ('HEAD', 'GET') or
471
(env.get('REMOTE_USER') and
472
env.get('HTTP_X_WEB_MODE', 'f').lower() not in TRUE_VALUES) or
473
(not env.get('REMOTE_USER') and
474
env.get('HTTP_X_WEB_MODE', 't').lower() not in TRUE_VALUES)):
475
return self.app(env, start_response)
476
if self.obj:
477
return self._handle_object(env, start_response)
478
elif self.container:
479
return self._handle_container(env, start_response)
480
return self.app(env, start_response)
481
482
def _log_response(self, env, status_int):
483
"""
484
Logs an access line for StaticWeb responses; use when the next app in
485
the pipeline will not be handling the final response to the remote
486
user.
487
488
Assumes that the request and response bodies are 0 bytes or very near 0
489
so no bytes transferred are tracked or logged.
490
491
This does mean that the listings responses that actually do transfer
492
content will not be logged with any bytes transferred, but in counter
493
to that the full bytes for the underlying listing will be logged by the
494
proxy even if the remote client disconnects early for the StaticWeb
495
listing.
496
497
I didn't think the extra complexity of getting the bytes transferred
498
exactly correct for these requests was worth it, but perhaps someone
499
else will think it is.
500
501
To get things exact, this filter would need to use an
502
eventlet.posthooks logger like the proxy does and any log processing
503
systems would need to ignore some (but not all) proxy requests made by
504
StaticWeb if they were just interested in the bytes transferred to the
505
remote client.
506
"""
507
trans_time = '%.4f' % (time.time() -
508
env.get('staticweb.start_time', time.time()))
509
the_request = quote(unquote(env['PATH_INFO']))
510
if env.get('QUERY_STRING'):
511
the_request = the_request + '?' + env['QUERY_STRING']
512
# remote user for zeus
513
client = env.get('HTTP_X_CLUSTER_CLIENT_IP')
514
if not client and 'HTTP_X_FORWARDED_FOR' in env:
515
# remote user for other lbs
516
client = env['HTTP_X_FORWARDED_FOR'].split(',')[0].strip()
517
logged_headers = None
518
if self.log_headers:
519
logged_headers = '\n'.join('%s: %s' % (k, v)
520
for k, v in req.headers.items())
521
self.access_logger.info(' '.join(quote(str(x)) for x in (
522
client or '-',
523
env.get('REMOTE_ADDR', '-'),
524
time.strftime('%d/%b/%Y/%H/%M/%S', time.gmtime()),
525
env['REQUEST_METHOD'],
526
the_request,
527
env['SERVER_PROTOCOL'],
528
status_int,
529
env.get('HTTP_REFERER', '-'),
530
env.get('HTTP_USER_AGENT', '-'),
531
env.get('HTTP_X_AUTH_TOKEN', '-'),
532
'-',
533
'-',
534
env.get('HTTP_ETAG', '-'),
535
env.get('HTTP_X_CF_TRANS_ID', '-'),
536
logged_headers or '-',
537
trans_time)))
538
539
540
def filter_factory(global_conf, **local_conf):
541
""" Returns a Static Web WSGI filter for use with paste.deploy. """
542
conf = global_conf.copy()
543
conf.update(local_conf)
544
545
def staticweb_filter(app):
546
return StaticWeb(app, conf)
547
return staticweb_filter
Loggerhead is a web-based interface for Breezy
Version: 2.0.1