2
ngx_chunkin - HTTP 1.1 chunked-encoding request body support for Nginx.
4
*This module is not distributed with the Nginx source.* See the
5
installation instructions.
8
This module is considered production ready.
11
This document describes chunkin-nginx-module v0.22
12
(<http://github.com/agentzh/chunkin-nginx-module/tags>) released on
18
error_page 411 = @my_411_error;
19
location @my_411_error {
24
# your fastcgi_pass/proxy_pass/set/if and
25
# any other config directives go here...
31
error_page 411 = @my_411_error;
32
location @my_411_error {
37
chunkin_keepalive on; # WARNING: too experimental!
39
# your fastcgi_pass/proxy_pass/set/if and
40
# any other config directives go here...
44
This module adds HTTP 1.1 chunked
45
(<http://tools.ietf.org/html/rfc2616#section-3.6.1>) input support for
46
Nginx without the need of patching the Nginx core.
48
Behind the scene, it registers an access-phase handler that will eagerly
49
read and decode incoming request bodies when a "Transfer-Encoding:
50
chunked" header triggers a 411 error page in Nginx. For requests that
51
are not in the "chunked" transfer encoding, this module is a "no-op".
53
To enable the magic, just turn on the chunkin config option and define a
54
custom "411 error_page" using chunkin_resume, like this:
59
error_page 411 = @my_411_error;
60
location @my_411_error {
67
No other modification is required in your nginx.conf file and everything
68
should work out of the box including the standard [module
69
(HttpProxyModule)] (except for those known issues). Note that the
70
chunkin directive is not allowed in the location block while the
71
chunkin_resume directive is only allowed on in "locations".
73
The core module's client_body_buffer_size, client_max_body_size, and
74
client_body_timeout directive settings are honored. Note that, the "body
75
sizes" here always indicate chunked-encoded body, not the data that has
76
already been decoded. Basically, the chunked-encoded body will always be
77
slightly larger than the original data that is not encoded.
79
The client_body_in_file_only and client_body_in_single_buffer settings
80
are followed partially. See Know Issues.
82
This module is not supposed to be merged into the Nginx core because
83
I've used Ragel (<http://www.complang.org/ragel/>) to generate the
84
chunked encoding parser for joy :)
87
Nginx explicitly checks chunked "Transfer-Encoding" headers and absent
88
content length header in its very early phase. Well, as early as the
89
"ngx_http_process_request_header" function. So this module takes a
90
rather tricky approach. That is, use an output filter to intercept the
91
"411 Length Required" error page response issued by
92
"ngx_http_process_request_header", fix things and finally issue an
93
internal redirect to the current location, thus starting from those
94
phases we all know and love, this time bypassing the horrible
95
"ngx_http_process_request_header" function.
97
In the "rewrite" phase of the newly created request, this module eagerly
98
reads in the chunked request body in a way similar to that of the
99
standard "ngx_http_read_client_request_body" function, but using its own
100
chunked parser generated by Ragel. The decoded request body will be put
101
into "r->request_body->bufs" and a corresponding "Content-Length" header
102
will be inserted into "r->headers_in".
104
Those modules using the standard "ngx_http_read_client_request_body"
105
function to read the request body will just work out of box because
106
"ngx_http_read_client_request_body" returns immediately when it sees
107
"r->request_body->bufs" already exists.
109
Special efforts have been made to reduce data copying and dynamic memory
114
syntax: *chunkin on|off*
118
context: *http, server*
122
Enables or disables this module's hooks.
125
syntax: *chunkin_resume*
133
This directive must be used in your custom "411 error page" location to
134
help this module work correctly. For example:
136
error_page 411 = @my_error;
141
For the technical reason behind the necessity of this directive, please
142
read the "nginx-devel" thread Content-Length is not ignored for chunked
143
requests: Nginx violates RFC 2616
144
(<http://nginx.org/pipermail/nginx-devel/2009-December/000041.html>).
146
This directive was first introduced in the v0.17 release.
148
chunkin_max_chunks_per_buf
149
syntax: *chunkin_max_chunks_per_buf <number>*
153
context: *http, server, location*
155
Set the max chunk count threshold for the buffer determined by the
156
client_body_buffer_size directive. If the average chunk size is "1 KB"
157
and your client_body_buffer_size setting is 1 meta bytes, then you
158
should set this threshold to 1024 or 2048.
160
When the raw body size is exceeding client_body_buffer_size *or* the
161
chunk counter is exceeding this "chunkin_max_chunks_per_buf" setting,
162
the decoded data will be temporarily buffered into disk files, and then
163
the main buffer gets cleared and the chunk counter gets reset back to 0
164
(or 1 if there's a "pending chunk").
166
This directive was first introduced in the v0.17 release.
169
syntax: *chunkin_keepalive on|off*
173
context: *http, server, location, if*
175
Turns on or turns off HTTP 1.1 keep-alive and HTTP 1.1 pipelining
178
Keep-alive without pipelining should be quite stable but pipelining
179
support is very preliminary, limited, and almost untested.
181
This directive was first introduced in the v0.07 release.
183
Technical note on the HTTP 1.1 pipeling support
185
The basic idea is to copy the bytes left by my chunked parser in
186
"r->request_body->buf" over into "r->header_in" so that nginx's
187
"ngx_http_set_keepalive" and "ngx_http_init_request" functions will pick
188
it up for the subsequent pipelined requests. When the request body is
189
small enough to be completely preread into the "r->header_in" buffer,
190
then no data copy is needed here -- just setting "r->header_in->pos"
191
correctly will suffice.
193
The only issue that remains is how to enlarge "r->header_in" when the
194
data left in "r->request_body->buf" is just too large to be hold in the
195
remaining room between "r->header_in->pos" and "r->header_in->end". For
196
now, this module will just give up and simply turn off "r->keepalive".
198
I know we can always use exactly the remaining room in "r->header_in" as
199
the buffer size when reading data from "c->recv", but's suboptimal when
200
the remaining room in "r->header_in" happens to be very small while
201
"r->request_body->buf" is quite large.
203
I haven't fully grokked all the details among "r->header_in",
204
"c->buffer", busy/free lists and those so-called "large header buffers".
205
Is there a clean and safe way to reallocate or extend the "r->header_in"
209
Grab the nginx source code from nginx.org (<http://nginx.org/>), for
210
example, the version 1.0.8 (see nginx compatibility), and then build the
211
source with this module:
213
wget 'http://nginx.org/download/nginx-1.0.8.tar.gz'
214
tar -xzvf nginx-1.0.8.tar.gz
217
# Here we assume you would install you nginx under /opt/nginx/.
218
./configure --prefix=/opt/nginx \
219
--add-module=/path/to/chunkin-nginx-module
224
Download the latest version of the release tarball of this module from
225
chunkin-nginx-module file list
226
(<http://github.com/agentzh/chunkin-nginx-module/tags>).
229
The chunked parser is generated by Ragel
230
(<http://www.complang.org/ragel/>). If you want to regenerate the
231
parser's C file, i.e., src/chunked_parser.c
232
(<http://github.com/agentzh/chunkin-nginx-module/blob/master/src/chunked
233
_parser.c>), use the following command from the root of the chunkin
234
module's source tree:
236
$ ragel -G2 src/chunked_parser.rl
240
The following source and binary rpm files are contributed by Ernest
241
Folch, with nginx 0.8.54, ngx_chunkin v0.21 and ngx_headers_more v0.13:
243
* nginx-0.8.54-1.fc13.src.rpm
244
(<http://agentzh.org/misc/nginx/ernest/nginx-0.8.54-1.fc13.src.rpm>)
246
* nginx-0.8.54-1.fc13.x86_64.rpm
247
(<http://agentzh.org/misc/nginx/ernest/nginx-0.8.54-1.fc13.x86_64.rp
251
The following versions of Nginx should work with this module:
253
* 1.1.x (last tested: 1.1.5)
255
* 1.0.x (last tested: 1.0.8)
257
* 0.8.x (last tested: 0.8.54)
259
* 0.7.x >= 0.7.21 (last tested: 0.7.67)
261
Earlier versions of Nginx like 0.6.x and 0.5.x will *not* work.
263
If you find that any particular version of Nginx above 0.7.21 does not
264
work with this module, please consider reporting a bug.
267
Although a lot of effort has been put into testing and code tuning,
268
there must be some serious bugs lurking somewhere in this module. So
269
whenever you are bitten by any quirks, please don't hesitate to
271
1. send a bug report or even patches to <agentzh@gmail.com>,
273
2. or create a ticket on the issue tracking interface
274
(<http://github.com/agentzh/chunkin-nginx-module/issues>) provided
278
Available on github at agentzh/chunkin-nginx-module
279
(<http://github.com/agentzh/chunkin-nginx-module>).
283
* now we remove the request header Transfer-Encoding completely
284
because at least Apache will complain about the empty-value
285
"Transfer-Encoding" request header. thanks hoodoos and Sandesh
288
* now we allow DELETE requests with chunked request bodies per
291
* now we use the 2-clause BSD license.
294
* applied a patch from Gong Kaihui (龚开晖) to always call "post_handler"
295
in "ngx_http_chunkin_read_chunked_request_body".
298
* fixed a bug that may read incomplete chunked body. thanks Gong
301
* fixed various memory issues in the implementation which may cause
302
nginx processes to crash.
304
* added support for chunked PUT requests.
306
* now we always require "error_page 411 @resume" and no default
307
(buggy) magic any more. thanks Gong Kaihui (龚开晖).
310
* we now use ragel -G2 to generate the chunked parser and we're 36%
313
* we now eagerly read the data octets in the chunked parser and we're
317
* added support for "chunk-extension" to the chunked parser as per RFC
318
2616 (<http://tools.ietf.org/html/rfc2616#section-3.6.1>), but we
319
just ignore them (if any) because we don't understand them.
321
* added more diagnostic information for certian error messages.
324
* implemented the chunkin_max_chunks_per_buf directive to allow
325
overriding the default 512 setting.
327
* we now bypass nginx's discard requesty body bug
328
(<http://nginx.org/pipermail/nginx-devel/2009-December/000041.html>)
329
by requiring our users to define explicit "411 error_page" with
330
chunkin_resume in the error page location. Thanks J for reporting
333
* fixed "r->phase_handler" in our post read handler. our handler may
334
run one more time before :P
336
* the chunkin handler now returns "NGX_DECLINED" rather than "NGX_OK"
337
when our "ngx_http_chunkin_read_chunked_request_body" function
338
returns "NGX_OK", to avoid bypassing other access-phase handlers.
341
* turned off ddebug in the previous release. thanks J for reporting
345
* fixed a regression that ctx->chunks_count never incremented in
349
* now we no longer skip those operations between the (interrupted)
350
ngx_http_process_request_header and the server rewrite phase. this
351
fixed the security issues regarding the internal directive as well
354
* try to ignore CR/LF/SP/HT at the begining of the chunked body.
356
* now we allow HT as padding spaces and ignore leading CRLFs.
358
* improved diagnostic info in the error.log messages when parsefail
362
* added a random valid-chunked-request generator in t/random.t.
364
* fixed a new connection leak issue caught by t/random.t.
367
* fixed a serious bug in the chunked parser grammer: there would be
368
ambiguity when CRLF appears in the chunked data sections. Thanks J
372
* fixed gcc compilation errors on x86_64, thanks J for reporting it.
374
* used the latest Ragel 6.6 to generate the "chunked_parser.c" file in
378
* marked the disgarded 411 error page's output chain bufs as consumed
379
by setting "buf->pos = buf->last". (See this nginx-devel thread
380
(<http://nginx.org/pipermail/nginx-devel/2009-December/000025.html>)
383
* added the chunkin_keepalive directive which can enable HTTP 1.1
384
keep-alive and HTTP 1.1 pipelining, and defaults to "off".
386
* fixed the "alphtype" bug in the Ragel parser spec; which caused
387
rejection of non-ascii octets in the chunked data. Thanks J for his
390
* added "Test::Nginx::Socket" to test our nginx module on the socket
391
level. Thanks J for his bug report.
393
* rewrote the bufs recycling part and preread-buf-to-rb-buf transition
394
part, also refactored the Ragel parser spec, thus eliminating lots
397
* provided better diagnostics in the error log message for "bad
398
chunked body" parsefails in the chunked parser. For example:
400
2009/12/02 17:35:52 [error] 32244#0: *1 bad chunked body (offset 7, near "4^M
404
", marked by " <-- HERE ").
405
, client: 127.0.0.1, server: localhost, request: "POST /main
406
HTTP/1.1", host: "localhost"
408
* added some code to let the chunked parser handle special 0-size
409
chunks that are not the last chunk.
411
* fixed a connection leak bug regarding incorrect "r->main->count"
412
reference counter handling for nginx 0.8.11+ (well, the
413
"ngx_http_read_client_request_body" function in the nginx core also
414
has this issue, I'll report it later.)
417
* minor optimization: we won't traverse the output chain link if the
418
chain count is not large enough.
421
This module comes with a Perl-driven test suite. The test cases
422
(<http://github.com/agentzh/chunkin-nginx-module/tree/master/test/t/>)
424
(<http://github.com/agentzh/chunkin-nginx-module/blob/master/test/t/sani
425
ty.t>) too. Thanks to the Test::Base
426
(<http://search.cpan.org/perldoc?Test::Base>) module in the Perl world.
428
To run it on your side:
431
$ PATH=/path/to/your/nginx-with-chunkin-module:$PATH prove -r t
433
You need to terminate any Nginx processes before running the test suite
434
if you have changed the Nginx server binary.
436
At the moment, LWP::UserAgent
437
(<http://search.cpan.org/perldoc?LWP::UserAgent>) is used by the test
439
(<http://github.com/agentzh/chunkin-nginx-module/blob/master/test/lib/Te
440
st/Nginx/LWP.pm>) for simplicity.
442
Because a single nginx server (by default, "localhost:1984") is used
443
across all the test scripts (".t" files), it's meaningless to run the
444
test suite in parallel by specifying "-jN" when invoking the "prove"
447
Some parts of the test suite requires modules proxy and echo to be
448
enabled as well when building Nginx.
451
* May not work with certain 3rd party modules like the upload module
452
(<http://www.grid.net.ru/nginx/upload.en.html>) because it
453
implements its own request body reading mechanism.
455
* "client_body_in_single_buffer on" may *not* be obeyed for short
456
contents and fast network.
458
* "client_body_in_file_only on" may *not* be obeyed for short contents
461
* HTTP 1.1 pipelining may not fully work yet.
464
* make the chunkin handler run at the end of the "access phase" rather
467
* add support for "trailers" as specified in the RFC 2616
468
(<http://tools.ietf.org/html/rfc2616#section-3.6.1>).
470
* fix the known issues.
473
You'll be very welcomed to submit patches to the author or just ask for
474
a commit bit to the source repository on GitHub.
477
Zhang "agentzh" Yichun (章亦春) *<agentzh@gmail.com>*
479
This wiki page is also maintained by the author himself, and everybody
480
is encouraged to improve this page as well.
483
The basic client request body reading code is based on the
484
"ngx_http_read_client_request_body" function and its utility functions
485
in the Nginx 0.8.20 core. This part of code is copyrighted by Igor
488
Copyright (c) 2009, Taobao Inc., Alibaba Group ( http://www.taobao.com
491
Copyright (c) 2009, 2010, 2011, Zhang "agentzh" Yichun (章亦春)
494
This module is licensed under the terms of the BSD license.
496
Redistribution and use in source and binary forms, with or without
497
modification, are permitted provided that the following conditions are
500
* Redistributions of source code must retain the above copyright
501
notice, this list of conditions and the following disclaimer.
503
* Redistributions in binary form must reproduce the above copyright
504
notice, this list of conditions and the following disclaimer in the
505
documentation and/or other materials provided with the distribution.
507
* Neither the name of the Taobao Inc. nor the names of its
508
contributors may be used to endorse or promote products derived from
509
this software without specific prior written permission.
511
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
512
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
513
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
514
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
515
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
516
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
517
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
518
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
519
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
520
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
521
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
524
* The original thread on the Nginx mailing list that inspires this
525
module's development: "'Content-Length' header for POSTs"
526
(<http://forum.nginx.org/read.php?2,4453,20543>).
528
* The orginal announcement thread on the Nginx mailing list: "The
529
chunkin module: Experimental chunked input support for Nginx"
530
(<http://forum.nginx.org/read.php?2,22967>).
532
* The original blog post
533
(<http://agentzh.spaces.live.com/blog/cns!FF3A735632E41548!481.entry
534
>) about this module's initial development.
536
* The thread discussing chunked input support on the nginx-devel
537
mailing list: "Chunked request body and HTTP header parser"
538
(<http://nginx.org/pipermail/nginx-devel/2009-December/000021.html>)
541
* The [module (HttpEchoModule)] for Nginx module's automated testing.
543
* RFC 2616 - Chunked Transfer Coding
544
(<http://tools.ietf.org/html/rfc2616#section-3.6.1>).