~ubuntu-branches/ubuntu/hardy/lighttpd/hardy-updates : revision 14

24

25

unified test-scripts to use the same framework

26

27

14.06.2005 23:38

27

14.06.2005 23:38

28

- documentation

29

30

updated docs for mod_proxy and mod_scgi

31

32

12.06.2005 12:55 - pre-release

33

- mod_fastcgi

34

35

fixed event handling after delayed connect

36

37

make retry timeout configurable

85

fixed automatic redirects if SSL is used

86

87

fixed compilation on Solaris [283]

88

89

disable accept-filtering on FreeBSD if ssl is used [320]

90

91

fixed handling of range-request larger than 2Gb

92

93

fixed comparisions of mime-types to case-insensitive [266]

94

95

added quotation of ETags [376], [378]

121

- fastcgi

122

123

fixed path-info for prefix-patterns

124

125

fixed path-info for PHP

126

127

02.03.2005 14:47 - 1.3.12

131

initd and not to lighttpd. Moving the daemonize before starting

132

the fastcgi procs fixes this.

133

134

01.03.2005 23:59

135

- irix

134

01.03.2005 23:59

135

- irix

136

137

fixed minor compile issues with MIPSpro

138

142

handle most duplicate headers as 400, Bug #25

143

144

- mod_fastcgi

145

146

added more checks

147

148

28.02.2005 18:26

156

really compare timestamps, Bug #34

157

158

28.02.2005 11:54

159

- buffer

159

- buffer

160

161

rewrote int2buffer functions

162

173

174

- cygwin

175

176

moved functions used by plugin and main-program to *-glue.c

177

176

moved functions used by plugin and main-program to *-glue.c

177

178

- lemon

179

180

removed warnings about shadowed variables

190

- test cases

191

192

rewrote to test-framework to use Perl + Test::More

193

ported more test-cases

193

ported more test-cases

194

195

22.02.2005 01:20

196

- mod_fastcgi

251

compile error on MacOS X due to missing environ (patch by Johan S�rensen)

252

(Bug #2)

253

254

- indexfiles

254

- indexfiles

255

256

append the detected indexfile only once to uri.path (reported by Thomas

257

Seifert)

258

(Bug #3)

259

260

06.02.2005 15:16 - 1.3.10

261

- fastcgi

262

263

display a error-message if a hostname if specified in fastcgi.server->host

264

we need an IP here

265

266

- debug

267

268

added debug.log-state-handling

269

270

- spawn-fcgi

271

272

accept a full commandline for spawning

275

- fastcgi

276

277

fixed openssl handling

278

279

- network_freebsd_sendfilev

280

281

gracefull handling of connections closed on client-side

294

- fastcgi

295

296

added bin-environment to setup the environment of the spawned process

297

298

added bin-copy-environment to copy only the specified set of options

299

from the old environment

300

301

added handling of cmd-line options to bin-path

302

303

- setenv

303

- setenv

304

305

fixed crashed in setenv.add-response-header

306

308

- configure

309

310

fixed docs for --with-mysql

311

312

- fastcgi

313

314

improved performance of building the header (drop strlen())

315

316

04.02.2005 01:59

316

04.02.2005 01:59

317

- cgi

318

319

don't send file on error

320

321

check if cgi-handler exists before executing it

322

323

added support for nph-...

324

325

02.02.2005 21:18 - pre-release

336

- fastcgi

337

338

delete sockets on shutdown

339

340

- http/1.1

341

342

adding option to disable http/1.1

343

344

01.02.2005 12:03

352

- configure

353

354

make check for valgrind.h covered by --with-valgrind

355

356

- mod_localizer, mod_maps

357

358

remove both plugins from the distribution

359

360

- file-not-found

361

362

handle file not found again

363

364

30.01.2005 16:44

365

- HEAD requests

366

368

369

30.01.2005 15:16 - 1.3.8

370

- network-handler

371

372

remove debug output on writev() if the remote side closed the connection

373

374

- directory index

375

376

handle EACCES correctly

379

- mod_alias

380

381

fixed mod_alias + pathinfo handling

382

383

- mod_accesslog

384

385

added access-log to syslog patch from allan

387

28.01.2005 17:30

388

- directory redirect without Host-header

389

390

use server-ip instead of client-ip for the Location:

390

use server-ip instead of client-ip for the Location:

391

392

- fastcgi + pathinfo

393

394

if fastcgi-auth redirects to a directory which doesn't exist handle it

395

correctly (bug introduced in 1.3.8)

396

397

- requesting directories

398

399

clean physical.path if directory is requested and dir-listing is disabled

409

- fastcgi

410

411

send content and headers if authorizer mode is used

412

413

use a new connection if connection is died to fastcgi

414

and we have not used it yet

415

423

424

fixed write-failed after crash of fastcgi-child

425

426

16.01.2005 20:43

426

16.01.2005 20:43

427

- setenv

428

429

fixed setenv.add-environment

475

report an error if ssl.engine is enable but no ssl support compiled in

476

477

08.01.2005 12:23

478

- mod_status

478

- mod_status

479

480

added request time to the output

481

482

(late changelog) added host and filename to the output (fobax)

483

(late changelog) HTMLalized the output (fobax)

484

493

494

reworked the error-page handling

495

496

05.01.2005 13:10

496

05.01.2005 13:10

497

- keep-alive handling

498

499

made sure that keep-alive is really handled correctly

502

- mod_setenv

503

504

added a module to added request and response headers on the fly

505

506

- error-log

507

508

send error log to syslog() if no errorlogfile is specified (again)

512

513

cut of body for status 301, 304 and 205

514

515

- buffer

515

- buffer

516

517

optimized all _hex functions (Silvan Minghetti)

518

520

- fastcgi

521

522

if bin-path is not specified, don't die (bug introduced in the last pre-rel)

523

524

- auth

525

526

if userfile is empty don't auth.

533

02.01.2005 16:10

534

- conditional config

535

536

fixed !~ and !=

537

538

- buffer

536

fixed !~ and !=

537

538

- buffer

539

540

copy empty buffers correctly

541

542

31.12.2004 17:45

543

- ipv6 + pidfile

544

545

don't complain if we can't remove the pidfile (Silvan Minghetti)

546

547

remove ipv6 option from the commandline of lighttpd doesn't support

546

547

remove ipv6 option from the commandline of lighttpd doesn't support

548

ipv6 (Silvan Minghetti)

549

550

31.12.2004 15:41 - pre-release

571

572

28.12.2004 23:26

573

- traffic shaping

574

575

added traffic shaping per connection

576

577

25.12.2004 22:58

585

added check for signal and select (compile fix for netbsd 1.4 and 1.5)

586

587

11.12.2004 12:38 - 1.3.7

588

- fastcgi + php

588

- fastcgi + php

589

590

retry to connect to another PHP child if one of them dies after

590

retry to connect to another PHP child if one of them dies after

591

connect

592

593

- cgi + multipart

594

595

don't transform CONTENT_TYPE to HTTP_CONTENT_TYPE

596

597

- debian

598

599

more cleanup, updated changelog, added more deps and suggests

601

602

10.12.2004 22:33

603

- event handler

604

605

fixed crashes in kqueue

604

605

fixed crashes in kqueue

606

607

10.12.2004 13:57 - pre-release

608

- mod_status

608

- mod_status

609

610

fixed wraparound in total requests and total traffic

611

612

- debian

613

614

updated licence and packaging

615

616

- security

617

618

call setgroups() to get rid of all groups

619

620

- ssl

621

622

handle SSL_shutdown() == 0 correctly

623

624

fixed openssl detection in configure

625

626

fixed handling of chunked encoding

627

625

626

fixed handling of chunked encoding

627

628

- request handling

629

630

handle Connection: keep-alive correctly (case as not ignored)

631

632

21.11.2004 02:39

637

20.11.2004 18:43

638

- conditional

639

640

ported

640

ported

641

- cgi

642

- secdownload

643

- expire

644

- localizer

645

- usertrack

645

- usertrack

646

- status

647

- proxy

648

649

- server-tag

650

651

Server: ... can now be specified by server.tag = "..."

652

653

- spawn-fcgi

654

655

fixed typo in usage text

656

657

- ssl

658

659

fixed detection of libs and headers

666

04.11.2004 23:01

667

- ssi

668

669

added support for ${...}

669

added support for ${...}

670

671

03.11.2004 14:51 - 1.3.6

672

- fastcgi

681

682

02.11.2004 15:34

683

- fastcgi

684

685

handle END-OF-REQUEST correctly if chunk-encoding is not used

686

687

02.11.2004 10:53

694

695

add REMOTE_USER, suppress AUTHORIZATION

696

handle payloads > 4k

697

698

- mod_alias

699

700

fixed url checking

701

702

- follow-symlink

703

704

fixed config

736

fixed offset calculations

737

738

- ipv6

739

740

IPv6 might be disabled at compile-time

741

742

- rewrite

746

- auth

747

748

forgot to reset the global-config handler

749

750

- symlink

751

752

add option to disable follow-symlink

768

22.10.2004 17:00

769

- accesslog

770

771

added logging of user-supplied data via %{...}o and

771

added logging of user-supplied data via %{...}o and

772

X-LIGHTTPD-* header

773

774

22.10.2004 14:57 - pre-release

776

777

fixed configure-checks and Makefile.am's to build cleanly with a

778

cross-compiler

779

780

builds cleanly for openwrt

781

782

22.10.2004 13:03

782

22.10.2004 13:03

783

- out-of-fd

784

785

improved the out-of-fd handler

786

787

- cgi, fastcgi

788

789

set SERVER_NAME to server.name or the value submitted by Host:

790

789

set SERVER_NAME to server.name or the value submitted by Host:

790

791

- error-handler

792

793

only set old status code if it wasn't set by a handler

796

- fastcgi

797

798

don't crash on out-of-fd condition

799

800

- out-of-fd

801

802

try handle the out-of-fd condition in a sane way

805

- mod_auth

806

807

seperated auth.backend.*.userfile for plain, htpasswd and htdigest

808

809

added 'digest-auth' against 'plain-backend'

810

811

added auth.debug for debugging

812

813

16.10.2004 10:18 - 1.3.3

814

- mod_simple_vhost, mod_evhost

815

816

conditional-ized

816

conditional-ized

817

818

- mod_rrdtool

819

823

- accesslogs

824

825

cycle all access-logs

826

827

- mod_rewrite

828

829

tell the user to install pcre.h if he wants to use mod_rewrite

836

09.10.2004 16:28 - pre-release

837

- cgi

838

839

added support for \n in headers

840

839

added support for \n in headers

840

841

- mod_auth

842

843

added conditional auth

858

859

drop a unused file-cache entry after 10 seconds

860

reuse unused entries

861

862

- request-parser

863

864

accept IPv6 adresses in Host header

865

866

- tests

867

868

modified the scripts to work with zsh (check on Linux, Irix and FreeBSD)

876

- cgi + cygwin

877

878

cgi need s SYSTEMROOT environment

879

880

879

880

881

22.09.2004 08:55

882

- network

883

894

895

fixed cleanup code (matt)

896

897

21.09.2004 20:08

898

- rrdtool

897

21.09.2004 20:08

898

- rrdtool

899

900

rrdtool.db-name is now conditional

901

902

fixed check if write() failed (adam)

903

904

17.09.2004 17:50 - 1.3.0

907

added url.rewrite-final = ...

908

909

17.09.2004 15:55

910

- code cleanup

910

- code cleanup

911

912

integrated the fixes from cygwin into the main tree

913

914

- kqueue

915

916

init kqueue after daemonizing (broken since 12.09.2004 14:02)

939

- mod_proxy

940

941

pass remote-addr as X-Forwarded-For to the real server behind the proxy

942

943

- code cleanup

944

945

moved all cut'n'paste versions of the inet_ntop cache to inet_ntop_cache.c

946

947

- fcgi

948

949

don't overwrite the fd in fcgi_establish connection if connect fails. this

956

957

- last-modified

958

959

don't complain if the If-Modified-Since contains a valid comment/option

959

don't complain if the If-Modified-Since contains a valid comment/option

960

like <timestamp>; length = ...

961

962

05.09.2004 09:13

963

- expires

964

965

overwrite the Expire if it is set by a previous plugin

966

967

- conditional config

968

969

conditional config as disabled in 1.2.7 by accident

977

978

fixed hanging process if cgi-crash terminates to fast (before we read its

979

response)

980

981

- extented attributes

982

983

added xattr support, submitted by Ari

984

985

29.08.2004 16:00

986

- rrdtool

986

- rrdtool

987

988

moved the rrdtool support from mod_status into its own module mod_rrdtool

989

990

rrdtool.binary = "/usr/bin/rrdtool"

991

rrdtool.db-name = "/var/www/lighttpd.rrd"

992

993

29.08.2004 11:00 - pre-release

994

- timeouts

995

996

server.max-keep-alive-requests = 0 replaces

997

server.use-keep-alive = "disable"

998

999

added

996

server.max-keep-alive-requests = 0 replaces

997

server.use-keep-alive = "disable"

998

999

added

1000

server.max-keep-alive-idle

1001

server.max-read-idle

1002

server.max-write-idle

1003

1002

server.max-write-idle

1003

1004

- docs

1005

1006

added a entry for each config-value into configuration.txt

1007

added simple docs for

1007

added simple docs for

1008

rewrite

1009

redirect

1010

compress

1015

- config options

1016

1017

complain if no configfile is specified

1018

1019

- fastcgi

1020

1021

removed stupid allocation bug which might cause a problem in really rare

1022

cases

1023

1024

26.08.2004 22:06 - 1.2.6

1025

- optimize

1025

- optimize

1026

1027

use array_strcasecmp() in favour of strcasecmp() as it is slightly

1027

use array_strcasecmp() in favour of strcasecmp() as it is slightly

1028

faster.

1029

1030

apply the case-insentive conversion also on the last character. (adam)

1031

1032

sort the checked elements in request.c and filter apply the logic to

1033

compare some less fields, if the header is not used.

1034

1035

improved the config-patch function to use our internal buffer-compare

1036

functions instead of strcmp

1037

1039

- cgi

1040

1041

added missing cleanup code

1042

1043

- fastcgi

1044

1045

remove double-free

1048

- leaks

1049

1050

fixed some leaks in the new config code

1051

1052

- array_strcasecmp

1053

1054

fixed alignment in the improved array_strcasecmp function (adam)

1058

1059

optimized a few useless strlen() away as we either know the length from

1060

buffer->used - 1 or by sizeof(str) - 1 if it is constant.

1061

1062

optimized the 'find the \r\n\r\n' function.

1063

1064

improved the array_strcasecmp() based on another idea from (ralf)

1065

1066

- accesslog

1068

enabled the strftime cache again

1069

1070

15.08.2004 23:41

1071

- accesslog

1071

- accesslog

1072

1073

added apache-like CustomLog handling in accesslog.format

1074

1075

accesslog.format = "..."

1076

1077

15.08.2004 21:08

1078

- test-cases

1079

1080

remove testdir

1080

remove testdir

1081

1082

- configfiles

1083

1084

handle escaping of " in strings properly

1084

handle escaping of " in strings properly

1085

1086

13.08.2004 12:07

1087

- array

1089

improved inner-loop of array_strcasecmp() (ralf)

1090

1091

11.08.2004 14:14

1092

- fcgi socket

1092

- fcgi socket

1093

1094

use SUN_LEN if available

1095

1096

- keep-alive

1097

1098

disable keep-alive on request

1099

1100

server.use-keep-alive = "disable"

1101

1102

10.08.2004 15:59 - 1.2.5

1108

mod_access

1109

mod_compress

1110

mod_accesslog

1111

1112

are ported

1113

1114

10.08.2004 13:05

1114

10.08.2004 13:05

1115

- pipelining

1116

1117

fixed very stupid pipelining bug

1119

09.08.2004 22:07 - pre-release

1120

- conditional config

1121

1122

first code for conditional config

1122

first code for conditional config

1123

1124

09.08.2004 14:21

1125

- fcgi

1126

1127

fixed access to free()'d memory (doesn't create any harm)

1128

1129

- isdigit, warings

1130

1131

signed -> unsigned for 2 more isdigit() calls (adam)

1138

1139

08.08.2004 12:05

1140

- kqueue

1141

1142

use EV_SET() instead of setting the ev-struct by hand (adam)

1143

1144

- fcgi

1143

1144

- fcgi

1145

1146

fixed the EINPROGRESS handling to use getsockopt (er)

1147

fixed a leak of server is disabled (er)

1148

1149

- solaris 10 port-api

1150

1151

added a skeleton for the sol10 port api

1151

added a skeleton for the sol10 port api

1152

1153

06.08.2004 10:18

1154

- mod_ssi

1160

1161

dropped usage of MAX() in buffer.c

1162

added prober includes for md5.h if openssl is enabled (brad)

1163

1164

- memory usage

1165

1166

documented the way how lighttpd caches memory blocks

1167

reset the buffers after they have been written by the network-layer

1168

1169

- kqueue

1170

1171

modify fd-bitmask only if kevent succeeded (adam)

1180

- Makefile

1181

1182

fixed dependencies for parallel build in mod_ssi_expr.c

1183

1184

- combo patch

1185

1186

* Tinker with kqueue(). Add a reset method so that the kqueue file

1188

in that adds and deletes are sent to the notification mechanism im-

1189

mediately, which should cut down on phantom events. Use

1190

ev->kq_results as a sliding window.

1191

1192

* Change F_SETFD calls to use the preferred FD_CLOEXEC instead of 1.

1193

1194

* Remove unnecessary fdevent fcntl handlers. It appears that the only

1195

driver that needs one is Linux RT signals.

1196

1197

* Quiet compiler warning about unused parameter.

1198

1199

* Set the close-on-exec flag for the /dev/poll and epoll_create() file

1200

descriptors.

1201

1202

* Return failure if /dev/poll could not be opened instead of logging

1203

and continuing.

1204

1205

* Detect EAGAIN after writev() failures. FreeBSD sendfile() doesn't need

1206

protection, as the man page says:

1207

1208

When using a socket marked for non-blocking I/O, sendfile() may send

1209

fewer bytes than requested. In this case, the number of bytes success-

1210

fully written is returned in *sbytes (if specified), and the error

1211

EAGAIN is returned.

1212

1213

(adam, georg, matt)

1214

1215

1217

- mod_ssi

1218

1219

check for pcre before compiling the module

1220

1221

- fdevents

1222

1223

dropped fdevent_fcntl added by the last patch (adam)

1244

fd in one kqueue. The handler maintains a separate bitmask for fd

1245

filters (1 == EVFILT_IN, 0 == EVFILT_OUT).

1246

(adam)

1247

1248

- server side includes

1249

1250

added native server-side includes based on the docs from apache:

1251

http://httpd.apache.org/docs/mod/mod_include.html

1252

1253

not supported are:

1254

- exec

1255

- nested virtual

1254

- exec

1255

- nested virtual

1256

- config.errmsg

1257

- echo.encoding

1258

1259

24.07.2004

1259

24.07.2004

1260

- fdevents

1261

1262

added a bitset to figure out if we received a event for an unregistered fd

1270

- configfile

1271

1272

parse keys correctly that contain a digit (Geoff Adams)

1273

1274

- fcgi

1275

1276

fixed large post uploads (Geoff Adams)

1281

1282

rewrite uri.path to the index-file instead of keeping it at .../

1283

this fixes index-file handling in FastCGI/CGI docroot is used

1284

1285

- close-on-exec

1284

1285

- close-on-exec

1286

1287

enable close-on-exec handling to simplify FD handling in CGI code

1288

1289

- cgi

1290

1291

keep error-log-fd open to catch the error handling for execve()

1292

1293

report error if cgi-exec file doesn't exist

1294

1295

- proxy

1296

1297

pass page-content on error to the user (E.R.)

1298

code cleanup (E.R.)

1299

1300

- ssi

1299

1300

- ssi

1301

1302

first skeleton of a plugin for ServerSideIncludes

1303

1304

- security

1305

1306

limit the headerlength again to 64k max

1321

1322

connect stdin, stdout and stderr to /dev/null instead of just closing it

1323

use dup2() instead of dup()

1324

1325

- accesslog

1326

1327

if accesslog.filename starts with a | spawn a process which will get data

1362

- mmap cache

1363

1364

fixed mmap-caching in network_write.c and network_writev.c after a direct

1365

hint by E.Rodichev

1365

hint by E.Rodichev

1366

1367

- sendfile + linux

1368

1391

response headers behaved the same way but are not grouped anymore. They

1392

stay seperated. Actually they are concated by \r\n<key>: <value> which is

1393

the same in the end.

1394

1395

- file uploads

1396

1397

the handling of longer post requests is fixed now.

1406

1407

the code cleanup is finished successfully. Now all cases of pipelining are

1408

handle the right way. POST pipelining was still not working up to now.

1409

1410

22.05.2004 12:55

1411

- code cleanup

1412

1417

- network backends

1418

1419

enabled sendfile support again (__FreeBSD__ instead of __freebsd__)

1420

1421

added a mmap cache as part of the filedescriptor cache

1422

1423

added AIX specific send_file() support (untested)

1424

1425

20.05.2004 10:40

1431

- documentation

1432

1433

reformated the documentation the doc/ directory

1434

1435

15.05.2004 14:45

1434

1435

15.05.2004 14:45

1436

- localizer

1437

1438

fixed build of localizer extension

1441

- POST requests

1442

1443

there is no need to die if we spot a simple POST request on a static file

1444

1445

- pipelining

1446

1447

fixed HTTP/1.1 pipelining which caused the problems with opera

1458

1. the application (.php, ...) controls the access to the files

1459

2. the webserver is handling the transfer (and check the app generated

1460

tokens)

1461

1462

the token is based on

1461

1462

the token is based on

1463

- a secret

1464

- a timestamp

1465

- the filename

1466

this means even if the token is is distributed by the user it will get

1467

invalid after a given timeout (default 60 seconds)

1468

1469

- errorfiles

1470

1471

check for errorfiles before using them

1472

1473

- code cleanup

1474

1475

applied cleanup fixes from adam

1477

14.05.2004 18:47

1478

- fdevent handling

1479

1480

added a more generic callback interface to the fdevent structures which

1480

added a more generic callback interface to the fdevent structures which

1481

simplifies the writing of plugins. this might destabilize lighttpd for a

1482

while

1483

1484

- cgi

1485

1486

fixed header parsing of the header is sent in chunks and the terminator is

1487

sent in a single chunk

1488

1489

- EINTR

1490

1491

fixed some occurences of EINTR which read()

1501

1502

added incomplete support for Expect: 100-continue (RFC 2616 8.2.3) by

1503

sending always 417 for every Expect-request (see 14.20)

1504

1505

we have been blamed for not supporting it:

1504

1505

we have been blamed for not supporting it:

1506

http://lists.w3.org/Archives/Public/ietf-http-wg/2004JanMar/0059.html

1507

1508

29.04.2004 23:07 - 1.1.9

1512

1513

29.04.2004 19:37

1514

- network-writev

1515

1516

Adam:

1517

* call munmap() on error in write()

1518

1519

- docs

1520

1521

E.R:

1522

* documented 'check-local'

1523

1522

* documented 'check-local'

1523

1524

- test-env

1525

1526

made the shell scripts more portable (checked with zsh, bash and ksh)

1527

1528

fixed compilation on NetBSD

1529

1530

28.04.2004 22:22

1538

1539

Alexey Slynko:

1540

* handle SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE in SSL_write the

1541

right way.

1541

right way.

1542

1543

- FastCGI

1544

1560

- mem-leak

1561

1562

fixed mem-leak on broken HTTP-headers

1563

1564

- FastCGI

1565

1566

patches from E.Rodichev:

1567

1568

1. CONTENT_LENGTH

1569

"If no data are attached, then this metavariable is either NULL or not

1570

defined".

1571

1572

2. QUERY_STRING

1573

If the Script-URI does not include a query component, the QUERY_STRING

1574

metavariable MUST be defined as an empty string ("").

1575

1576

3. Doubling of REMOTE_ADDR removed.

1577

1578

patches from adam:

1579

* fixed bug in the sizeof() patches from yesterday.

1580

* some more *_long to *_off_t

1580

* some more *_long to *_off_t

1581

1582

24.04.2004 14:34

1583

- FastCGI

1588

- POST

1589

1590

fixed POST request handling

1591

1592

- chunk-encoding

1593

1594

the generated HEX strings where broken since the 1.1.8

1601

* sizeof(int) -> sizeof(<variable>)

1602

* assign fd_set instead of memcpy()

1603

* init fd -> connection fd pointers to -1

1604

1605

16.04.2004 08:48 - 1.1.8

1606

- code cleanup

1607

1608

don't reuse buffer > 64k (see settings.h)

1609

1610

added server.max-request-size to limit the maximum request-body size

1609

1610

added server.max-request-size to limit the maximum request-body size

1611

(in kBytes)

1612

1613

don't accept HTTP-request headers larger then 32kBytes (see settings.h)

1614

1615

minor speed improvements in the request-parser

1616

1617

More cleanup patches from adam:

1618

1619

* change pre-ANSI C/valid C++ syntax for function declarations/definitions

1620

from using () to (void). Ex: int foo(); --> int foo(void);

1621

* use static linkage as much as possible, to limit possible symbol

1622

collisions

1623

* whack more unneeded variables

1624

* try and prevent any errno clobbering by storing the old errno value before

1625

any subsequent system calls, and restoring before function exit.

1625

any subsequent system calls, and restoring before function exit.

1626

* change printf syntax for unsigned variables from %d to %u

1627

1628

15.04.2004 18:41

1629

- code cleanup

1630

1631

handle all int != size_t cases in fcgi.c correctly

1632

1633

check headerfields to have a value

1634

1635

handle both EINVAL cases of writev() before the can occur

1636

1637

limit content-length to SSIZE_MAX

1636

1637

limit content-length to SSIZE_MAX

1638

disallow negative content-length

1639

1640

the usage of ltostr() has been reduced to the minimum in favour of

1639

1640

the usage of ltostr() has been reduced to the minimum in favour of

1641

buffer_/append|copy)_(long|off_t)

1642

1643

dropped ultostr() and ultohex() in favour of buffer_*

1644

1645

15.04.2004 16:35

1646

- portablity

1646

- portablity

1647

1648

more patches from adam:

1649

1650

* remove warnings for unused parameters and variables

1651

* remove warnings for mismatched pointer assignments

1652

* change "gtime_r" to "gmtime_r"

1653

1654

13.04.2004 20:48

1655

- test-cases

1655

- test-cases

1656

1657

made the 'make check' target self-contained

1658

1659

In our case we have to call the targets in the following order:

1660

1661

$ ./configure ...

1662

$ make

1662

$ make

1663

$ make install

1664

$ make check

1665

1666

because the path to the plugins is hardcoded in the binary itself

1667

1668

using

1669

1667

1668

using

1669

1670

$ ./configure --prefix=/tmp/lighttpd-1.1.x/

1671

...

1672

1673

will help if you don't really want to install before testing.

1674

1675

13.04.2004 00:05

1676

- portability

1676

- portability

1677

1678

adam sent another patchset:

1679

1680

* Wrap PCRE-specific data member access with an #ifdef

1681

* Add const to pointer using return value from dlerror()

1682

* Explicitly initialize pointer in the lemon parser to 0,

1688

- fastcgi

1689

1690

strip WS after HTTP-response headers coming from the FastCGI process

1691

1692

added REMOTE_USER to the Server->FastCGI headers

1693

removed HTTP_AUTHORIZATION from the Server->FastCGI headers

1694

1695

12.04.2004 10:24

1696

- cgi

1697

1698

if we don't get a partial HTTP-response-header send the content out as soon

1698

if we don't get a partial HTTP-response-header send the content out as soon

1699

as the cgi script is finished

1700

1701

12.04.2004 01:23

1706

12.04.2004 00:12

1707

- configfile

1708

1709

add some usefull error messages if the tokenizer or the parser fail to

1709

add some usefull error messages if the tokenizer or the parser fail to

1710

read the configfile

1711

1712

11.04.2004 22:04

1717

11.04.2004 20:28

1718

- 64bit offset size

1719

1720

disable linux-sendfile support for linux 2.4.x for now as it don't

1721

support 64bit transfers

1722

1720

disable linux-sendfile support for linux 2.4.x for now as it don't

1721

support 64bit transfers

1722

1723

fixed all assignments on the path from the stat() to the Content-Length

1724

HTTP-header

1725

1726

- head requests

1727

set content-length in HEAD requests

1728

1729

- accesslog

1730

1731

write accesslog entry on network error

1732

1733

write the correct amount of byte written to the accesslog

1734

1735

11.04.2004 11:48

1736

- code cleanup

1737

1738

moved the config for the cgi-plugin from config.c to the plugin.

1739

1740

moved some buffers which were only used by a one or two plugin from

1741

the server-structure to the plugins

1742

1743

keeping the plugins independent from the server-core is a 'good thing'

1744

1745

10.04.2004 19:06

1758

the hand-written configfile parser has been replaced by a LALR(1) one.

1759

'lemon' from the sqlite guys has been used to generate the parser.

1760

1761

- by-host, by-url, by-file, by ...

1761

- by-host, by-url, by-file, by ...

1762

1763

$HTTP["url"] =~ "~$" {

1764

access.deny = "all"

1765

}

1766

1767

$HTTP["host"] =~ "." {

1768

simple-vhost.server-root = "/home/weigon/wwwroot/servers/"

1769

simple-vhost.default-host = "grisu.home.kneschke.de"

1770

simple-vhost.document-root = "pages"

1771

}

1772

1773

$HTTP["host"] == "incremental.home.kneschke.de" {

1774

server.docroot = "/hasdasd"

1775

}

1776

1777

at least the parser can handle it now. Currently there is no real support

1778

for this context-based config-option. But the syntax and the parser are

1779

done.

1780

1781

09.04.2004 10:58

1782

- ssl support

1783

1784

enable ssl support again

1785

1786

- mmap

1787

1788

enabled mmap+write again

1789

1788

enabled mmap+write again

1789

1790

08.04.2004 12:34

1791

- stricter http-parser

1792

1793

based on a thread at

1794

1793

based on a thread at

1794

1795

http://lists.w3.org/Archives/Public/ietf-http-wg/2004JanMar/0050.html

1796

1797

the HTTP-parser has been adjusted to be more correct when it comes to

1796

1797

the HTTP-parser has been adjusted to be more correct when it comes to

1798

request-header fieldnames

1799

1800

the whitespace handling around the ':' has been relaxed as requested

1801

1802

07.04.2004 17:06

1803

- sigaction

1804

1823

- pidfile

1824

1825

fixed the permissions of the pidfile (Matthijs van der Klip)

1826

1827

- specfile

1828

1829

merge the RedHat and SuSE specfile with Matthijs

1830

1831

- 64bit file-offsets

1832

1833

moved the FILE_OFFSET_BITS settings from the config.h the Makefile to enable

1838

1839

added an apache compatible mod_expire which adds Expires: headers to the

1840

request

1841

1842

expire.url = ( "/buggy/" => "access 2 hours" )

1843

1844

05.04.2004 22:34

1847

forgot to provide the infrastructure to actually enable the devpoll event

1848

handler. the same has been done for the freebsd-kqueue handler (which

1849

doesn't work yet)

1850

1851

fixed the devpoll support while testing it on a real solaris box

1852

1853

05.04.2004 09:49

1854

- debian

1855

1856

added debian packaging support written by Vincent Wagelaar

1857

1858

- solaris

1859

1860

Another set of patches for Solaris from Adam

1879

- fdevent

1880

1881

added framework for freebsd_kqueue and solaris_devpoll

1882

1883

the solaris_devpoll one might event work (untested)

1884

1885

03.04.2004 16:41

1886

- network

1886

- network

1887

1888

added framework for filebased chunks

1889

- read-write + mmap-write

1890

- linux-sendfile

1891

- freebsd-sendfile

1892

- solaris-sendfilev (untested)

1893

1894

and memorybased chunks

1895

- write

1896

- writev

1897

1898

made TCP_CORK a 'global' flag around the write_chunkqueue-calls

1899

1900

the writev() support should improve the performance for all non-static

1901

pages.

1902

1901

pages.

1902

1903

170 req/s against 158 req/s for the following script if writev() is used

1904

instead of write()

1905

1906

<?php

1907

1908

for ($i = 0; $i < 1000; $i++) {

1909

print $i." \n";

1910

flush();

1911

}

1912

1913

?>

1914

1915

28.03.2004 13:42

1921

* STD{IN,OUT,ERR}_FILENO preferred instead of {0,1,2}

1922

* dieing --> dying

1923

* SEGFAULT calls abort directly, instead of derefencing a NULL pointer

1924

1925

- mod_accesslog

1926

1927

modified the accesslog format to be CLF compatible

1928

set locale for LC_TIME to C

1929

1930

26.03.2004 16:13

1929

1930

26.03.2004 16:13

1931

- path info

1932

1933

enabled the pathinfo code again

1934

1935

1936

25.03.2004 13:30 - 1.1.3

1937

- portability

1937

- portability

1938

1939

compiles fine now without any patches on IRIX

1940

1941

- hostname detection

1942

1943

reworked "get the hostname for HTTP/1.0 requests which don't specify a

1944

Host: ..." to only query the name in this single case for the server side

1945

of the connection

1946

1947

- errorlog handling

1948

1949

stderr is only used until the errorlog is init'ed

1950

if no error-log is specified, syslog() is used

1951

if cycling error-log fails, syslog() is used

1952

1953

- accesslog cycling

1954

1955

don't fall back to stdout anymore

1967

- configure

1968

1969

some protability changes to get the 'inline' working with the MIPS CC

1970

1971

21.03.2004 22:00

1972

- mod_rewrite, mod_redirect

1973

1974

reading the config for those two plugins was not working

1975

1976

- fdevents

1977

1978

changed the compile time setting for the event handling into a run-time

1979

setting.

1980

1981

server.event-handler = "select" # poll, linux-rtsig, linux-sysepoll

1982

1983

added sys_epoll() for linux 2.6

1984

1985

select - all systems

1986

poll - Linux 2.1.23+, all XPG4-UNIX

1987

sigio - linux 2.4.0+

1987

sigio - linux 2.4.0+

1988

sysepoll - linux 2.5.66+

1989

1990

1000-4k-nok 1000-100k-nok

1991

select 1776.99 296.52

1992

poll 678.02 607.28

2003

2004

as the docroot on external hosts might be different than the webserver

2005

docroot it can now be specified in the config:

2006

2007

fastcgi.server = ( ".php" =>

2008

( "grisu" =>

2009

(

2008

( "grisu" =>

2009

(

2010

"host" => "192.168.2.41",

2011

"docroot" => "/home/jan/servers/",

2012

"port" => 1026

2013

)

2014

)

2015

)

2016

2017

a huge internal cleanup in the config handling made the code more readable.

2018

some more warnings and error checking should track most of the config

2019

errors for the fastcgi plugin

2019

errors for the fastcgi plugin

2020

2021

19.03.2004 12:34

2022

- external patches

2026

- fix for a debug message

2027

- get the hostname for HTTP/1.0 requests which don't specify a Host: ...

2028

- rc-script for RedHat

2029

2030

- documentation

2031

2032

added a documentation section about authentification

2035

19.03.2004 05:11

2036

- optimizations

2037

2038

implemented special versions of

2038

implemented special versions of

2039

- strcasecmp (array_strcasecmp)

2040

- isdigit, isalpha, isalnum (light_*)

2041

2042

added a faster check for a finished header

2043

2044

after disabling all modules it is still possible to get 20 kreq/s.

2045

2046

15.03.2004 19:45 - 1.1.1

2047

- localizer server

2048

2049

added the localizer-server module to the code

2050

2049

added the localizer-server module to the code

2050

2051

- chunked-encoding

2052

2053

Apple's Safari or HTTP-Handler doesn't handle chunked-extensions defined by

2053

Apple's Safari or HTTP-Handler doesn't handle chunked-extensions defined by

2054

RFC 2616 correctly and doesn't ignore them. Disabled the chunked-extension

2055

which were just used for debugging purposes.

2056

2059

2060

moved the checks if a plugins support a given function from the dispatcher

2061

(plugins_call) to plugins_call_init() to do the check only once.

2062

2063

equalized the plugins functions to only two types:

2064

SERVER_FUNC()

2065

CONNECTION_FUNC()

2066

2067

replaced all handwritten plugin_call-handlers with macros

2068

2069

made all plugin-functions 'static'

2070

2071

with all plugins loaded we are at 15kreq/s gain

2072

without any plugin loaded at 16kreq/s

2073

2074

before the change we were at 13kreq/s

2075

2076

06.03.2004 21:13

2076

06.03.2004 21:13

2077

- compilation fixes

2078

2079

fixed some warnings on FreeBSD and NetBSD by adding

2080

#include <netinet/in.h>

2081

2080

#include <netinet/in.h>

2081

2082

ifdef'ed a pcre-entry in base.h

2083

2084

remove #define _XOPEN_SOURCE from http_auth.c for IRIX

2085

crypt() on Linux needs _XOPEN_SOURCE

2086

2089

2090

added htpasswd and htdigest backends to work against user-files generated

2091

by htpasswd and htdigest.

2092

2093

for basic auth:

2094

- plain

2095

- htpasswd (crypt only)

2096

- htdigest

2097

2098

for digest auth:

2099

- plain

2100

- htdigest

2101

2102

06.03.2004 17:35

2103

- authentification

2104

2105

check the method in the authorization header againt the configured method

2105

check the method in the authorization header againt the configured method

2106

2107

06.03.2004 14:54

2108

- hostname parsing

2114

- configuration

2115

2116

added a warning for unknown configuration variables.

2117

2118

dropped the 'specific-for.url' ideas for now as it is not known yet how to

2119

implement it the right way

2120

2121

renamed some config-variables to reflect there actual meaning

2122

- server.host -> server.bind

2123

- server.virtual-* -> simple-vhost.*

2124

- server.userid -> server.username

2125

- server.groupid -> server.groupname

2126

- server.docroot -> server.document-root

2127

2128

2129

05.03.2004 15:36

2130

- fastcgi

2133

plugin itself

2134

2135

- vhosting

2136

2137

got a patch for enhanced virtualhosting from christian kruse -> mod_evhost

2138

2139

moved the virtual hosting config (server.virtual-*) to it's own module

2140

called mod_simple_vhost

2141

2142

23.02.2004 10:06

2143

- configfile

2144

2145

rearragned the config-file structure again to be able to add settings for

2146

a given URL, Host, Filename, ...

2146

a given URL, Host, Filename, ...

2147

This change affects some config-options directly (access.deny,

2148

url.rewrite, url.redirect, virtual-hosting, ...)

2149

2150

added 'specific-for.url' for url specific config settings

2151

2149

2150

added 'specific-for.url' for url specific config settings

2151

2152

- digest auth

2153

2154

FIX: md5-sess type

2155

2156

seperate the auth-backend stuff

2157

2158

13.02.2004 22:23 - lighttpd 1.0.3

2171

2172

FIX: the search for header fields was up to now case-sensitive. Now it is

2173

like required by the standard case-in-sensitive.

2174

2175

- browser bugs

2176

2177

w3m 0.2.5 adds an additional \r\n at the end of the POST requests which is

2178

handled now

2179

2180

10.02.2004 10:12

2181

- start script

2182

2183

took the suse rc-skeleton which states to be LSB compliant and modified it

2184

for lighttpd needs

2185

2186

09.02.2004 11:48

2185

2186

09.02.2004 11:48

2187

- HEAD requests

2188

2189

FIX: HEAD requests for static files delived the content from the GET

2190

request. (test case is added)

2191

2192

08.02.2004 15:53

2192

08.02.2004 15:53

2193

- directory listings

2194

2195

FIX: the pathname has not encoded at all before it was transfered to the

2194

2195

FIX: the pathname has not encoded at all before it was transfered to the

2196

browser. a proper url and html-encoding has been added.

2197

2198

added modification-time and filesize to the output

2199

2197

2198

added modification-time and filesize to the output

2199

2200

added a basic css for the virtual listings

2201

2202

07.02.2004 22:15 - lighttpd 1.0.2

2203

- sample configfile

2204

2205

rearranged the config-file to have all the important options at the top

2206

2207

- docs

2208

2209

added a mod-rewrite options

2210

2211

- mod_accesslog

2212

2213

stdout is no longer used a default for the accesslog

2214

2215

- error-messages

2216

2217

the 404 is now reported in the error-log

2220

- configfile handling

2221

2222

if a key is used twice like

2223

2224

url.rewrite = ( "url1" => "url")

2225

url.rewrite = ( "url2" => "url")

2226

2223

2224

url.rewrite = ( "url1" => "url")

2225

url.rewrite = ( "url2" => "url")

2226

2227

you get an error now. You have to write:

2228

2229

url.rewrite = ( "url1" => "url",

2230

"url2" => "url")

2228

2229

url.rewrite = ( "url1" => "url",

2230

"url2" => "url")

2231

2232

31.01.2004 12:22 - lighttpd 1.0.1

2233

- log cycling

2234

2235

added a sighup-handler to the plugin interface and fixed the log-cycling

2236

for access-logs

2237

2238

- portability

2239

2240

disabled the interval-time optimization on IRIX

2241

2242

- documentation

2243

2244

added a lot of new documentation to README

2245

2246

31.01.2004 10:59

2247

- status module

2248

2249

added two new options rrd-dir and rrd-reports:

2250

2251

rrd-reports = (boolean) # enables RRD-reports

2252

rrd-dir = (string) # path for the daily status-files

2253

2254

fixed the "status-files in /"-bug this way

2255

2256

2257

22.01.2004 13:38 - lighttpd 1.0.0

2258

- simple docroot

2269

2270

Verified again that the code compiles and runs cleanly on Linux, FreeBSD,

2271

NetBSD and IRIX

2272

2273

compiling with gcc and the option -pedantic works fine

2274

compiling with mipspro cc works fine, too

2275

2276

- tests

2277

2278

added some more tests: 39 tests

2281

- protability

2282

2283

some compile fixes for FreeBSD have been applied and a new switch has been

2284

added to choose between IPv4 and IPv6 on FreeBSD.

2284

added to choose between IPv4 and IPv6 on FreeBSD.

2285

(cmdline: -6, configfile: server.use-ipv6)

2286

2287

- packaging

2288

2289

cleaned up the specfile for building RPMs

2290

2291

21.12.2003 01:00

2302

- compression

2303

2304

added gzip compression (gzip-header + deflate + crc)

2305

2306

- test harness framework

2307

2308

added a 22 tests to verify the correct behaviour of lighttpd

2310

- request parsing

2311

2312

GET http://www.yahoo.com/ HTTP/1.0

2313

2314

is handled now as

2315

2316

GET / HTTP/1.0

2317

2318

- plugins

2325

the directory structure is now build automaticly

2326

2327

15.12.2003 01:00

2328

- compression

2329

2328

- compression

2329

2330

added a compression cache to the compression module (mod_compress)

2331

2332

Vary: Accept-Encoding is set now as it influences to delivered stream of

2336

- config files

2337

2338

a new config-file format is ready for the final release of lighttpd.

2339

2340

it supports:

2341

2342

server.docroot = "string"

2343

server.host = integer

2344

server.modules = ( "string", "string" )

2345

server.mimetypes = ( "key" => "value" )

2346

server.complex = ( "key" => ( "string", integer ),

2347

"string",

2346

server.complex = ( "key" => ( "string", integer ),

2347

"string",

2348

integer )

2349

2350

the syntax should look familar to all who worked with Perl or PHP.

2351

2352

config-file handling has been seperated into a parser and a lexer. Both

2353

are currently written by hand and will propably be rewritten into bison +

2354

flex if time permits. But that would be a cosmetical change.

2358

2359

improved the status page to display

2360

- the current connection-stati

2361

- average throughput over 5 seconds

2361

- average throughput over 5 seconds

2362

(requests/s and output-bound traffic)

2363

2364

now you can see what is going on in the server

2365

2366

- access-log

2367

2368

the accesslog is now a module and can be disabled by just removing the

2372

- chroot-ing

2373

2374

how-to: using chroot

2375

2376

- chroot to /home/www/

2377

- docroot at /servers/<hostname>/pages/

2378

- defaulthost www.example.org

2379

2380

e.g. (external view)

2381

/home/www/servers/www.example.org/pages/index.html

2382

2383

(in chroot)

2384

/servers/www.example.org/pages/index.html

2385

2386

config:

2387

2388

chroot /home/www/

2389

userid wwwrun

2390

groupid nogroup

2391

2392

virtual-server-root /servers/

2393

virtual-server-default-host www.example.org

2394

virtual-server-docroot /pages/

2395

2396

The FastCGI process is living outside this chroot definition as it is

2395

2396

The FastCGI process is living outside this chroot definition as it is

2397

started seperatly.

2398

2399

01.12.2003 02:06

2400

- cleanup

2401

2402

in preparation for the first stable release some internals had to be

2403

cleaned up. Basicly it was a cleanup of workflow of the creation of the

2404

response-header. All modules can use a clean interface for this purpose

2403

cleaned up. Basicly it was a cleanup of workflow of the creation of the

2404

response-header. All modules can use a clean interface for this purpose

2405

now. This is espacially usefull for all modules which have to pass some

2406

HTTP-headers to the client.

2407

2410

2411

finally moved the modules to shared libraries and cleaned up some code

2412

path to become more readable.

2413

2414

So far we have:

2415

2416

mod_rewrite

2417

mod_redirect

2418

mod_access

2426

- redirects

2427

2428

as lighttpd supports url-rewriting redirection was a few lines of new code.

2429

2430

redirect ^/wishlist/(.+) http://jan.kneschke.de/wishlist/$1

2431

2432

rewrite ^/wishlist/(.+) /new/wishlist/$1

2433

2434

28.11.2003 17:00

2435

- signal stuff

2435

- signal stuff

2436

2437

setitimer is used to send a event every second to call time() only once a

2438

second. another system call which has been remove from the main-loop.

2439

2440

sending HUP to lighttpd will close and re-open the logfiles. this is used

2441

for cycling logfiles.

2442

2443

#! /bin/sh

2444

2445

###

2445

###

2446

#

2447

# a simple logfile rotator for lighttpd

2448

#

2459

a first attempt for a native win32 has been done. For now mingw is the

2460

base for the development as it provides a basic unix-like framework for

2461

building native win32 applications.

2462

2463

the most internal files have been ported and the over all progress is

2464

going well. At the end this will just be a prove of concept.

2465

2467

- access denied

2468

2469

added a access-deny filter to block specific urls like

2470

2471

access-deny ~

2472

access-deny .inc

2473

2476

2477

a fstat() on a opened fd which has changed reports wrong the file-info.

2478

Using stat() again helps to solve this problem.

2479

2480

the sig-io version doesn't suffer from this problem.

2481

2482

the etags are now used for verifing file-cache-entries.

2483

2484

- chat

2485

2486

enabled the internal login mechanism again

2487

2488

added support transfering session-infos over MySQL.

2489

2490

15.11.2003 00:19

2493

added a buffer_equal_reverse function which is optimized version of strcmp

2494

which is going backwards as pathnames are often the same for in the first

2495

bytes.

2496

2497

wrote a one-pass parser for the request-header. The combination of

2498

strstr(..., "\r\n") to seperate lines and strchr(..., ':') to seperate

2499

keys from values more or less touched every byte twice.

2500

2501

we are still at 18.000 req/s for 4kb keep-alive requests even with etags

2502

and handling all header fields.

2503

2504

14.11.2003 17:26

2505

- fcgi

2506

2507

Content-Type wasn't passed correctly to the FastCGI app. HTTP_CONTENT_TYPE

2508

was sent instead.

2509

2510

- cache

2511

2512

provided access to the Session-ID

2513

2514

- error-log

2515

2516

the timestamp is now written in a human readable form

2519

- fcgi, cgi

2520

2521

added a special set of array-functions which are optimised for the "insert

2522

only once" case.

2523

2522

only once" case.

2523

2524

this provides access to the headers which are now forwarded to the external

2525

interfaces. Before this change only a limited set of request-headers were

2526

forwarded.

2527

2528

21.10.2003 11:58

2529

- modules

2529

- modules

2530

2531

added a new module-hook after the basic-init of the module for handling

2532

config-settings and prepare the overall operartion (like building

2533

db-connections, compiling regexes, ...)

2534

2535

- cache

2536

2537

use turckmm-cache 2.4.3 to get some numbers for the php-latency:

2538

2539

/usr/sbin/ab -n 10000 -c 10 http://alba.home.kneschke.de:1025/index.php

2540

2541

handling the cache-decision and the cache-hit in php:

2542

2543

cache-miss: 100% (-> $version = 0)

2544

2545

plain : 108.13 req/s

2546

turckmm-cache: 218.39 req/s

2547

2548

cache-hit: 100% (-> $version = 1)

2549

2550

plain : 164.45 req/s

2551

turckmm-cache: 653.98 req/s

2552

2553

handling the cache-decision and the cache-hit in the server:

2554

2555

cache-hit: 100%, but using index.cml

2556

2557

cml: 4918.84 req/s (no keep-alive)

2558

cml: 6901.31 req/s (keep-alive)

2559

2560

cache-miss: 100%, but using index.cml

2561

2562

plain : 108.39 req/s

2563

turckmm-cache: 217.84 req/s

2564

2565

Conclusion:

2566

- there is no loss in the cache-miss case through the cml-handling

2567

- the cache-hit case can be improved dramaticly with lighttpd-cache

2568

- turckmm-cache improves the cache-miss case alot

2569

2570

20.10.2003 00:40

2571

- cache

2572

2573

the first 'real-life' test showed dramatic improvements in the req/s

2574

handling.

2575

2576

The basic idea was to move the decision if a php-page can be taken from

2577

the cache from the php-code to the webserver.

2578

2577

the cache from the php-code to the webserver.

2578

2579

See here why this is a good thing:

2580

2581

the quite common code which works for http://jan.kneschke.de/ is using

2582

templates and is quite static, but depends on 4 external files (the

2583

menu-structure, the template, the current content, the class-file).

2584

2585

the index-file is always:

2586

2587

include_once "jk.inc";

2588

2589

$v = new view();

2590

print $v->get(array(array ("file" => "content.html")));

2591

2592

It is more or less the same for all pages.

2593

2594

This basic setup can deliver 100 requests/s.

2595

2596

The next step has:

2593

2594

This basic setup can deliver 100 requests/s.

2595

2596

The next step has:

2597

- application bases caching

2598

2599

as we know that each pages depends on those 4 files, you can check if they

2600

have been modified since the last request and deliver the content from the

2601

cache otherwise.

2602

2603

this increased the throughput to 150 req/s. (cache-hit ratio 100%)

2604

2605

The next logic step is to move the decision-making process out of the PHP

2606

code as PHP is to slow for the cache-hit path:

2607

2608

a CML (Cache-Markup-Language) has been written which describes the whole

2609

decision process which has been written in PHP-code before:

2610

2611

2610

2611

2612

output.content-type text/html

2613

2614

output.include _cache.html

2615

2616

trigger.handler index.php

2617

2618

trigger.if file.mtime("../lib/php/menu.csv") > file.mtime("_cache.html")

2619

trigger.if file.mtime("templates/jk.tmpl") > file.mtime("_cache.html")

2620

trigger.if file.mtime("content.html") > file.mtime("_cache.html")

2621

2622

if one of the 'trigger.if' statements is true the 'trigger.handler' is

2623

called to generate the pages.

2624

2625

if none of the them is trigger the files from 'output.include' are sent to

2626

the browser with content-type specified in the first line:

2627

2628

The result was very 'promissing':

2629

2627

2628

The result was very 'promissing':

2629

2630

5900 req/s with keep-alive

2631

3800 req/s without keep-alive

2632

2633

(both for a cache-hit ratio of 100%)

2634

2635

for keep-alive this is factor 59 against the plain un-cached

2636

version and still 39 againt the php-cache-version which is doing

2637

exactly the same.

2638

2639

Time for party. :)

2640

2641

19.10.2003 - 18:55

2642

- cache

2643

2644

the handling of functions has been improved. they are now 'plugable'. just

2645

the dlopen() stuff is missing.

2646

2647

a new datatype has been added: the string

2648

2649

this makes it possible to evaluate something like:

2650

2651

trigger.if unix.time.now() - file.mtime("head.html") > 30

2652

trigger.if mysql.query("SELECT count(*) " + \

2653

" FROM structure AS struc, session AS sess " + \

2654

" WHERE struct.user = sess.user" + \

2655

" AND sess.id = \"" + mysql.escape("ab\"c") + "\"")

2656

2657

you see:

2656

2657

you see:

2658

- string operations (concat)

2659

- handing of escape-sequences

2660

- functions

2667

handler a lot easier. The CVS server was up all the time but not used.

2668

Importing required 5 minutes of work which included merge the freebsd and

2669

the linux tree.

2670

2671

- compilers + platforms

2672

2673

on a regular various compilers and platform are check to compile with with

2674

the current code base:

2675

2676

platform | os | compiler | state

2677

---------+---------------------+--------------+---------

2678

ia32 | Linux 2.4.22 | gcc 2.95.3 | ok

2679

ia32 | FreeBSD 5.1-CURRENT | gcc 3.3.1 | ok

2680

mips64 | IRIX 6.5 | gcc 3.2.2 | ok

2681

misp64 | IRIX 6.5 | MIPSpro 7.41 | ok

2682

2683

the mipspro compiler revealed some warning which resulted in a nice

2684

code-cleanup that made the code more readable.

2685

2687

- e-tags and friends

2688

2689

in preparation for the php-conference at the begin of november in

2690

frankfurt/main the server has the support some more caching/proxy tags

2691

like:

2690

frankfurt/main the server has the support some more caching/proxy tags

2691

like:

2692

- E-Tag (14.19) [done]

2693

- If-Match (14.24)

2694

- If-None-Match (14.26) [done]

2695

- If-Range (14.27)

2696

2695

- If-Range (14.27)

2696

2697

Section (13.3.3) binds them together. [RFC 2616]

2698

2699

using lxr.kde.org and lxr.mozilla.org revealed that

2700

- konqui only uses if-none-match

2701

- mozilla uses if-none-match and if-range

2702

2703

it looks like it isn't that easy to trigger the if-range case.

2704

2705

As ulf suggested the etag is a hash of file-size, inode-number and mtime.

2706

2704

2705

As ulf suggested the etag is a hash of file-size, inode-number and mtime.

2706

2707

- fd-caching

2708

2709

ulf just phoned me ask proposed the free the cache more agressivly in case

2711

2712

17.10.2003 12:45

2713

- chat

2714

2715

finished the mysql-support for storing the sessions

2716

2717

12.10.2003 20:56

2718

- valgrind

2719

2720

used valgrind again to verify that the code a free of mem-leaks and found

2721

a 'leak generator' in the chunk-api.

2722

2721

a 'leak generator' in the chunk-api.

2722

2723

the last few leaks were just some missing free()'s at the end of the

2724

program run which would have been freed anyway.

2725

2726

at the end valgrind couldn't find any missing free()'s.

2727

2728

11.10.2003 12:09

2729

- FastCGI

2730

2731

reduced the number of system calls for FastCGI to WebServer to 2 calls per

2731

reduced the number of system calls for FastCGI to WebServer to 2 calls per

2732

fd-event. (ioctl() + read())

2733

2734

this has no direct effect of the performance of the server, but improves

2735

the possible througput of the load-balancer.

2736

2739

2740

a brown paper bug has been fixed which caused to decreasing throughput if

2741

load-balancing was enabled.

2742

2743

2742

2743

2744

benchmarking the req/s with load-balancing shows really nice results:

2745

2746

2748

---------+--------------------------------------------------

2749

ulf : 764.06 (php)

2750

lappi : 800.06 (php)

2751

2752

ulf+lappi: 1526.95 (2 * php)

2753

2754

grisu : 1265.66 (php + ab + lighttpd)

2755

2756

2753

2754

grisu : 1265.66 (php + ab + lighttpd)

2755

2756

2757

all : 1647.72 (3 * php + ab + lighttpd)

2758

all(nice): 1904.40 (same as all, but the local php on grisu

2758

all(nice): 1904.40 (same as all, but the local php on grisu

2759

has been 'nice -20'd)

2760

2761

2760

2761

2762

if a php is run on the load-balancer it has to get a lower priority than

2763

the load-balancer itself as to handle the work of (here) 3 php-servers.

2764

2765

2766

10.10.2003 15:11

2767

- java ?

2773

2774

Last week 3 RTL8169S gigE cards arrived and were installed in the

2775

test-framework.

2776

2777

They are very cheap (20 Euros) and are a good start for a low-level

2778

benchmark network.

2779

2778

benchmark network.

2779

2780

First result show us:

2781

2782

48 Mb/s with ab.

2783

2784

The webserver and the test-server are equipped with a 2000+ AMD CPU. The

2785

system load 95%, user load is 3%, the rest is idle on both plattforms.

2786

2787

After some calculations at gets clear that there are various bottlenecks:

2788

2789

1. The PCI-Bus (32bit/33Mhz) can only transfer 133Mb/s

2788

2789

1. The PCI-Bus (32bit/33Mhz) can only transfer 133Mb/s

2790

- 48 Mb/s TCP-Traffic results in 55Mb/s Ethernet-Traffic (verified with

2791

slurm) [outgoing]

2792

- there is about 10Mb/s incomming traffic at the PCI bus which the

2793

requests

2794

- the rest of the devices at the PCI bus are eating the last few mb/s

2794

- the rest of the devices at the PCI bus are eating the last few mb/s

2795

2. IRQ-Handling

2796

- There are reasons why the RTL8169S cards are so cheap

2797

- they can't send jumbo-frames

2798

- only 8k/64k buffers which results in a interrupt every 3 packets

2798

- only 8k/64k buffers which results in a interrupt every 3 packets

2799

- they are at least handling checksum offloading for ip, udp and tcp

2800

2801

In the end there is a need for new hardware to limit the throughput by the

2802

Ethernet again.

2803

- PCI-X, 64bit-PCI, ...

2804

- 'real' gigE-Network cards

2805

- ...

2806

2807

2806

2807

2808

BTW: 2.4.22 + the r8139 driver are very flacky and resulted in 3 lock-ups

2809

for today.

2810

2809

for today.

2810

2811

In another test the dual-PPro-FreeBSD 5.1-CURRENT machine has been used as

2812

server with a gigE interface (re0) at has shown that the CPU is the limit

2813

for this combination. The maximum thoughput was 7Mb/s.

2814

2815

Another small benchmark:

2814

2815

Another small benchmark:

2816

$ ab -n 1000 -c 10 http://192.168.2.41:<port>/lighttpd-20030925.tar

2817

2818

port | server | CPU Idle

2817

2818

port | server | CPU Idle

2819

------+-------�---------+-----------

2820

1025 | lighttpd | 75%

2821

1026 | thttpd 2.23b1 | 75%

2822

1027 | boa-0.94.14rc17 | 69% *

2823

1028 | apache 1.3.x | 77%

2824

2825

2824

2825

2826

In all 4 cases the thoughput was 8600 kbytes/s.

2827

2828

* boa had 9 failed transfers.

2829

2830

- FreeBSD

2831

2829

2830

- FreeBSD

2831

2832

The problems with FreeBSD 5.0-RELEASE vanished after updating to

2833

5.1-CURRENT.

2834

2835

26.09.2003 18:22

2836

- FreeBSD

2837

2838

A SMP-machine has been added to the test-farm. It is running FreeBSD

2839

5.1-RELEASE and will help to the improve the scalability.

2840

2841

fixed sendfile() handling.

2842

2843

- FreeBSD problems

2844

2845

FreeBSD-5.1-RELEASE-SMP

2846

2 * Pentium Pro 200 MHz

2847

192.168.2.38 (doubleheart)

2848

(webserver [lighttpd at port 1025, thttpd at port 1027])

2849

2850

Linux 2.4.20

2851

1 * AMD 2000+

2852

192.168.2.10 (grisu)

2853

(ab)

2854

2855

Problem:

2856

Connections are reset by the kernel without any application intervention.

2857

2858

Calling 'ab' (apachebench) at grisu with the following paramters:

2859

2860

/usr/sbin/ab -n 10000 -c 10 http://192.168.2.38:1025/index.html

2861

2862

results in the following output:

2863

2864

...

2865

Time taken for tests: 40.610 seconds

2866

Complete requests: 10000

2867

Failed requests: 5980

2868

(Connect: 0, Length: 5980, Exceptions: 0)

2869

...

2870

2871

2872

This is reproducable and the number of failed requests is always

2870

2871

2872

This is reproducable and the number of failed requests is always

2873

5980 +/- 50 requests. In other words: after 4000 requests tcpdump shows

2874

the following output:

2875

2876

tcpdump shows:

2877

2878

00:53:48.923029 192.168.2.10.39774 > 192.168.2.38.1025: S [tcp sum ok]

2879

1013737315:1013737315(0) win 5840 <mss 1460,sackOK,timestamp 5208461

2880

0,nop,wscale0> (DF) (ttl 64, id 7918, len 60)

2892

0x0030 0101 080a 0044 0d12 004f 798d .....D...Oy.

2893

2894

00:53:48.924009 192.168.2.10.39774 > 192.168.2.38.1025: . [tcp sum ok] ack 1

2895

win 5840 <nop,nop,timestamp 5208461 4459794> (DF)

2895

win 5840 <nop,nop,timestamp 5208461 4459794> (DF)

2896

(ttl 64, id 7919, len 52)

2897

0x0000 4500 0034 1eef 4000 4006 9654 c0a8 020a E..4..@.@..T....

2898

0x0010 c0a8 0226 9b5e 0401 3c6c 6764 6b84 bac1 ...&.^..<lgdk...

2900

0x0030 0044 0d12

2901

2902

00:53:48.924150 192.168.2.10.39774 > 192.168.2.38.1025: P [tcp sum ok]

2903

1:29(28) ack 1 win 5840 <nop,nop,timestamp 5208461 4459794> (DF)

2903

1:29(28) ack 1 win 5840 <nop,nop,timestamp 5208461 4459794> (DF)

2904

(ttl 64, id 7920, len 80)

2905

0x0000 4500 0050 1ef0 4000 4006 9637 c0a8 020a E..P..@.@..7....

2906

0x0010 c0a8 0226 9b5e 0401 3c6c 6764 6b84 bac1 ...&.^..<lgdk...

2918

2919

strace shows that no connection attempt has been reported to the

2920

application which is poll()'ing the server socket.

2921

2922

/* the common loop without any block attempts */

2923

2924

accept(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, [0]) = 5

2925

fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0

2926

read(5, "GET /index.html HTTP/1.0\r\n\r\n", 4095) = 28

2930

write(2, "1064875136: (network.c.210) 235 "..., 33) = 33

2931

syscall_393(0x6, 0x5, 0, 0, 0x10fc, 0, 0xbfbff2c0, 0) = 0

2932

close(5) = 0

2933

2934

/* no futher waiting connections */

2935

2936

accept(3, 0xbfbff700, [1852702730]) = -1 EAGAIN (Resource temporarily unavailable)

2937

2938

/* enter the main-loop */

2939

2940

gettimeofday({1769235301, 1663069807}, NULL) = 0

2941

poll([{fd=3, events=POLLIN}], 1, 1000) = 0

2942

gettimeofday({4294967295, 65537}, NULL) = 0

2944

gettimeofday({4294967295, 65537}, NULL) = 0

2945

poll([{fd=3, events=POLLIN}], 1, 1000) = 0

2946

...

2947

2948

(strace is broken for accept() and gettimeofday(), syscall_393() is sendfile())

2949

2950

after entering the main-loop the webserver doesn't receive any new POLLIN

2951

events at all for the next 10-15 seconds. Any connection-attempt within

2952

the period is, as you can see in the tcpdump output, accepted by the

2954

After those 10-15 seconds the application gets a POLLIN event for the

2955

server socket and the normal data-transfer taken place for the next 4000

2956

requests.

2957

2958

This behaviour is reproducable with thttpd 2.20c and the current lighttpd.

2959

2960

2961

26.09.2003 11:42

2962

- localizer

2963

2964

ported the localizer-server application into a module for lighttpd. After

2965

30 minutes the code was ported and funtional. This modules provides access

2966

to the localizer-db by a HTTP-Interface and generates the response as

2964

ported the localizer-server application into a module for lighttpd. After

2965

30 minutes the code was ported and funtional. This modules provides access

2966

to the localizer-db by a HTTP-Interface and generates the response as

2967

HTML, CSV and plain-text.

2968

2969

So far, the module-interface looks good and flexible enough.

2970

2971

7000 req/s is good enough too. Generating the HTML and querying the DB

2972

needs some time.

2973

2974

22.09.2003 08:40

2975

- modules

2976

2979

auth-sub-system and the cache out of the main-code. The module-interface

2980

will be extended to allow the cgi and the fastcgi sub-processes to be

2981

moved into a module.

2982

2983

15.09.2003 09:36

2984

- error-handling

2985

2986

fastcgi and cgi connections are now closed correctly if the corresponding

2987

client-connection has died.

2988

2989

14.09.2003 10:40

2990

- cgi

2991

2992

finally streaming works with CGI, too

2993

2994

this enable support for cgiirc.sf.net and friends which use streaming as

2995

there transport mechanism.

2996

2997

streaming has been verified with cgiirc-0.5.2.

2998

2999

- fdevents

3000

3001

there where some reports that sigio didn't work as expected. It just

3002

reported no events at all. Looks like a known bug in the glibc on

3002

reported no events at all. Looks like a known bug in the glibc on

3003

those platforms.

3004

3005

Using poll() instead solve the problems.

3006

3007

27.08.2003 22:12

3008

- rewrite

3009

3010

a pcre based rewrite engine has been integrated

3011

3012

- cgi

3013

3014

the file-based cgi interface has been replace by two pipes.

3016

23.07.2003 13:29

3017

- fdevents

3018

3019

the whole fd-event handling has been reworked and several smaller bugs

3019

the whole fd-event handling has been reworked and several smaller bugs

3020

and design-errors have fixed.

3021

3022

sigio, poll() and select() are working fine again.

3023

3024

On IRIX 6.5 SIGIO could be used, but without further testing poll() is used.

3025

3026

- cgi

3027

CGI-support is still broken.

3028

3031

3032

using the F_NOTIFY feautre of the Linux 2.4.x kernel gives anothre nice

3033

performance boost as lighttpd can cache the stat()'s the right way, now.

3034

3035

23009.66 @ 5-4k-k which means +10% against 30.06.2003 14:03

3036

3037

for 1000-4k-nok the performace nearly doubles: 3730.23 goes up to 6144.39

3038

3039

17.07.2003 13:21

3040

- FreeBSD

3041

3042

a small patch (#include <>) to compile lighttpd on FreeBSD again.

3043

3044

08.07.2003 10:48

3045

- fileinfo-cache

3046

3052

3053

After some problems with other benchmarking tools for webservers the first

3054

version of lighttpd-bench has been written.

3055

3056

It a revealed a nasty strange behaviour which was fixed by increasing the

3057

listen-backlog from 5 to 1024.

3058

3059

30.06.2003 14:03

3060

- fileinfo-cache

3061

3062

the fileinfo-cache has been relaxed a little bit and there are always 2

3063

fstat()'s per file-request now. This isn't that dramatic:

3064

3065

21800.74 req/s is still a very good result (7-4k-k).

3066

3067

29.06.2003 03:29

3068

- fileinfo-cache

3069

3070

a fileinfo cache has been added to reduce the number of system-calls to

3071

stat and open a file.

3072

3071

stat and open a file.

3072

3073

in the releases before the same file was stat'ed at least 2 times plus a

3074

stat on the docroot for each request. Now the stat()'s and the

3075

corresponding open() + close() calls are cached and the number of system

3076

calls has been reduced to the minimum:

3077

3078

$ strace -eopen,stat64,read,write,sendfile,accept,shutdown,close \

3079

-p `pidof lighttpd`

3080

3081

/* first connection */

3082

accept(3, {sin_family=AF_INET6, sin6_port=htons(56211),

3083

inet_pton(AF_INET6, "::ffff:192.168.2.10", &sin6_addr), sin6_flowinfo=0,

3082

accept(3, {sin_family=AF_INET6, sin6_port=htons(56211),

3083

inet_pton(AF_INET6, "::ffff:192.168.2.10", &sin6_addr), sin6_flowinfo=0,

3084

sin6_scope_id=0}, [28]) = 5

3085

accept(3, 0xbffff470, [28]) = -1 EAGAIN

3086

read(5, "GET /index.html HTTP/1.0\r\nUser-A"..., 4095) = 91

3108

3109

3110

In the end we have a new speed record:

3111

3112

(ab -c 8 -n 100000 http://192.168.2.10:1025/index.html with poll())

3113

Requests per second: 23435.67 [#/sec] (mean)

3114

3115

This is a speed increasement of 20-30% against the last internal benchmark.

3116

3117

3118

28.06.2003 02:30

3119

- caching framework

3119

- caching framework

3120

3121

the trigger.if directive is working fine. the basic algebra is done ( +,

3122

-, *, / and comperations =, <=, >=, >, < and the boolean logic

3123

&& and || including braces)

3124

3125

two basic function are available:

3126

- unix.time.now

3127

- file.mtime(...)

3130

- caching framework

3131

3132

added output.include, output.content-type for a cache-hit and

3133

trigger.handler for a cache-miss.

3134

3133

trigger.handler for a cache-miss.

3134

3135

the actual decision is made be trigger.if which will be added tomorrow.

3136

3137

15.06.2003 15:33

3138

- sig-io

3138

- sig-io

3139

3140

After several other smaller optimisations lighttpd performs better with

3141

sig-io under high load:

3142

3143

sigio+poll poll

3144

c1000-4k-nok : 2635.95 1643.39

3145

c1000-4k-k : 7335.68 6788.87

3146

c1000-100k-nok: 2353.49 1217.73

3147

c1000-100k-k : 3097.89 2080.73

3148

3149

3148

3149

3150

The user-space part has been optimized a lot. Now it is time to optimized

3151

the number of context-switches between user and kernel-mode.

3152

3153

user 0m0.180s

3154

sys 0m1.160s

3155

3156

a fileinfo-cache is the way to go.

3157

3158

struct {

3159

buffer *name;

3160

struct stat st;

3161

int handler;

3162

...

3163

3164

int fd;

3165

void *mmap_p;

3166

}

3169

- sig-io is back again

3170

3171

and it works fine.

3172

3173

sigio (Realtime-Signals under Linux 2.4.x) sends one signal per event and

3174

buffers the rest of the events in a kernel queue. If the is full a SIGIO

3175

is sent and poll() gets all events at once for further processing.

3176

3177

3176

3177

3178

Currently the behaviour is a little bit strange:

3179

- sig-io + poll() is good for non-keep-alive connections

3180

- poll() for keep-alive connections

3181

3182

c1000-4k-nok: (1000 concurrent request, 4k filesize, no keepalive)

3183

poll : 1521.38

3184

sigio+poll: 2124.00

3185

3186

c1000-4k-k: (1000 concurrent request, 4k filesize, keepalive)

3187

poll : 5882.35

3188

sigio+poll: 1239.46

3189

3190

Very strange for now.

3191

3189

3190

Very strange for now.

3191

3192

09.06.2003 23:59

3193

- code-cleanup

3194

3195

the event-handling code has been rewritten to handle single events better

3196

as they are expected from sig-io.

3197

3198

the fallback-mode of sig-io is broken now, but the normal poll() mode got

3199

a 10% increasement in speed. This means the we are back the speed level of

3200

20030308-0155 are as fast as zeus again.

3201

3199

a 10% increasement in speed. This means the we are back the speed level of

3200

20030308-0155 are as fast as zeus again.

3201

3202

Especially under higher load the current lighttpd performes better.

3203

3204

09.06.2003 11:51

3205

- sig-io benchmark

3206

3207

20030609-1151 20030608-2110 20030308-0155

3207

20030609-1151 20030608-2110 20030308-0155

3208

-c 10 sigio select() poll() poll()

3209

4k : 7870.92 7937.77 8035.36 9443.76

3210

4k (keep-alive) : 14098.41 14590.02 14275.52 17985.61

3217

100k (keep-alive) : 4267.67 4283.94 4094.17 4314.06

3218

3219

3220

For -c 100 4k and 100k sig-io gives a small increasment.

3220

For -c 100 4k and 100k sig-io gives a small increasment.

3221

3222

09.06.2003 01:00

3222

09.06.2003 01:00

3223

- sig-io

3224

3225

writing large files works now as expected. After removing the limit for

3226

the chunks of sendfile the write-buffer-is-empty-again signal is

3227

generated. that was missing.

3228

3229

321 of 10000 connections still have the wrong length. After that is fixed

3230

it is time to start some benchmarks again.

3231

3233

- sig-io

3234

3235

first attempts in getting SIGIO support running which were not very

3236

successfull yet.

3237

3236

successfull yet.

3237

3238

- poll()/select() benchmark

3239

3240

20030608-2110 20030525-1623 20030308-0155

3240

20030608-2110 20030525-1623 20030308-0155

3241

-c 10 select() poll() poll() poll()

3242

4k : 7937.77 8035.36 8166.60 9443.76

3243

4k (keep-alive) : 14590.02 14275.52 14781.97 17985.61

3252

3253

as the overall processing cycle has been rearranged the overall figures

3254

changed in minor ranges. c100-4k-k increased, *-100k-k decreased.

3255

3256

At least it still works.

3257

3258

08.06.2003 12:45

3260

3261

implementate an abstration layer for fd-events (like eevry other webserver).

3262

currently we support poll() and select().

3263

3264

This should bring us support for Mac OS X and propably Windows.

3265

3266

04.06.2003 18:35

3275

3276

04.06.2003 14:34

3277

- pipelining

3278

3279

adding support for pipeling introduces a problem if the request-header

3280

was broken. this has been fixed now.

3281

3282

- unneccesary slashes

3283

3284

when building pathes from different portions of a string (e.g. docroot +

3291

3292

the error-log contains the start and end-times of the lighttpd process,

3293

now.

3294

3295

- configfile

3296

3297

the config-file parser has been relaxed to accepts tabs instead of spaces.

3298

3299

- better error-handling

3300

3301

fixed a <a

3301

fixed a <a

3302

href="http://bugs.php.net/?id=24009">bug in the FastCGI-SAPI of PHP</a>

3303

3304

if the fastcgi process dies or closes the connection unexpectedly we

3305

return 500 now instead of closing the connection to the client.

3306

3307

- Location

3308

3309

the CGI/1.1-rev-03 specification requires us to send Status 302 if a

3310

Location-header is sent by the client and a Status-header is missing.

3311

3312

- PATH_TRANSLATED

3313

3314

if PHP is compiled without --discard-path PATH_TRANSLATED has to be

3318

- directory listings

3319

3320

if a directory is requested and the directory doesn't contain a index-file

3321

a directory-listing can be displayed. You have to enable directory-listings

3321

a directory-listing can be displayed. You have to enable directory-listings

3322

in the config-file (directory-listings on)

3323

3324

- url-decoding

3330

- pipelining

3331

3332

as Sascha required pipelining for his benchmarking tool it has been

3333

implemented.

3334

3333

implemented.

3334

3335

pipelining allows sending a bunch of requests at once without waiting for

3336

the actual responses. This reduces the network-overhead and the

3337

round-trip-time.

3338

3339

- non-free()d memory

3340

3341

dmalloc helped to close some non-free()d memory. For the normal operation

3342

this isn't important as only memory chunks which had to be free()d at the

3343

end of the live-time of the lighttpd-process were not de-allocated.

3344

3345

- partly-initialized variables

3346

3347

lighttpd wasn't initializing the main-structure which resulted in strange

3348

behaviour in rare circumstances.

3349

3350

3351

25.05.2003 16:23

3352

- benchmarks

3353

3354

after removing some useless internal copies we are more or less at the old

3355

speed levels.

3356

3357

after adding virtual-hosts 10% of the performance were lost. Using less

3358

memcpy() operations might add several other boosts.

3359

3360

I've just checked how lighttpd compares to Zeus.

3361

3362

lighttpd (current) (old)

3363

-c 10 20030525-1623 20030308-0155 Zeus 4_2

3364

4k : 8166.60 9443.76 7278.55

3373

3374

23.05.2003 14:38

3375

- cgi-variables

3376

3377

HTTP_HOST was missing for the cgi-module while the fcgi-module passed it

3378

through to the handler. Fixed that.

3379

3380

- fcgi-errors

3381

3382

the connection to the fcgi was dropped and poll() reported an error, the

3383

error wasn't reported to the client the right way.

3384

3383

error wasn't reported to the client the right way.

3384

3385

22.05.2003 23:02

3386

- authorization

3387

3388

the first password-storage has been added:

3389

3390

[auth]

3391

backend plan

3392

plain-userfile <filename>

3393

3394

require /download/ user=jan|user=anom

3395

http-auth /download/ "download archiv" digest

3396

3397

groups are prepared but not implemented. basic and digest are working fine.

3398

3399

20.05.2003 17:53

3400

- authentification

3401

3402

The auth-methods from RFC 2617 have been added.

3401

3402

The auth-methods from RFC 2617 have been added.

3403

- auth basic

3404

- auth digest

3405

3406

The only source for accounts is currently only the config-file.

3407

3408

auth-digest needs the plain-text passwort. Are there any source which

3409

provide a plain-text password ?

3410

3411

12.05.2003 14:33

3412

- virtual hosts

3413

3414

added very basic virtual-host support

3415

3416

virtual-server-root /home/weigon/projects/lighttpd/servers/

3417

virtual-server-default-host grisu.home.kneschke.de:1025

3418

virtual-server-docroot /pages/

3419

3420

docroot is

3421

3419

3420

docroot is

3421

3422

- if http-host exists

3423

<virtual-server-root> + <http-host> + <virtual-server-docroot>

3424

3425

- otherwise

3426

<virtual-server-root> + <virtual-server-default-host> +

3426

<virtual-server-root> + <virtual-server-default-host> +

3427

<virtual-server-docroot>

3428

3429

- if even virtual-server-default-host does not exist, 500 is sent

3430

3431

3432

12.05.2003 13:02

3433

- code cleanup

3435

After two month of development it was time clean-up the internal

3436

structures. It looks like every went fine as lighttpd works es expected

3437

like before.

3438

3439

- deflate

3440

the on-the-fly compression has been verified to work fine with opera,

3440

the on-the-fly compression has been verified to work fine with opera,

3441

konqui, mozilla and the IE.

3442

3443

12.05.2003 02:10

3444

- on-the-fly compression: deflate

3445

3444

- on-the-fly compression: deflate

3445

3446

Why the hell are the defining a "deflate" encoding in the form of

3447

3448

_deflate_ The "zlib" format defined in RFC 1950 [31] in combination

3449

with the "deflate" compression mechanism described in RFC 1951 [29].

3447

3448

_deflate_ The "zlib" format defined in RFC 1950 [31] in combination

3449

with the "deflate" compression mechanism described in RFC 1951 [29].

3450

(RFC 2616)

3451

3452

and noone implements it that way ? Konqui and Mozilla expect a plain

3453

deflate() package without the zlib-header.

3454

3453

deflate() package without the zlib-header.

3454

3455

Konqui is using "inflate2(..., -MAX_WBITS); " which is noted in the zlib

3456

source as

3457

3456

source as

3457

3458

/* handle undocumented nowrap option (no zlib header or check) */

3459

3460

Funny. Very, very funny.

3461

3462

3461

3462

3463

Anyway. We have mimetype-depended compression support now.

3464

3465

11.05.2003 21:56

3466

- logging

3467

re-arranged the logfile structure to write CLF + useragent + referrer.

3469

11.05.2003 10:23

3470

- POST file-upload

3471

added the missing functionality to send more than a single packet to the

3472

FCGI-Server (or Client ? ... the PHP).

3473

3472

FCGI-Server (or Client ? ... the PHP).

3473

3474

This gives us the file-upload thing for eg. PHP and large user-forms

3475

(>16kb).

3476

3477

05.05.2003 15:21

3478

- PATH_INFO

3479

3480

added support for PATH_INFO. PHP is a little strange and doesn't trust the

3481

passed PATH_INFO setting. Works fine now.

3482

3483

30.04.2003 15:25

3484

- bug-fixing day

3485

3486

While testing the FastCGI interface with the MSIE Bj�rn Schotte discovered

3487

that sometimes the output repeats itself from the start in an endless loop.

3487

that sometimes the output repeats itself from the start in an endless loop.

3488

This bug has been fixed.

3489

3490

The read-write-fallback for ancient systems seeked the wrong FDs.

3491

3492

The FastCGI handler is now separting the HTTP-Header from the

3493

response-body what results in a cleaner interface. The "header too long

3494

for caching" message is gone now.

3495

3496

28.04.2003 18:18

3497

- chunked transfer-encoding

3498

3502

27.04.2003 23:10

3503

- chunked transfer-encoding

3504

added "Transfer-Encoding: chunked" which is currently used in the web-chat

3505

for the endless stream. Perhaps it helps some browsers.

3506

3505

for the endless stream. Perhaps it helps some browsers.

3506

3507

The FastCGI interface will get a the chunked-support too, as it will

3508

enable keep-alive even if no content-length is returned from the FastCGI

3509

process. We know the size of the chunks and will report it to the browser.

3512

- gigE deatchmatch

3513

3514

sascha compared the current lighttpd to his premium thttpd in his

3515

gigabit-Ethernet-Network.

3515

gigabit-Ethernet-Network.

3516

3517

<a

3518

href="http://schumann.cx/gbit_deathmatch.txt">http://schumann.cx/gbit_deathmatch.txt</a>

3519

3520

small comment:

3521

lighttpd provides the same performance (req/s and thoughput), but uses more

3522

CPU-Time.

3522

CPU-Time.

3523

3524

10.04.2003 17:22

3525

- works on IRIX

3527

J�rg Behrens provided me login to his SGI Origin and after fixing two small

3528

typos it worked fine on IRIX. (#if define HAVE... was missing a 'd' and

3529

getopt returns a 'int' and not a 'char')

3530

3531

IPv6 support is currently disabled for IRIX a gethostbyname2() isn't

3532

available.

3533

3535

- another bug-fixing day

3536

The upper limit of open connections was enforced and a caused a seg-fault.

3537

The current limit is set to 4096 parallel connections.

3538

3539

As sascha benchmarked lighttpd and his premium thttpd I tried to reproduce

3540

his reported results and had to fix 2 flaws in the async-io handling of

3541

httpd_load which was used for the testing.

3542

3543

If sascha starts another benchmark session I'll put a link to it here.

3544

3545

09.04.2003 00:08

3547

Date: and Last-Modified: where sending a timezone != GMT what was invalid.

3548

The timestamp itself was correct, just the timezone use the wrong

3549

characters.

3550

3551

the fcgi-code had an buffer-overflow for larger POST-Requests.

3552

3553

The fcgi-code still has problems with POST-Request larger than 16kb as the

3554

the write buffer is full. Currently we don't handle this case except from

3555

reporting it in the error-log.

3560

The CGI Interface wasn't tested for a longer time. It don't really

3561

survived the introduction of the config-file handling and the internal

3562

changes that were part of it.

3563

3564

A small test with

3565

3566

#! /usr/bin/perl

3567

print "Content-Type: text/html\r\n\r\n";

3568

print time()."\n";

3569

0;

3570

3571

produced 219 req/s.

3572

3571

produced 219 req/s.

3572

3573

Calling PHP via the CGI interface results in 100 req/s. Same script, same

3574

parameters for 'ab' and same PHP result in 1400 req/s if we use the

3575

FastCGI interface.

3576

3575

FastCGI interface.

3576

3577

Don't forget: these benchmarks only represent figures for scenarios which

3578

are not very realistic. They are just usefull for comparisions of the

3579

internals. We don't want to benchmark applications.

3580

3581

01.04.2003 23:04

3582

- new benchmarks

3583

3584

This time we wanted to see if we can get the fastcgi-Interface a little bit

3585

faster. We use this small script for the testing the interface.

3585

faster. We use this small script for the testing the interface.

3586

3587

<?php

3588

3603

3604

01.04.2003 19:28 (no april fools joke)

3605

- added SSL support

3606

I took -lssl and -lcrypto and added SSLv2/SSLv3/TLS support to lighttpd.

3607

3606

I took -lssl and -lcrypto and added SSLv2/SSLv3/TLS support to lighttpd.

3607

3608

After some reading I realisized that libcrypto contained some of my code.

3609

They have a similar buffer-struct with similar functions and they use the

3610

same MD5-code from the RFC. :)

3612

Adding basic SSL-support was quite esay: replacing all write/read-calls by

3613

SSL_write/read, initializing the ssl-context the right way and telling

3614

openssl where to get the data from (SSL_set_fd). It fits really well into

3615

lighttpd.

3616

3615

lighttpd.

3616

3617

- sidenote

3618

sooner or later the con->filename handler will vanish in favour of the

3619

direct interface to the chunk-API which will simplifiy the design a little

3621

3622

31.03.2003 20:50

3623

- added a web-chat module

3624

1400 lines of C-code are neccesary to add a web-chat to lighttpd.

3625

3624

1400 lines of C-code are neccesary to add a web-chat to lighttpd.

3625

3626

Features:

3627

- bb-code for text-layouting

3628

- auto-highlight for URLs and Email-adresses

3629

- unlimited number of channels

3629

- unlimited number of channels

3630

- max. 32.000 users per channel

3631

- fast as it is directly integrated into the web-server

3632

- easy to install (as easy as lighttpd)

3635

- uses as little JavaScript as possible to stay compatible with all browsers

3636

- flood-protection (2 Levels: ignore and kick)

3637

- uses CSS for customized look-and-feel

3638

3639

25.03.2003 00:36

3640

- finished the config-file support

3641

now we have config-sections

3642

3641

now we have config-sections

3642

3643

[fastcgi]

3644

fastcgi .php 192.168.2.76 1025

3645

3646

The only this missing is -HUP handling to re-read the config. But that

3647

will follow when I'm up again.

3648

3653

3654

19.03.2003 03:05

3655

- more on sendfile support

3656

added support for freebsd's version of sendfile().

3657

3656

added support for freebsd's version of sendfile().

3657

3658

Linux:

3659

ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

3660

3661

FreeBSD:

3662

int sendfile(int fd, int s, off_t offset, size_t nbytes,

3663

struct sf_hdtr *hdtr, off_t *sbytes, int flags);

3664

3665

the first 2 params are swap, the next 2 are the same and the last 3 are

3666

set to NULL or 0.

3667

3668

Solaris 9 is providing a direct interface to the chunk-API with the

3669

sendfilev() system call:

3670

3671

ssize_t sendfilev(int fildes,

3670

3671

ssize_t sendfilev(int fildes,

3672

const struct sendfilevec *vec, int sfvcnt, size_t *xferred);

3673

3674

recovering from a failure is a little bit more complex, but it should be

3675

fast. I just need a system to test it. Anyone ?

3676

3677

18.03.2003 17:32

3678

- in the news

3679

what a surprise: lighttpd is announced in entwickler.com.

3680

3679

what a surprise: lighttpd is announced in entwickler.com.

3680

3681

<a href="http://www.entwickler.com/itr/news/psecom,id,9483,nodeid,82.html">http://www.entwickler.com/itr/news/psecom,id,9483,nodeid,82.html</a>

3682

3683

this was not planned.

3684

3685

3684

3685

3686

- implemented the chunked-API

3687

rather writing the content of multi-range'd requests to a temp-file we

3688

just register the parts in a chunk-queue.

3689

3688

just register the parts in a chunk-queue.

3689

3690

chunk 1 [mem] - part-header

3691

chunk 2 [file] - source-file (offset, len)

3692

chunk 3 [mem] - part-header

3693

chunk 4 [file] - source-file (offset, len)

3694

chunk 5 [mem] - part-footer

3695

3696

If everything is prepared, the content-length is calculated and the

3697

http-header is generated and prepended to the chunk-queue.

3698

3699

the handle_write() function which handles the write-process just sends the

3700

chunks to the network without only further modifications.

3701

3702

the chunk-api even simplifies the

3703

'my-fastcgi-process-needs-some-time-to-create-the-response-

3704

and-sends-the-data-in-chunks' case. Just add the chunk to the queue if it

3706

3707

13.03.2003 10:49

3708

- another set of real-life benchmarks

3709

3710

the chairman was benchmarked a little bit. Chairman is full-flegded WCMS.

3711

As every CMS it needs some time to generate pages. To speed the page

3712

generating the static-parts of the page can be cached internally.

3713

3714

without caching with caching

3715

apache + mod_php : 2.50 10.64 req/s

3716

lighttpd + fcgi-php: 2.50 22.74 req/s

3725

- another large application works with lighttpd

3726

3727

<a

3728

href="http://www.thinkphp.de/content/content2.php?CatID=44&NewsID=95">Chairman</a> from <a href="http://www.thinkphp.de/">thinkphp</a> has been verified to work with lighttpd.

3729

3728

href="http://www.thinkphp.de/content/content2.php?CatID=44&NewsID=95">Chairman</a> from <a href="http://www.thinkphp.de/">thinkphp</a> has been verified to work with lighttpd.

3729

3730

Three things are worth to note:

3731

3732

1. ALWAYS use log_error in the php.ini if you use FastCGI

3731

3732

1. ALWAYS use log_error in the php.ini if you use FastCGI

3733

otherwise it will put the errormessages directly into fastcgi-socket

3734

without any FastCGI Header. This will result in strange errors.

3735

(its a PHP bug)

3744

3745

11.03.2003 00:13

3746

- another target

3747

after adding some headers lighttpd compiles without any warnings

3747

after adding some headers lighttpd compiles without any warnings

3748

under cygwin on Windows.

3749

3750

A native windows port will take some time as I have to figure out the

3752

3753

10.03.2003 11:52

3754

- added Solaris 8 and 9

3755

lighttpd has been confirmed to compile and work on

3756

3757

- linux

3755

lighttpd has been confirmed to compile and work on

3756

3757

- linux

3758

- FreeBSD, NetBSD

3759

- Solaris 8, 9

3760

3761

Perhaps I can get lcc to compile it on windows.

3762

3763

10.03.2003 01:30

3764

- another platform was confirmed

3765

lighttpd works on NetBSD

3766

3767

- another webserver benchmark

3768

mathopd (http://www.mathopd.org/)

3769

3770

Mathopd/1.4

3771

-c 10

3772

4k : 6329.11 [#/sec]

3776

-c 100

3777

4k : ---- (connections dropped)

3778

4k (keep-alive) : ----

3779

100k : ----

3779

100k : ----

3780

100k (keep-alive) : ----

3781

3782

Very good values for small files, but the rest ?

3786

1. sendfile (linux only)

3787

2. mmap + write

3788

3. read + write

3789

3790

This means that lighttpd runs on some more systems. It has been verified

3791

to compile and work on linux 2.4.x and FreeBSD.

3792

3793

3792

3793

3794

-c 100

3795

sendfile mmap read

3796

4k 6476.68 5698.01 5363.08 [#/sec]

3798

3799

The mmap() numbers could be better with a fd+mmap-cache. The zero-copy thing

3800

is good for our performance.

3801

3802

- tested another small httpd - BOA

3803

just to get a better view about the performance of lighttpd i've tested

3804

another httpd: Boa/0.94.14rc16 (http://www.boa.org/)

3805

3806

Boa/0.94.14rc16

3807

-c 10

3808

4k : 5659.63 [#/sec]

3824

09.03.2003 13:26

3825

- load-balancing works great

3826

3827

I used the framework from http://framework.netuse.de/ as a

3827

I used the framework from http://framework.netuse.de/ as a

3828

real-life example again to test the load-balancer.

3829

3830

setup:

3832

- AMD athlon XP 2000+

3833

- runs lighttpd + ab + a 'nice -15'ed FastCGI-PHP

3834

- generates 153 req/s alone

3835

3836

- laptop

3837

- Intel PIII 850

3838

- runs a FastCGI-PHP

3839

- generates 88 req/s

3840

3841

using the internal load-balancer of lighttpd which balances the

3840

3841

using the internal load-balancer of lighttpd which balances the

3842

php-requests over grisu and laptop generate:

3843

221 req/s

3844

3845

An apache with mod_php running on grisu gives 117.04 req/s only.

3846

3847

09.03.2003 12:46

3848

- building the connect connection to the fcgi-php is not non-blocking

3849

socket() -> connect() -> fcntl(non-blocking) resulted in some problems as

3850

the connect() call blocked sometimes for 1 seconds.

3851

3850

the connect() call blocked sometimes for 1 seconds.

3851

3852

socket() -> fcntl(non-blocking) -> connect() solves this problem, but

3853

addes more overhead. The first connect-attempt seems always to

3853

addes more overhead. The first connect-attempt seems always to

3854

return EINPROGRESS.

3855

3856

08.03.2003 15:06

3857

- the first 'real-life' PHP test showed 2 bugs in the POST handling.

3857

- the first 'real-life' PHP test showed 2 bugs in the POST handling.

3858

1. the Content-Type header was not forwarded via fastcgi

3859

2. the internal content_length handler wasn't reset after the request

3860

3861

now lighttpd + php work with

3862

3863

http://framework.netuse.de/

3864

3865

the basic application gives us 145 req/s

3864

3865

the basic application gives us 145 req/s

3866

3867

08.03.2003 11:06

3868

- use diet to 'test' your code

3869

diet gives some use full warnings which should be followed. I took the

3870

chance and removed the last few fprintf() and sprintf() from the code.

3871

now, we don't need stdio.h anymore.

3872

3873

a staticly linked, stripped lighttpd is 42884 bytes large.

3872

3873

a staticly linked, stripped lighttpd is 42884 bytes large.

3874

3875

08.03.2003 01:55

3876

- buffer_strcat() uses strlen() to get the length of the 'to-be-catted-string'

3877

if the string is constant or has a known length you can pass that function

3878

which gives use buffer_strcat_len() which passes the string length as a

3879

parameter.

3880

3881

- using a ramdisk as the base for the tempfiles generated mkstemp() is a

3882

very cood idea

3882

very cood idea

3883

it reduces the io-load on the system and gives as more cpu-time for

3884

load-balancing the php-requests.

3885

3886

- cache the output of strftime(), gmtime() and localtime

3887

we generate more then 10.000 req/s which results in using the same

3888

timestamp over and over again. It is sufficient to generate the timestamp

3893

3894

- don't try to overoptimize for code for the sake of clean code

3895

the range support require some hack in the first versions like setting the

3896

write_offset to the start of the range and keeping the rest of the

3897

"code 200" send code as usual. Only a few lines added this first range

3898

feature. fast, but a hack. It made things more complicated then neccesary

3896

write_offset to the start of the range and keeping the rest of the

3897

"code 200" send code as usual. Only a few lines added this first range

3898

feature. fast, but a hack. It made things more complicated then neccesary

3899

to add multi-range support in this scenario.

3900

3901

removing the hack doing small clean preprocessing helps to clean up the

3902

whole 'write a chunk on the wire' code. At the end we use less code and

3903

have a cleaner design.

3904

3905

Sascha proposed a chunked layer which just takes the chunks (http-header,

3903

have a cleaner design.

3904

3905

Sascha proposed a chunked layer which just takes the chunks (http-header,

3906

part-header, part-message, ... and the last boundary) and passes them to

3907

the write-to-the-wire-code. This is far more elegant and will perhaps same

3908

us from other problems, too. Currently I think that it is not necessary.

3909

First we have to fix some protability issues.

3910

3911

Ok, today benches:

3912

3913

today last

3925

Top Transfer Rate:

3926

3927

86365.77 [Kbytes/sec] received

3928

3929

Too much for a GigE-link ?

3930

3931

3932

so far:

3933

3934

- use state-engines

3935

3936

a non-forking webserver like described in

3937

3936

a non-forking webserver like described in

3937

3938

http://www.kegel.com/c10k.html

3939

3940

- don't use fprintf() for logging.

3940

- don't use fprintf() for logging.

3941

writing one line to log file takes more time that sending responsing the a

3942

HTTP-request

3943

3944

- take care of your memory

3945

3946

typedef struct {

3948

size_t used;

3949

size_t size;

3950

} buffer;

3951

3952

buffer* buffer_init();

3953

void buffer_free(buffer *b);

3954

int buffer_prepare_copy(buffer *b, size_t size);

3957

int buffer_strcpy_len(buffer *b, const char *s, int s_len);

3958

int buffer_strcat(buffer *b, const char *s);

3959

int buffer_strcat_len(buffer *b, const char *s, int s_len);

3960

3961

buffer_init() is only called once per buffer. If you don't need the

3962

buffer, set 'used' to zero and reused it afterward.

3962

buffer, set 'used' to zero and reused it afterward.

3963

buffer_strcpy() + buffer_strcat() check if the 'used' + the new strlen()

3964

fit in to the 'size' of the buffer. If not the realloc() the buffer.

3965

buffer_strcat() uses memcpy(ptr + used - 1, s, s_len + 1); which should

3966

be faster than the original strcat().

3967

3968

Using those buffers keep the memory usage at 2Mb. malloc() and free() are

3969

only called at the beginnig of the programm and at the end.

3970

3971

- don't used sprintf() to convert a integer into a string as like

3972

sprintf(buf, "%d", l);

3973

3974

write your own ltostr function. sprintf() is to general for the case and

3975

is slow.

3976

3980

3981

Let's assume that <a

3982

href="http://www.acme.com/software/thttpd/">thttpd</a> is (one of) the

3983

fastest webservers out there.

3983

fastest webservers out there.

3984

3985

all httpds were started with:

3986

3987

$ ./thttpd -p 1026 -D \

3988

-d /home/weigon/projects/localizer/src/lighttpd/docroot/ \

3989

-l thttpd.access.log

3990

3991

(lighttpd uses the same parameters).

3992

3993

We used ApacheBench for testing the RPS (requests per second).

4007

100k (keep-a.) : 4035.35 893.51[3] 970.21 955.05 [#/sec]

4008

4009

thttpd/2.21b+php has been patched with the keep-alive + php patches from PHP4

4010

sapi/thttpd/thttpd_patch.

4010

sapi/thttpd/thttpd_patch.

4011

4012

[1] ab reported:

4013

Failed requests: 1