~ubuntu-branches/ubuntu/dapper/wget/dapper-updates : revision 1

1

\input texinfo @c -*-texinfo-*-

2

3

@c %**start of header

4

@setfilename wget.info

5

@include version.texi

6

@set UPDATED May 2003

7

@settitle GNU Wget @value{VERSION} Manual

8

@c Disable the monstrous rectangles beside overfull hbox-es.

9

@finalout

10

@c Use `odd' to print double-sided.

11

@setchapternewpage on

12

@c %**end of header

13

14

@iftex

15

@c Remove this if you don't use A4 paper.

16

@afourpaper

17

@end iftex

18

19

@c Title for man page. The weird way texi2pod.pl is written requires

20

@c the preceding @set.

21

@set Wget Wget

22

@c man title Wget The non-interactive network downloader.

23

24

@dircategory Network Applications

25

@direntry

26

* Wget: (wget). The non-interactive network downloader.

27

@end direntry

28

29

@ifnottex

30

This file documents the the GNU Wget utility for downloading network

31

data.

32

33

@c man begin COPYRIGHT

34

Copyright @copyright{} 1996, 1997, 1998, 2000, 2001, 2002, 2003 Free

35

Software Foundation, Inc.

36

37

Permission is granted to make and distribute verbatim copies of

38

this manual provided the copyright notice and this permission notice

39

are preserved on all copies.

40

41

@ignore

42

Permission is granted to process this file through TeX and print the

43

results, provided the printed document carries a copying permission

44

notice identical to this one except for the removal of this paragraph

45

(this paragraph not being relevant to the printed manual).

46

@end ignore

47

Permission is granted to copy, distribute and/or modify this document

48

under the terms of the GNU Free Documentation License, Version 1.1 or

49

any later version published by the Free Software Foundation; with the

50

Invariant Sections being ``GNU General Public License'' and ``GNU Free

51

Documentation License'', with no Front-Cover Texts, and with no

52

Back-Cover Texts. A copy of the license is included in the section

53

entitled ``GNU Free Documentation License''.

54

@c man end

55

@end ifnottex

56

57

@titlepage

58

@title GNU Wget @value{VERSION}

59

@subtitle The non-interactive download utility

60

@subtitle Updated for Wget @value{VERSION}, @value{UPDATED}

61

@author by Hrvoje Nik@v{s}i@'{c} and the developers

62

63

@ignore

64

@c man begin AUTHOR

65

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.

66

@c man end

67

@c man begin SEEALSO

68

GNU Info entry for @file{wget}.

69

@c man end

70

@end ignore

71

72

@page

73

@vskip 0pt plus 1filll

74

Copyright @copyright{} 1996, 1997, 1998, 2000, 2001, 2003 Free Software

75

Foundation, Inc.

76

77

Permission is granted to copy, distribute and/or modify this document

78

under the terms of the GNU Free Documentation License, Version 1.1 or

79

any later version published by the Free Software Foundation; with the

80

Invariant Sections being ``GNU General Public License'' and ``GNU Free

81

Documentation License'', with no Front-Cover Texts, and with no

82

Back-Cover Texts. A copy of the license is included in the section

83

entitled ``GNU Free Documentation License''.

84

@end titlepage

85

86

@ifnottex

87

@node Top, Overview, (dir), (dir)

88

@top Wget @value{VERSION}

89

90

This manual documents version @value{VERSION} of GNU Wget, the freely

91

available utility for network downloads.

92

93

Copyright @copyright{} 1996, 1997, 1998, 2000, 2001, 2003 Free Software

94

Foundation, Inc.

95

96

@menu

97

* Overview:: Features of Wget.

98

* Invoking:: Wget command-line arguments.

99

* Recursive Retrieval:: Description of recursive retrieval.

100

* Following Links:: The available methods of chasing links.

101

* Time-Stamping:: Mirroring according to time-stamps.

102

* Startup File:: Wget's initialization file.

103

* Examples:: Examples of usage.

104

* Various:: The stuff that doesn't fit anywhere else.

105

* Appendices:: Some useful references.

106

* Copying:: You may give out copies of Wget and of this manual.

107

* Concept Index:: Topics covered by this manual.

108

@end menu

109

@end ifnottex

110

111

@node Overview, Invoking, Top, Top

112

@chapter Overview

113

@cindex overview

114

@cindex features

115

116

@c man begin DESCRIPTION

117

GNU Wget is a free utility for non-interactive download of files from

118

the Web. It supports @sc{http}, @sc{https}, and @sc{ftp} protocols, as

119

well as retrieval through @sc{http} proxies.

120

121

@c man end

122

This chapter is a partial overview of Wget's features.

123

124

@itemize @bullet

125

@item

126

@c man begin DESCRIPTION

127

Wget is non-interactive, meaning that it can work in the background,

128

while the user is not logged on. This allows you to start a retrieval

129

and disconnect from the system, letting Wget finish the work. By

130

contrast, most of the Web browsers require constant user's presence,

131

which can be a great hindrance when transferring a lot of data.

132

@c man end

133

134

@sp 1

135

@item

136

@ignore

137

@c man begin DESCRIPTION

138

139

@c man end

140

@end ignore

141

@c man begin DESCRIPTION

142

Wget can follow links in @sc{html} and @sc{xhtml} pages and create local

143

versions of remote web sites, fully recreating the directory structure of

144

the original site. This is sometimes referred to as ``recursive

145

downloading.'' While doing that, Wget respects the Robot Exclusion

146

Standard (@file{/robots.txt}). Wget can be instructed to convert the

147

links in downloaded @sc{html} files to the local files for offline

148

viewing.

149

@c man end

150

151

@sp 1

152

@item

153

File name wildcard matching and recursive mirroring of directories are

154

available when retrieving via @sc{ftp}. Wget can read the time-stamp

155

information given by both @sc{http} and @sc{ftp} servers, and store it

156

locally. Thus Wget can see if the remote file has changed since last

157

retrieval, and automatically retrieve the new version if it has. This

158

makes Wget suitable for mirroring of @sc{ftp} sites, as well as home

159

pages.

160

161

@sp 1

162

@item

163

@ignore

164

@c man begin DESCRIPTION

165

166

@c man end

167

@end ignore

168

@c man begin DESCRIPTION

169

Wget has been designed for robustness over slow or unstable network

170

connections; if a download fails due to a network problem, it will

171

keep retrying until the whole file has been retrieved. If the server

172

supports regetting, it will instruct the server to continue the

173

download from where it left off.

174

@c man end

175

176

@sp 1

177

@item

178

Wget supports proxy servers, which can lighten the network load, speed

179

up retrieval and provide access behind firewalls. However, if you are

180

behind a firewall that requires that you use a socks style gateway, you

181

can get the socks library and build Wget with support for socks. Wget

182

also supports the passive @sc{ftp} downloading as an option.

183

184

@sp 1

185

@item

186

Built-in features offer mechanisms to tune which links you wish to follow

187

(@pxref{Following Links}).

188

189

@sp 1

190

@item

191

The retrieval is conveniently traced with printing dots, each dot

192

representing a fixed amount of data received (1KB by default). These

193

representations can be customized to your preferences.

194

195

@sp 1

196

@item

197

Most of the features are fully configurable, either through command line

198

options, or via the initialization file @file{.wgetrc} (@pxref{Startup

199

File}). Wget allows you to define @dfn{global} startup files

200

(@file{/usr/local/etc/wgetrc} by default) for site settings.

201

202

@ignore

203

@c man begin FILES

204

@table @samp

205

@item /usr/local/etc/wgetrc

206

Default location of the @dfn{global} startup file.

207

208

@item .wgetrc

209

User startup file.

210

@end table

211

@c man end

212

@end ignore

213

214

@sp 1

215

@item

216

Finally, GNU Wget is free software. This means that everyone may use

217

it, redistribute it and/or modify it under the terms of the GNU General

218

Public License, as published by the Free Software Foundation

219

(@pxref{Copying}).

220

@end itemize

221

222

@node Invoking, Recursive Retrieval, Overview, Top

223

@chapter Invoking

224

@cindex invoking

225

@cindex command line

226

@cindex arguments

227

@cindex nohup

228

229

By default, Wget is very simple to invoke. The basic syntax is:

230

231

@example

232

@c man begin SYNOPSIS

233

wget [@var{option}]@dots{} [@var{URL}]@dots{}

234

@c man end

235

@end example

236

237

Wget will simply download all the @sc{url}s specified on the command

238

line. @var{URL} is a @dfn{Uniform Resource Locator}, as defined below.

239

240

However, you may wish to change some of the default parameters of

241

Wget. You can do it two ways: permanently, adding the appropriate

242

command to @file{.wgetrc} (@pxref{Startup File}), or specifying it on

243

the command line.

244

245

@menu

246

* URL Format::

247

* Option Syntax::

248

* Basic Startup Options::

249

* Logging and Input File Options::

250

* Download Options::

251

* Directory Options::

252

* HTTP Options::

253

* FTP Options::

254

* Recursive Retrieval Options::

255

* Recursive Accept/Reject Options::

256

@end menu

257

258

@node URL Format, Option Syntax, Invoking, Invoking

259

@section URL Format

260

@cindex URL

261

@cindex URL syntax

262

263

@dfn{URL} is an acronym for Uniform Resource Locator. A uniform

264

resource locator is a compact string representation for a resource

265

available via the Internet. Wget recognizes the @sc{url} syntax as per

266

@sc{rfc1738}. This is the most widely used form (square brackets denote

267

optional parts):

268

269

@example

270

http://host[:port]/directory/file

271

ftp://host[:port]/directory/file

272

@end example

273

274

You can also encode your username and password within a @sc{url}:

275

276

@example

277

ftp://user:password@@host/path

278

http://user:password@@host/path

279

@end example

280

281

Either @var{user} or @var{password}, or both, may be left out. If you

282

leave out either the @sc{http} username or password, no authentication

283

will be sent. If you leave out the @sc{ftp} username, @samp{anonymous}

284

will be used. If you leave out the @sc{ftp} password, your email

285

address will be supplied as a default password.@footnote{If you have a

286

@file{.netrc} file in your home directory, password will also be

287

searched for there.}

288

289

@strong{Important Note}: if you specify a password-containing @sc{url}

290

on the command line, the username and password will be plainly visible

291

to all users on the system, by way of @code{ps}. On multi-user systems,

292

this is a big security risk. To work around it, use @code{wget -i -}

293

and feed the @sc{url}s to Wget's standard input, each on a separate

294

line, terminated by @kbd{C-d}.

295

296

You can encode unsafe characters in a @sc{url} as @samp{%xy}, @code{xy}

297

being the hexadecimal representation of the character's @sc{ascii}

298

value. Some common unsafe characters include @samp{%} (quoted as

299

@samp{%25}), @samp{:} (quoted as @samp{%3A}), and @samp{@@} (quoted as

300

@samp{%40}). Refer to @sc{rfc1738} for a comprehensive list of unsafe

301

characters.

302

303

Wget also supports the @code{type} feature for @sc{ftp} @sc{url}s. By

304

default, @sc{ftp} documents are retrieved in the binary mode (type

305

@samp{i}), which means that they are downloaded unchanged. Another

306

useful mode is the @samp{a} (@dfn{ASCII}) mode, which converts the line

307

delimiters between the different operating systems, and is thus useful

308

for text files. Here is an example:

309

310

@example

311

ftp://host/directory/file;type=a

312

@end example

313

314

Two alternative variants of @sc{url} specification are also supported,

315

because of historical (hysterical?) reasons and their widespreaded use.

316

317

@sc{ftp}-only syntax (supported by @code{NcFTP}):

318

@example

319

host:/dir/file

320

@end example

321

322

@sc{http}-only syntax (introduced by @code{Netscape}):

323

@example

324

host[:port]/dir/file

325

@end example

326

327

These two alternative forms are deprecated, and may cease being

328

supported in the future.

329

330

If you do not understand the difference between these notations, or do

331

not know which one to use, just use the plain ordinary format you use

332

with your favorite browser, like @code{Lynx} or @code{Netscape}.

333

334

@node Option Syntax, Basic Startup Options, URL Format, Invoking

335

@section Option Syntax

336

@cindex option syntax

337

@cindex syntax of options

338

339

Since Wget uses GNU getopts to process its arguments, every option has a

340

short form and a long form. Long options are more convenient to

341

remember, but take time to type. You may freely mix different option

342

styles, or specify options after the command-line arguments. Thus you

343

may write:

344

345

@example

346

wget -r --tries=10 http://fly.srk.fer.hr/ -o log

347

@end example

348

349

The space between the option accepting an argument and the argument may

350

be omitted. Instead @samp{-o log} you can write @samp{-olog}.

351

352

You may put several options that do not require arguments together,

353

like:

354

355

@example

356

wget -drc @var{URL}

357

@end example

358

359

This is a complete equivalent of:

360

361

@example

362

wget -d -r -c @var{URL}

363

@end example

364

365

Since the options can be specified after the arguments, you may

366

terminate them with @samp{--}. So the following will try to download

367

@sc{url} @samp{-x}, reporting failure to @file{log}:

368

369

@example

370

wget -o log -- -x

371

@end example

372

373

The options that accept comma-separated lists all respect the convention

374

that specifying an empty list clears its value. This can be useful to

375

clear the @file{.wgetrc} settings. For instance, if your @file{.wgetrc}

376

sets @code{exclude_directories} to @file{/cgi-bin}, the following

377

example will first reset it, and then set it to exclude @file{/~nobody}

378

and @file{/~somebody}. You can also clear the lists in @file{.wgetrc}

379

(@pxref{Wgetrc Syntax}).

380

381

@example

382

wget -X '' -X /~nobody,/~somebody

383

@end example

384

385

@c man begin OPTIONS

386

387

@node Basic Startup Options, Logging and Input File Options, Option Syntax, Invoking

388

@section Basic Startup Options

389

390

@table @samp

391

@item -V

392

@itemx --version

393

Display the version of Wget.

394

395

@item -h

396

@itemx --help

397

Print a help message describing all of Wget's command-line options.

398

399

@item -b

400

@itemx --background

401

Go to background immediately after startup. If no output file is

402

specified via the @samp{-o}, output is redirected to @file{wget-log}.

403

404

@cindex execute wgetrc command

405

@item -e @var{command}

406

@itemx --execute @var{command}

407

Execute @var{command} as if it were a part of @file{.wgetrc}

408

(@pxref{Startup File}). A command thus invoked will be executed

409

@emph{after} the commands in @file{.wgetrc}, thus taking precedence over

410

them.

411

@end table

412

413

@node Logging and Input File Options, Download Options, Basic Startup Options, Invoking

414

@section Logging and Input File Options

415

416

@table @samp

417

@cindex output file

418

@cindex log file

419

@item -o @var{logfile}

420

@itemx --output-file=@var{logfile}

421

Log all messages to @var{logfile}. The messages are normally reported

422

to standard error.

423

424

@cindex append to log

425

@item -a @var{logfile}

426

@itemx --append-output=@var{logfile}

427

Append to @var{logfile}. This is the same as @samp{-o}, only it appends

428

to @var{logfile} instead of overwriting the old log file. If

429

@var{logfile} does not exist, a new file is created.

430

431

@cindex debug

432

@item -d

433

@itemx --debug

434

Turn on debug output, meaning various information important to the

435

developers of Wget if it does not work properly. Your system

436

administrator may have chosen to compile Wget without debug support, in

437

which case @samp{-d} will not work. Please note that compiling with

438

debug support is always safe---Wget compiled with the debug support will

439

@emph{not} print any debug info unless requested with @samp{-d}.

440

@xref{Reporting Bugs}, for more information on how to use @samp{-d} for

441

sending bug reports.

442

443

@cindex quiet

444

@item -q

445

@itemx --quiet

446

Turn off Wget's output.

447

448

@cindex verbose

449

@item -v

450

@itemx --verbose

451

Turn on verbose output, with all the available data. The default output

452

is verbose.

453

454

@item -nv

455

@itemx --non-verbose

456

Non-verbose output---turn off verbose without being completely quiet

457

(use @samp{-q} for that), which means that error messages and basic

458

information still get printed.

459

460

@cindex input-file

461

@item -i @var{file}

462

@itemx --input-file=@var{file}

463

Read @sc{url}s from @var{file}, in which case no @sc{url}s need to be on

464

the command line. If there are @sc{url}s both on the command line and

465

in an input file, those on the command lines will be the first ones to

466

be retrieved. The @var{file} need not be an @sc{html} document (but no

467

harm if it is)---it is enough if the @sc{url}s are just listed

468

sequentially.

469

470

However, if you specify @samp{--force-html}, the document will be

471

regarded as @samp{html}. In that case you may have problems with

472

relative links, which you can solve either by adding @code{<base

473

href="@var{url}">} to the documents or by specifying

474

@samp{--base=@var{url}} on the command line.

475

476

@cindex force html

477

@item -F

478

@itemx --force-html

479

When input is read from a file, force it to be treated as an @sc{html}

480

file. This enables you to retrieve relative links from existing

481

@sc{html} files on your local disk, by adding @code{<base

482

href="@var{url}">} to @sc{html}, or using the @samp{--base} command-line

483

option.

484

485

@cindex base for relative links in input file

486

@item -B @var{URL}

487

@itemx --base=@var{URL}

488

When used in conjunction with @samp{-F}, prepends @var{URL} to relative

489

links in the file specified by @samp{-i}.

490

@end table

491

492

@node Download Options, Directory Options, Logging and Input File Options, Invoking

493

@section Download Options

494

495

@table @samp

496

@cindex bind() address

497

@cindex client IP address

498

@cindex IP address, client

499

@item --bind-address=@var{ADDRESS}

500

When making client TCP/IP connections, @code{bind()} to @var{ADDRESS} on

501

the local machine. @var{ADDRESS} may be specified as a hostname or IP

502

address. This option can be useful if your machine is bound to multiple

503

IPs.

504

505

@cindex retries

506

@cindex tries

507

@cindex number of retries

508

@item -t @var{number}

509

@itemx --tries=@var{number}

510

Set number of retries to @var{number}. Specify 0 or @samp{inf} for

511

infinite retrying. The default is to retry 20 times, with the exception

512

of fatal errors like ``connection refused'' or ``not found'' (404),

513

which are not retried.

514

515

@item -O @var{file}

516

@itemx --output-document=@var{file}

517

The documents will not be written to the appropriate files, but all will

518

be concatenated together and written to @var{file}. If @var{file}

519

already exists, it will be overwritten. If the @var{file} is @samp{-},

520

the documents will be written to standard output. Including this option

521

automatically sets the number of tries to 1.

522

523

@cindex clobbering, file

524

@cindex downloading multiple times

525

@cindex no-clobber

526

@item -nc

527

@itemx --no-clobber

528

If a file is downloaded more than once in the same directory, Wget's

529

behavior depends on a few options, including @samp{-nc}. In certain

530

cases, the local file will be @dfn{clobbered}, or overwritten, upon

531

repeated download. In other cases it will be preserved.

532

533

When running Wget without @samp{-N}, @samp{-nc}, or @samp{-r},

534

downloading the same file in the same directory will result in the

535

original copy of @var{file} being preserved and the second copy being

536

named @samp{@var{file}.1}. If that file is downloaded yet again, the

537

third copy will be named @samp{@var{file}.2}, and so on. When

538

@samp{-nc} is specified, this behavior is suppressed, and Wget will

539

refuse to download newer copies of @samp{@var{file}}. Therefore,

540

``@code{no-clobber}'' is actually a misnomer in this mode---it's not

541

clobbering that's prevented (as the numeric suffixes were already

542

preventing clobbering), but rather the multiple version saving that's

543

prevented.

544

545

When running Wget with @samp{-r}, but without @samp{-N} or @samp{-nc},

546

re-downloading a file will result in the new copy simply overwriting the

547

old. Adding @samp{-nc} will prevent this behavior, instead causing the

548

original version to be preserved and any newer copies on the server to

549

be ignored.

550

551

When running Wget with @samp{-N}, with or without @samp{-r}, the

552

decision as to whether or not to download a newer copy of a file depends

553

on the local and remote timestamp and size of the file

554

(@pxref{Time-Stamping}). @samp{-nc} may not be specified at the same

555

time as @samp{-N}.

556

557

Note that when @samp{-nc} is specified, files with the suffixes

558

@samp{.html} or (yuck) @samp{.htm} will be loaded from the local disk

559

and parsed as if they had been retrieved from the Web.

560

561

@cindex continue retrieval

562

@cindex incomplete downloads

563

@cindex resume download

564

@item -c

565

@itemx --continue

566

Continue getting a partially-downloaded file. This is useful when you

567

want to finish up a download started by a previous instance of Wget, or

568

by another program. For instance:

569

570

@example

571

wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z

572

@end example

573

574

If there is a file named @file{ls-lR.Z} in the current directory, Wget

575

will assume that it is the first portion of the remote file, and will

576

ask the server to continue the retrieval from an offset equal to the

577

length of the local file.

578

579

Note that you don't need to specify this option if you just want the

580

current invocation of Wget to retry downloading a file should the

581

connection be lost midway through. This is the default behavior.

582

@samp{-c} only affects resumption of downloads started @emph{prior} to

583

this invocation of Wget, and whose local files are still sitting around.

584

585

Without @samp{-c}, the previous example would just download the remote

586

file to @file{ls-lR.Z.1}, leaving the truncated @file{ls-lR.Z} file

587

alone.

588

589

Beginning with Wget 1.7, if you use @samp{-c} on a non-empty file, and

590

it turns out that the server does not support continued downloading,

591

Wget will refuse to start the download from scratch, which would

592

effectively ruin existing contents. If you really want the download to

593

start from scratch, remove the file.

594

595

Also beginning with Wget 1.7, if you use @samp{-c} on a file which is of

596

equal size as the one on the server, Wget will refuse to download the

597

file and print an explanatory message. The same happens when the file

598

is smaller on the server than locally (presumably because it was changed

599

on the server since your last download attempt)---because ``continuing''

600

is not meaningful, no download occurs.

601

602

On the other side of the coin, while using @samp{-c}, any file that's

603

bigger on the server than locally will be considered an incomplete

604

download and only @code{(length(remote) - length(local))} bytes will be

605

downloaded and tacked onto the end of the local file. This behavior can

606

be desirable in certain cases---for instance, you can use @samp{wget -c}

607

to download just the new portion that's been appended to a data

608

collection or log file.

609

610

However, if the file is bigger on the server because it's been

611

@emph{changed}, as opposed to just @emph{appended} to, you'll end up

612

with a garbled file. Wget has no way of verifying that the local file

613

is really a valid prefix of the remote file. You need to be especially

614

careful of this when using @samp{-c} in conjunction with @samp{-r},

615

since every file will be considered as an "incomplete download" candidate.

616

617

Another instance where you'll get a garbled file if you try to use

618

@samp{-c} is if you have a lame @sc{http} proxy that inserts a

619

``transfer interrupted'' string into the local file. In the future a

620

``rollback'' option may be added to deal with this case.

621

622

Note that @samp{-c} only works with @sc{ftp} servers and with @sc{http}

623

servers that support the @code{Range} header.

624

625

@cindex progress indicator

626

@cindex dot style

627

@item --progress=@var{type}

628

Select the type of the progress indicator you wish to use. Legal

629

indicators are ``dot'' and ``bar''.

630

631

The ``bar'' indicator is used by default. It draws an @sc{ascii} progress

632

bar graphics (a.k.a ``thermometer'' display) indicating the status of

633

retrieval. If the output is not a TTY, the ``dot'' bar will be used by

634

default.

635

636

Use @samp{--progress=dot} to switch to the ``dot'' display. It traces

637

the retrieval by printing dots on the screen, each dot representing a

638

fixed amount of downloaded data.

639

640

When using the dotted retrieval, you may also set the @dfn{style} by

641

specifying the type as @samp{dot:@var{style}}. Different styles assign

642

different meaning to one dot. With the @code{default} style each dot

643

represents 1K, there are ten dots in a cluster and 50 dots in a line.

644

The @code{binary} style has a more ``computer''-like orientation---8K

645

dots, 16-dots clusters and 48 dots per line (which makes for 384K

646

lines). The @code{mega} style is suitable for downloading very large

647

files---each dot represents 64K retrieved, there are eight dots in a

648

cluster, and 48 dots on each line (so each line contains 3M).

649

650

Note that you can set the default style using the @code{progress}

651

command in @file{.wgetrc}. That setting may be overridden from the

652

command line. The exception is that, when the output is not a TTY, the

653

``dot'' progress will be favored over ``bar''. To force the bar output,

654

use @samp{--progress=bar:force}.

655

656

@item -N

657

@itemx --timestamping

658

Turn on time-stamping. @xref{Time-Stamping}, for details.

659

660

@cindex server response, print

661

@item -S

662

@itemx --server-response

663

Print the headers sent by @sc{http} servers and responses sent by

664

@sc{ftp} servers.

665

666

@cindex Wget as spider

667

@cindex spider

668

@item --spider

669

When invoked with this option, Wget will behave as a Web @dfn{spider},

670

which means that it will not download the pages, just check that they

671

are there. For example, you can use Wget to check your bookmarks:

672

673

@example

674

wget --spider --force-html -i bookmarks.html

675

@end example

676

677

This feature needs much more work for Wget to get close to the

678

functionality of real web spiders.

679

680

@cindex timeout

681

@item -T seconds

682

@itemx --timeout=@var{seconds}

683

Set the network timeout to @var{seconds} seconds. This is equivalent

684

to specifying @samp{--dns-timeout}, @samp{--connect-timeout}, and

685

@samp{--read-timeout}, all at the same time.

686

687

Whenever Wget connects to or reads from a remote host, it checks for a

688

timeout and aborts the operation if the time expires. This prevents

689

anomalous occurrences such as hanging reads or infinite connects. The

690

only timeout enabled by default is a 900-second timeout for reading.

691

Setting timeout to 0 disables checking for timeouts.

692

693

Unless you know what you are doing, it is best not to set any of the

694

timeout-related options.

695

696

@cindex DNS timeout

697

@cindex timeout, DNS

698

@item --dns-timeout=@var{seconds}

699

Set the DNS lookup timeout to @var{seconds} seconds. DNS lookups that

700

don't complete within the specified time will fail. By default, there

701

is no timeout on DNS lookups, other than that implemented by system

702

libraries.

703

704

@cindex connect timeout

705

@cindex timeout, connect

706

@item --connect-timeout=@var{seconds}

707

Set the connect timeout to @var{seconds} seconds. TCP connections that

708

take longer to establish will be aborted. By default, there is no

709

connect timeout, other than that implemented by system libraries.

710

711

@cindex read timeout

712

@cindex timeout, read

713

@item --read-timeout=@var{seconds}

714

Set the read (and write) timeout to @var{seconds} seconds. Reads that

715

take longer will fail. The default value for read timeout is 900

716

seconds.

717

718

@cindex bandwidth, limit

719

@cindex rate, limit

720

@cindex limit bandwidth

721

@item --limit-rate=@var{amount}

722

Limit the download speed to @var{amount} bytes per second. Amount may

723

be expressed in bytes, kilobytes with the @samp{k} suffix, or megabytes

724

with the @samp{m} suffix. For example, @samp{--limit-rate=20k} will

725

limit the retrieval rate to 20KB/s. This kind of thing is useful when,

726

for whatever reason, you don't want Wget to consume the entire available

727

bandwidth.

728

729

Note that Wget implements the limiting by sleeping the appropriate

730

amount of time after a network read that took less time than specified

731

by the rate. Eventually this strategy causes the TCP transfer to slow

732

down to approximately the specified rate. However, it may take some

733

time for this balance to be achieved, so don't be surprised if limiting

734

the rate doesn't work well with very small files.

735

736

@cindex pause

737

@cindex wait

738

@item -w @var{seconds}

739

@itemx --wait=@var{seconds}

740

Wait the specified number of seconds between the retrievals. Use of

741

this option is recommended, as it lightens the server load by making the

742

requests less frequent. Instead of in seconds, the time can be

743

specified in minutes using the @code{m} suffix, in hours using @code{h}

744

suffix, or in days using @code{d} suffix.

745

746

Specifying a large value for this option is useful if the network or the

747

destination host is down, so that Wget can wait long enough to

748

reasonably expect the network error to be fixed before the retry.

749

750

@cindex retries, waiting between

751

@cindex waiting between retries

752

@item --waitretry=@var{seconds}

753

If you don't want Wget to wait between @emph{every} retrieval, but only

754

between retries of failed downloads, you can use this option. Wget will

755

use @dfn{linear backoff}, waiting 1 second after the first failure on a

756

given file, then waiting 2 seconds after the second failure on that

757

file, up to the maximum number of @var{seconds} you specify. Therefore,

758

a value of 10 will actually make Wget wait up to (1 + 2 + ... + 10) = 55

759

seconds per file.

760

761

Note that this option is turned on by default in the global

762

@file{wgetrc} file.

763

764

@cindex wait, random

765

@cindex random wait

766

@item --random-wait

767

Some web sites may perform log analysis to identify retrieval programs

768

such as Wget by looking for statistically significant similarities in

769

the time between requests. This option causes the time between requests

770

to vary between 0 and 2 * @var{wait} seconds, where @var{wait} was

771

specified using the @samp{--wait} option, in order to mask Wget's

772

presence from such analysis.

773

774

A recent article in a publication devoted to development on a popular

775

consumer platform provided code to perform this analysis on the fly.

776

Its author suggested blocking at the class C address level to ensure

777

automated retrieval programs were blocked despite changing DHCP-supplied

778

addresses.

779

780

The @samp{--random-wait} option was inspired by this ill-advised

781

recommendation to block many unrelated users from a web site due to the

782

actions of one.

783

784

@cindex proxy

785

@item -Y on/off

786

@itemx --proxy=on/off

787

Turn proxy support on or off. The proxy is on by default if the

788

appropriate environment variable is defined.

789

790

For more information about the use of proxies with Wget, @xref{Proxies}.

791

792

@cindex quota

793

@item -Q @var{quota}

794

@itemx --quota=@var{quota}

795

Specify download quota for automatic retrievals. The value can be

796

specified in bytes (default), kilobytes (with @samp{k} suffix), or

797

megabytes (with @samp{m} suffix).

798

799

Note that quota will never affect downloading a single file. So if you

800

specify @samp{wget -Q10k ftp://wuarchive.wustl.edu/ls-lR.gz}, all of the

801

@file{ls-lR.gz} will be downloaded. The same goes even when several

802

@sc{url}s are specified on the command-line. However, quota is

803

respected when retrieving either recursively, or from an input file.

804

Thus you may safely type @samp{wget -Q2m -i sites}---download will be

805

aborted when the quota is exceeded.

806

807

Setting quota to 0 or to @samp{inf} unlimits the download quota.

808

809

@cindex DNS cache

810

@cindex caching of DNS lookups

811

@item --dns-cache=off

812

Turn off caching of DNS lookups. Normally, Wget remembers the addresses

813

it looked up from DNS so it doesn't have to repeatedly contact the DNS

814

server for the same (typically small) set of addresses it retrieves

815

from. This cache exists in memory only; a new Wget run will contact DNS

816

again.

817

818

However, in some cases it is not desirable to cache host names, even for

819

the duration of a short-running application like Wget. For example,

820

some HTTP servers are hosted on machines with dynamically allocated IP

821

addresses that change from time to time. Their DNS entries are updated

822

along with each change. When Wget's download from such a host gets

823

interrupted by IP address change, Wget retries the download, but (due to

824

DNS caching) it contacts the old address. With the DNS cache turned

825

off, Wget will repeat the DNS lookup for every connect and will thus get

826

the correct dynamic address every time---at the cost of additional DNS

827

lookups where they're probably not needed.

828

829

If you don't understand the above description, you probably won't need

830

this option.

831

832

@cindex file names, restrict

833

@cindex Windows file names

834

@item --restrict-file-names=@var{mode}

835

Change which characters found in remote URLs may show up in local file

836

names generated from those URLs. Characters that are @dfn{restricted}

837

by this option are escaped, i.e. replaced with @samp{%HH}, where

838

@samp{HH} is the hexadecimal number that corresponds to the restricted

839

character.

840

841

By default, Wget escapes the characters that are not valid as part of

842

file names on your operating system, as well as control characters that

843

are typically unprintable. This option is useful for changing these

844

defaults, either because you are downloading to a non-native partition,

845

or because you want to disable escaping of the control characters.

846

847

When mode is set to ``unix'', Wget escapes the character @samp{/} and

848

the control characters in the ranges 0--31 and 128--159. This is the

849

default on Unix-like OS'es.

850

851

When mode is set to ``windows'', Wget escapes the characters @samp{\},

852

@samp{|}, @samp{/}, @samp{:}, @samp{?}, @samp{"}, @samp{*}, @samp{<},

853

@samp{>}, and the control characters in the ranges 0--31 and 128--159.

854

In addition to this, Wget in Windows mode uses @samp{+} instead of

855

@samp{:} to separate host and port in local file names, and uses

856

@samp{@@} instead of @samp{?} to separate the query portion of the file

857

name from the rest. Therefore, a URL that would be saved as

858

@samp{www.xemacs.org:4300/search.pl?input=blah} in Unix mode would be

859

saved as @samp{www.xemacs.org+4300/search.pl@@input=blah} in Windows

860

mode. This mode is the default on Windows.

861

862

If you append @samp{,nocontrol} to the mode, as in

863

@samp{unix,nocontrol}, escaping of the control characters is also

864

switched off. You can use @samp{--restrict-file-names=nocontrol} to

865

turn off escaping of control characters without affecting the choice of

866

the OS to use as file name restriction mode.

867

@end table

868

869

@node Directory Options, HTTP Options, Download Options, Invoking

870

@section Directory Options

871

872

@table @samp

873

@item -nd

874

@itemx --no-directories

875

Do not create a hierarchy of directories when retrieving recursively.

876

With this option turned on, all files will get saved to the current

877

directory, without clobbering (if a name shows up more than once, the

878

filenames will get extensions @samp{.n}).

879

880

@item -x

881

@itemx --force-directories

882

The opposite of @samp{-nd}---create a hierarchy of directories, even if

883

one would not have been created otherwise. E.g. @samp{wget -x

884

http://fly.srk.fer.hr/robots.txt} will save the downloaded file to

885

@file{fly.srk.fer.hr/robots.txt}.

886

887

@item -nH

888

@itemx --no-host-directories

889

Disable generation of host-prefixed directories. By default, invoking

890

Wget with @samp{-r http://fly.srk.fer.hr/} will create a structure of

891

directories beginning with @file{fly.srk.fer.hr/}. This option disables

892

such behavior.

893

894

@cindex cut directories

895

@item --cut-dirs=@var{number}

896

Ignore @var{number} directory components. This is useful for getting a

897

fine-grained control over the directory where recursive retrieval will

898

be saved.

899

900

Take, for example, the directory at

901

@samp{ftp://ftp.xemacs.org/pub/xemacs/}. If you retrieve it with

902

@samp{-r}, it will be saved locally under

903

@file{ftp.xemacs.org/pub/xemacs/}. While the @samp{-nH} option can

904

remove the @file{ftp.xemacs.org/} part, you are still stuck with

905

@file{pub/xemacs}. This is where @samp{--cut-dirs} comes in handy; it

906

makes Wget not ``see'' @var{number} remote directory components. Here

907

are several examples of how @samp{--cut-dirs} option works.

908

909

@example

910

@group

911

No options -> ftp.xemacs.org/pub/xemacs/

912

-nH -> pub/xemacs/

913

-nH --cut-dirs=1 -> xemacs/

914

-nH --cut-dirs=2 -> .

915

916

--cut-dirs=1 -> ftp.xemacs.org/xemacs/

917

...

918

@end group

919

@end example

920

921

If you just want to get rid of the directory structure, this option is

922

similar to a combination of @samp{-nd} and @samp{-P}. However, unlike

923

@samp{-nd}, @samp{--cut-dirs} does not lose with subdirectories---for

924

instance, with @samp{-nH --cut-dirs=1}, a @file{beta/} subdirectory will

925

be placed to @file{xemacs/beta}, as one would expect.

926

927

@cindex directory prefix

928

@item -P @var{prefix}

929

@itemx --directory-prefix=@var{prefix}

930

Set directory prefix to @var{prefix}. The @dfn{directory prefix} is the

931

directory where all other files and subdirectories will be saved to,

932

i.e. the top of the retrieval tree. The default is @samp{.} (the

933

current directory).

934

@end table

935

936

@node HTTP Options, FTP Options, Directory Options, Invoking

937

@section HTTP Options

938

939

@table @samp

940

@cindex .html extension

941

@item -E

942

@itemx --html-extension

943

If a file of type @samp{application/xhtml+xml} or @samp{text/html} is

944

downloaded and the URL does not end with the regexp

945

@samp{\.[Hh][Tt][Mm][Ll]?}, this option will cause the suffix @samp{.html}

946

to be appended to the local filename. This is useful, for instance, when

947

you're mirroring a remote site that uses @samp{.asp} pages, but you want

948

the mirrored pages to be viewable on your stock Apache server. Another

949

good use for this is when you're downloading CGI-generated materials. A URL

950

like @samp{http://site.com/article.cgi?25} will be saved as

951

@file{article.cgi?25.html}.

952

953

Note that filenames changed in this way will be re-downloaded every time

954

you re-mirror a site, because Wget can't tell that the local

955

@file{@var{X}.html} file corresponds to remote URL @samp{@var{X}} (since

956

it doesn't yet know that the URL produces output of type

957

@samp{text/html} or @samp{application/xhtml+xml}. To prevent this

958

re-downloading, you must use @samp{-k} and @samp{-K} so that the original

959

version of the file will be saved as @file{@var{X}.orig} (@pxref{Recursive

960

Retrieval Options}).

961

962

@cindex http user

963

@cindex http password

964

@cindex authentication

965

@item --http-user=@var{user}

966

@itemx --http-passwd=@var{password}

967

Specify the username @var{user} and password @var{password} on an

968

@sc{http} server. According to the type of the challenge, Wget will

969

encode them using either the @code{basic} (insecure) or the

970

@code{digest} authentication scheme.

971

972

Another way to specify username and password is in the @sc{url} itself

973

(@pxref{URL Format}). Either method reveals your password to anyone who

974

bothers to run @code{ps}. To prevent the passwords from being seen,

975

store them in @file{.wgetrc} or @file{.netrc}, and make sure to protect

976

those files from other users with @code{chmod}. If the passwords are

977

really important, do not leave them lying in those files either---edit

978

the files and delete them after Wget has started the download.

979

980

For more information about security issues with Wget, @xref{Security

981

Considerations}.

982

983

@cindex proxy

984

@cindex cache

985

@item -C on/off

986

@itemx --cache=on/off

987

When set to off, disable server-side cache. In this case, Wget will

988

send the remote server an appropriate directive (@samp{Pragma:

989

no-cache}) to get the file from the remote service, rather than

990

returning the cached version. This is especially useful for retrieving

991

and flushing out-of-date documents on proxy servers.

992

993

Caching is allowed by default.

994

995

@cindex cookies

996

@item --cookies=on/off

997

When set to off, disable the use of cookies. Cookies are a mechanism

998

for maintaining server-side state. The server sends the client a cookie

999

using the @code{Set-Cookie} header, and the client responds with the

1000

same cookie upon further requests. Since cookies allow the server

1001

owners to keep track of visitors and for sites to exchange this

1002

information, some consider them a breach of privacy. The default is to

1003

use cookies; however, @emph{storing} cookies is not on by default.

1004

1005

@cindex loading cookies

1006

@cindex cookies, loading

1007

@item --load-cookies @var{file}

1008

Load cookies from @var{file} before the first HTTP retrieval.

1009

@var{file} is a textual file in the format originally used by Netscape's

1010

@file{cookies.txt} file.

1011

1012

You will typically use this option when mirroring sites that require

1013

that you be logged in to access some or all of their content. The login

1014

process typically works by the web server issuing an @sc{http} cookie

1015

upon receiving and verifying your credentials. The cookie is then

1016

resent by the browser when accessing that part of the site, and so

1017

proves your identity.

1018

1019

Mirroring such a site requires Wget to send the same cookies your

1020

browser sends when communicating with the site. This is achieved by

1021

@samp{--load-cookies}---simply point Wget to the location of the

1022

@file{cookies.txt} file, and it will send the same cookies your browser

1023

would send in the same situation. Different browsers keep textual

1024

cookie files in different locations:

1025

1026

@table @asis

1027

@item Netscape 4.x.

1028

The cookies are in @file{~/.netscape/cookies.txt}.

1029

1030

@item Mozilla and Netscape 6.x.

1031

Mozilla's cookie file is also named @file{cookies.txt}, located

1032

somewhere under @file{~/.mozilla}, in the directory of your profile.

1033

The full path usually ends up looking somewhat like

1034

@file{~/.mozilla/default/@var{some-weird-string}/cookies.txt}.

1035

1036

@item Internet Explorer.

1037

You can produce a cookie file Wget can use by using the File menu,

1038

Import and Export, Export Cookies. This has been tested with Internet

1039

Explorer 5; it is not guaranteed to work with earlier versions.

1040

1041

@item Other browsers.

1042

If you are using a different browser to create your cookies,

1043

@samp{--load-cookies} will only work if you can locate or produce a

1044

cookie file in the Netscape format that Wget expects.

1045

@end table

1046

1047

If you cannot use @samp{--load-cookies}, there might still be an

1048

alternative. If your browser supports a ``cookie manager'', you can use

1049

it to view the cookies used when accessing the site you're mirroring.

1050

Write down the name and value of the cookie, and manually instruct Wget

1051

to send those cookies, bypassing the ``official'' cookie support:

1052

1053

@example

1054

wget --cookies=off --header "Cookie: @var{name}=@var{value}"

1055

@end example

1056

1057

@cindex saving cookies

1058

@cindex cookies, saving

1059

@item --save-cookies @var{file}

1060

Save cookies to @var{file} at the end of session. Cookies whose expiry

1061

time is not specified, or those that have already expired, are not

1062

saved.

1063

1064

@cindex Content-Length, ignore

1065

@cindex ignore length

1066

@item --ignore-length

1067

Unfortunately, some @sc{http} servers (@sc{cgi} programs, to be more

1068

precise) send out bogus @code{Content-Length} headers, which makes Wget

1069

go wild, as it thinks not all the document was retrieved. You can spot

1070

this syndrome if Wget retries getting the same document again and again,

1071

each time claiming that the (otherwise normal) connection has closed on

1072

the very same byte.

1073

1074

With this option, Wget will ignore the @code{Content-Length} header---as

1075

if it never existed.

1076

1077

@cindex header, add

1078

@item --header=@var{additional-header}

1079

Define an @var{additional-header} to be passed to the @sc{http} servers.

1080

Headers must contain a @samp{:} preceded by one or more non-blank

1081

characters, and must not contain newlines.

1082

1083

You may define more than one additional header by specifying

1084

@samp{--header} more than once.

1085

1086

@example

1087

@group

1088

wget --header='Accept-Charset: iso-8859-2' \

1089

--header='Accept-Language: hr' \

1090

http://fly.srk.fer.hr/

1091

@end group

1092

@end example

1093

1094

Specification of an empty string as the header value will clear all

1095

previous user-defined headers.

1096

1097

@cindex proxy user

1098

@cindex proxy password

1099

@cindex proxy authentication

1100

@item --proxy-user=@var{user}

1101

@itemx --proxy-passwd=@var{password}

1102

Specify the username @var{user} and password @var{password} for

1103

authentication on a proxy server. Wget will encode them using the

1104

@code{basic} authentication scheme.

1105

1106

Security considerations similar to those with @samp{--http-passwd}

1107

pertain here as well.

1108

1109

@cindex http referer

1110

@cindex referer, http

1111

@item --referer=@var{url}

1112

Include `Referer: @var{url}' header in HTTP request. Useful for

1113

retrieving documents with server-side processing that assume they are

1114

always being retrieved by interactive web browsers and only come out

1115

properly when Referer is set to one of the pages that point to them.

1116

1117

@cindex server response, save

1118

@item -s

1119

@itemx --save-headers

1120

Save the headers sent by the @sc{http} server to the file, preceding the

1121

actual contents, with an empty line as the separator.

1122

1123

@cindex user-agent

1124

@item -U @var{agent-string}

1125

@itemx --user-agent=@var{agent-string}

1126

Identify as @var{agent-string} to the @sc{http} server.

1127

1128

The @sc{http} protocol allows the clients to identify themselves using a

1129

@code{User-Agent} header field. This enables distinguishing the

1130

@sc{www} software, usually for statistical purposes or for tracing of

1131

protocol violations. Wget normally identifies as

1132

@samp{Wget/@var{version}}, @var{version} being the current version

1133

number of Wget.

1134

1135

However, some sites have been known to impose the policy of tailoring

1136

the output according to the @code{User-Agent}-supplied information.

1137

While conceptually this is not such a bad idea, it has been abused by

1138

servers denying information to clients other than @code{Mozilla} or

1139

Microsoft @code{Internet Explorer}. This option allows you to change

1140

the @code{User-Agent} line issued by Wget. Use of this option is

1141

discouraged, unless you really know what you are doing.

1142

1143

@cindex POST

1144

@item --post-data=@var{string}

1145

@itemx --post-file=@var{file}

1146

Use POST as the method for all HTTP requests and send the specified data

1147

in the request body. @code{--post-data} sends @var{string} as data,

1148

whereas @code{--post-file} sends the contents of @var{file}. Other than

1149

that, they work in exactly the same way.

1150

1151

Please be aware that Wget needs to know the size of the POST data in

1152

advance. Therefore the argument to @code{--post-file} must be a regular

1153

file; specifying a FIFO or something like @file{/dev/stdin} won't work.

1154

It's not quite clear how to work around this limitation inherent in

1155

HTTP/1.0. Although HTTP/1.1 introduces @dfn{chunked} transfer that

1156

doesn't require knowing the request length in advance, a client can't

1157

use chunked unless it knows it's talking to an HTTP/1.1 server. And it

1158

can't know that until it receives a response, which in turn requires the

1159

request to have been completed -- a chicken-and-egg problem.

1160

1161

Note: if Wget is redirected after the POST request is completed, it will

1162

not send the POST data to the redirected URL. This is because URLs that

1163

process POST often respond with a redirection to a regular page

1164

(although that's technically disallowed), which does not desire or

1165

accept POST. It is not yet clear that this behavior is optimal; if it

1166

doesn't work out, it will be changed.

1167

1168

This example shows how to log to a server using POST and then proceed to

1169

download the desired pages, presumably only accessible to authorized

1170

users:

1171

1172

@example

1173

@group

1174

# @r{Log in to the server. This can be done only once.}

1175

wget --save-cookies cookies.txt \

1176

--post-data 'user=foo&password=bar' \

1177

http://server.com/auth.php

1178

1179

# @r{Now grab the page or pages we care about.}

1180

wget --load-cookies cookies.txt \

1181

-p http://server.com/interesting/article.php

1182

@end group

1183

@end example

1184

@end table

1185

1186

@node FTP Options, Recursive Retrieval Options, HTTP Options, Invoking

1187

@section FTP Options

1188

1189

@table @samp

1190

@cindex .listing files, removing

1191

@item -nr

1192

@itemx --dont-remove-listing

1193

Don't remove the temporary @file{.listing} files generated by @sc{ftp}

1194

retrievals. Normally, these files contain the raw directory listings

1195

received from @sc{ftp} servers. Not removing them can be useful for

1196

debugging purposes, or when you want to be able to easily check on the

1197

contents of remote server directories (e.g. to verify that a mirror

1198

you're running is complete).

1199

1200

Note that even though Wget writes to a known filename for this file,

1201

this is not a security hole in the scenario of a user making

1202

@file{.listing} a symbolic link to @file{/etc/passwd} or something and

1203

asking @code{root} to run Wget in his or her directory. Depending on

1204

the options used, either Wget will refuse to write to @file{.listing},

1205

making the globbing/recursion/time-stamping operation fail, or the

1206

symbolic link will be deleted and replaced with the actual

1207

@file{.listing} file, or the listing will be written to a

1208

@file{.listing.@var{number}} file.

1209

1210

Even though this situation isn't a problem, though, @code{root} should

1211

never run Wget in a non-trusted user's directory. A user could do

1212

something as simple as linking @file{index.html} to @file{/etc/passwd}

1213

and asking @code{root} to run Wget with @samp{-N} or @samp{-r} so the file

1214

will be overwritten.

1215

1216

@cindex globbing, toggle

1217

@item -g on/off

1218

@itemx --glob=on/off

1219

Turn @sc{ftp} globbing on or off. Globbing means you may use the

1220

shell-like special characters (@dfn{wildcards}), like @samp{*},

1221

@samp{?}, @samp{[} and @samp{]} to retrieve more than one file from the

1222

same directory at once, like:

1223

1224

@example

1225

wget ftp://gnjilux.srk.fer.hr/*.msg

1226

@end example

1227

1228

By default, globbing will be turned on if the @sc{url} contains a

1229

globbing character. This option may be used to turn globbing on or off

1230

permanently.

1231

1232

You may have to quote the @sc{url} to protect it from being expanded by

1233

your shell. Globbing makes Wget look for a directory listing, which is

1234

system-specific. This is why it currently works only with Unix @sc{ftp}

1235

servers (and the ones emulating Unix @code{ls} output).

1236

1237

@cindex passive ftp

1238

@item --passive-ftp

1239

Use the @dfn{passive} @sc{ftp} retrieval scheme, in which the client

1240

initiates the data connection. This is sometimes required for @sc{ftp}

1241

to work behind firewalls.

1242

1243

@cindex symbolic links, retrieving

1244

@item --retr-symlinks

1245

Usually, when retrieving @sc{ftp} directories recursively and a symbolic

1246

link is encountered, the linked-to file is not downloaded. Instead, a

1247

matching symbolic link is created on the local filesystem. The

1248

pointed-to file will not be downloaded unless this recursive retrieval

1249

would have encountered it separately and downloaded it anyway.

1250

1251

When @samp{--retr-symlinks} is specified, however, symbolic links are

1252

traversed and the pointed-to files are retrieved. At this time, this

1253

option does not cause Wget to traverse symlinks to directories and

1254

recurse through them, but in the future it should be enhanced to do

1255

this.

1256

1257

Note that when retrieving a file (not a directory) because it was

1258

specified on the command-line, rather than because it was recursed to,

1259

this option has no effect. Symbolic links are always traversed in this

1260

case.

1261

@end table

1262

1263

@node Recursive Retrieval Options, Recursive Accept/Reject Options, FTP Options, Invoking

1264

@section Recursive Retrieval Options

1265

1266

@table @samp

1267

@item -r

1268

@itemx --recursive

1269

Turn on recursive retrieving. @xref{Recursive Retrieval}, for more

1270

details.

1271

1272

@item -l @var{depth}

1273

@itemx --level=@var{depth}

1274

Specify recursion maximum depth level @var{depth} (@pxref{Recursive

1275

Retrieval}). The default maximum depth is 5.

1276

1277

@cindex proxy filling

1278

@cindex delete after retrieval

1279

@cindex filling proxy cache

1280

@item --delete-after

1281

This option tells Wget to delete every single file it downloads,

1282

@emph{after} having done so. It is useful for pre-fetching popular

1283

pages through a proxy, e.g.:

1284

1285

@example

1286

wget -r -nd --delete-after http://whatever.com/~popular/page/

1287

@end example

1288

1289

The @samp{-r} option is to retrieve recursively, and @samp{-nd} to not

1290

create directories.

1291

1292

Note that @samp{--delete-after} deletes files on the local machine. It

1293

does not issue the @samp{DELE} command to remote FTP sites, for

1294

instance. Also note that when @samp{--delete-after} is specified,

1295

@samp{--convert-links} is ignored, so @samp{.orig} files are simply not

1296

created in the first place.

1297

1298

@cindex conversion of links

1299

@cindex link conversion

1300

@item -k

1301

@itemx --convert-links

1302

After the download is complete, convert the links in the document to

1303

make them suitable for local viewing. This affects not only the visible

1304

hyperlinks, but any part of the document that links to external content,

1305

such as embedded images, links to style sheets, hyperlinks to non-@sc{html}

1306

content, etc.

1307

1308

Each link will be changed in one of the two ways:

1309

1310

@itemize @bullet

1311

@item

1312

The links to files that have been downloaded by Wget will be changed to

1313

refer to the file they point to as a relative link.

1314

1315

Example: if the downloaded file @file{/foo/doc.html} links to

1316

@file{/bar/img.gif}, also downloaded, then the link in @file{doc.html}

1317

will be modified to point to @samp{../bar/img.gif}. This kind of

1318

transformation works reliably for arbitrary combinations of directories.

1319

1320

@item

1321

The links to files that have not been downloaded by Wget will be changed

1322

to include host name and absolute path of the location they point to.

1323

1324

Example: if the downloaded file @file{/foo/doc.html} links to

1325

@file{/bar/img.gif} (or to @file{../bar/img.gif}), then the link in

1326

@file{doc.html} will be modified to point to

1327

@file{http://@var{hostname}/bar/img.gif}.

1328

@end itemize

1329

1330

Because of this, local browsing works reliably: if a linked file was

1331

downloaded, the link will refer to its local name; if it was not

1332

downloaded, the link will refer to its full Internet address rather than

1333

presenting a broken link. The fact that the former links are converted

1334

to relative links ensures that you can move the downloaded hierarchy to

1335

another directory.

1336

1337

Note that only at the end of the download can Wget know which links have

1338

been downloaded. Because of that, the work done by @samp{-k} will be

1339

performed at the end of all the downloads.

1340

1341

@cindex backing up converted files

1342

@item -K

1343

@itemx --backup-converted

1344

When converting a file, back up the original version with a @samp{.orig}

1345

suffix. Affects the behavior of @samp{-N} (@pxref{HTTP Time-Stamping

1346

Internals}).

1347

1348

@item -m

1349

@itemx --mirror

1350

Turn on options suitable for mirroring. This option turns on recursion

1351

and time-stamping, sets infinite recursion depth and keeps @sc{ftp}

1352

directory listings. It is currently equivalent to

1353

@samp{-r -N -l inf -nr}.

1354

1355

@cindex page requisites

1356

@cindex required images, downloading

1357

@item -p

1358

@itemx --page-requisites

1359

This option causes Wget to download all the files that are necessary to

1360

properly display a given @sc{html} page. This includes such things as

1361

inlined images, sounds, and referenced stylesheets.

1362

1363

Ordinarily, when downloading a single @sc{html} page, any requisite documents

1364

that may be needed to display it properly are not downloaded. Using

1365

@samp{-r} together with @samp{-l} can help, but since Wget does not

1366

ordinarily distinguish between external and inlined documents, one is

1367

generally left with ``leaf documents'' that are missing their

1368

requisites.

1369

1370

For instance, say document @file{1.html} contains an @code{<IMG>} tag

1371

referencing @file{1.gif} and an @code{<A>} tag pointing to external

1372

document @file{2.html}. Say that @file{2.html} is similar but that its

1373

image is @file{2.gif} and it links to @file{3.html}. Say this

1374

continues up to some arbitrarily high number.

1375

1376

If one executes the command:

1377

1378

@example

1379

wget -r -l 2 http://@var{site}/1.html

1380

@end example

1381

1382

then @file{1.html}, @file{1.gif}, @file{2.html}, @file{2.gif}, and

1383

@file{3.html} will be downloaded. As you can see, @file{3.html} is

1384

without its requisite @file{3.gif} because Wget is simply counting the

1385

number of hops (up to 2) away from @file{1.html} in order to determine

1386

where to stop the recursion. However, with this command:

1387

1388

@example

1389

wget -r -l 2 -p http://@var{site}/1.html

1390

@end example

1391

1392

all the above files @emph{and} @file{3.html}'s requisite @file{3.gif}

1393

will be downloaded. Similarly,

1394

1395

@example

1396

wget -r -l 1 -p http://@var{site}/1.html

1397

@end example

1398

1399

will cause @file{1.html}, @file{1.gif}, @file{2.html}, and @file{2.gif}

1400

to be downloaded. One might think that:

1401

1402

@example

1403

wget -r -l 0 -p http://@var{site}/1.html

1404

@end example

1405

1406

would download just @file{1.html} and @file{1.gif}, but unfortunately

1407

this is not the case, because @samp{-l 0} is equivalent to

1408

@samp{-l inf}---that is, infinite recursion. To download a single @sc{html}

1409

page (or a handful of them, all specified on the command-line or in a

1410

@samp{-i} @sc{url} input file) and its (or their) requisites, simply leave off

1411

@samp{-r} and @samp{-l}:

1412

1413

@example

1414

wget -p http://@var{site}/1.html

1415

@end example

1416

1417

Note that Wget will behave as if @samp{-r} had been specified, but only

1418

that single page and its requisites will be downloaded. Links from that

1419

page to external documents will not be followed. Actually, to download

1420

a single page and all its requisites (even if they exist on separate

1421

websites), and make sure the lot displays properly locally, this author

1422

likes to use a few options in addition to @samp{-p}:

1423

1424

@example

1425

wget -E -H -k -K -p http://@var{site}/@var{document}

1426

@end example

1427

1428

To finish off this topic, it's worth knowing that Wget's idea of an

1429

external document link is any URL specified in an @code{<A>} tag, an

1430

@code{<AREA>} tag, or a @code{<LINK>} tag other than @code{<LINK

1431

REL="stylesheet">}.

1432

1433

@cindex @sc{html} comments

1434

@cindex comments, @sc{html}

1435

@item --strict-comments

1436

Turn on strict parsing of @sc{html} comments. The default is to terminate

1437

comments at the first occurrence of @samp{-->}.

1438

1439

According to specifications, @sc{html} comments are expressed as @sc{sgml}

1440

@dfn{declarations}. Declaration is special markup that begins with

1441

@samp{<!} and ends with @samp{>}, such as @samp{<!DOCTYPE ...>}, that

1442

may contain comments between a pair of @samp{--} delimiters. @sc{html}

1443

comments are ``empty declarations'', @sc{sgml} declarations without any

1444

non-comment text. Therefore, @samp{} is a valid comment, and

1445

so is @samp{}, but @samp{} is not.

1446

1447

On the other hand, most @sc{html} writers don't perceive comments as anything

1448

other than text delimited with @samp{}, which is not

1449

quite the same. For example, something like @samp{}

1450

works as a valid comment as long as the number of dashes is a multiple

1451

of four (!). If not, the comment technically lasts until the next

1452

@samp{--}, which may be at the other end of the document. Because of

1453

this, many popular browsers completely ignore the specification and

1454

implement what users have come to expect: comments delimited with

1455

@samp{}.

1456

1457

Until version 1.9, Wget interpreted comments strictly, which resulted in

1458

missing links in many web pages that displayed fine in browsers, but had

1459

the misfortune of containing non-compliant comments. Beginning with

1460

version 1.9, Wget has joined the ranks of clients that implements

1461

``naive'' comments, terminating each comment at the first occurrence of

1462

@samp{-->}.

1463

1464

If, for whatever reason, you want strict comment parsing, use this

1465

option to turn it on.

1466

@end table

1467

1468

@node Recursive Accept/Reject Options, , Recursive Retrieval Options, Invoking

1469

@section Recursive Accept/Reject Options

1470

1471

@table @samp

1472

@item -A @var{acclist} --accept @var{acclist}

1473

@itemx -R @var{rejlist} --reject @var{rejlist}

1474

Specify comma-separated lists of file name suffixes or patterns to

1475

accept or reject (@pxref{Types of Files} for more details).

1476

1477

@item -D @var{domain-list}

1478

@itemx --domains=@var{domain-list}

1479

Set domains to be followed. @var{domain-list} is a comma-separated list

1480

of domains. Note that it does @emph{not} turn on @samp{-H}.

1481

1482

@item --exclude-domains @var{domain-list}

1483

Specify the domains that are @emph{not} to be followed.

1484

(@pxref{Spanning Hosts}).

1485

1486

@cindex follow FTP links

1487

@item --follow-ftp

1488

Follow @sc{ftp} links from @sc{html} documents. Without this option,

1489

Wget will ignore all the @sc{ftp} links.

1490

1491

@cindex tag-based recursive pruning

1492

@item --follow-tags=@var{list}

1493

Wget has an internal table of @sc{html} tag / attribute pairs that it

1494

considers when looking for linked documents during a recursive

1495

retrieval. If a user wants only a subset of those tags to be

1496

considered, however, he or she should be specify such tags in a

1497

comma-separated @var{list} with this option.

1498

1499

@item -G @var{list}

1500

@itemx --ignore-tags=@var{list}

1501

This is the opposite of the @samp{--follow-tags} option. To skip

1502

certain @sc{html} tags when recursively looking for documents to download,

1503

specify them in a comma-separated @var{list}.

1504

1505

In the past, the @samp{-G} option was the best bet for downloading a

1506

single page and its requisites, using a command-line like:

1507

1508

@example

1509

wget -Ga,area -H -k -K -r http://@var{site}/@var{document}

1510

@end example

1511

1512

However, the author of this option came across a page with tags like

1513

@code{<LINK REL="home" HREF="/">} and came to the realization that

1514

@samp{-G} was not enough. One can't just tell Wget to ignore

1515

@code{<LINK>}, because then stylesheets will not be downloaded. Now the

1516

best bet for downloading a single page and its requisites is the

1517

dedicated @samp{--page-requisites} option.

1518

1519

@item -H

1520

@itemx --span-hosts

1521

Enable spanning across hosts when doing recursive retrieving

1522

(@pxref{Spanning Hosts}).

1523

1524

@item -L

1525

@itemx --relative

1526

Follow relative links only. Useful for retrieving a specific home page

1527

without any distractions, not even those from the same hosts

1528

(@pxref{Relative Links}).

1529

1530

@item -I @var{list}

1531

@itemx --include-directories=@var{list}

1532

Specify a comma-separated list of directories you wish to follow when

1533

downloading (@pxref{Directory-Based Limits} for more details.) Elements

1534

of @var{list} may contain wildcards.

1535

1536

@item -X @var{list}

1537

@itemx --exclude-directories=@var{list}

1538

Specify a comma-separated list of directories you wish to exclude from

1539

download (@pxref{Directory-Based Limits} for more details.) Elements of

1540

@var{list} may contain wildcards.

1541

1542

@item -np

1543

@item --no-parent

1544

Do not ever ascend to the parent directory when retrieving recursively.

1545

This is a useful option, since it guarantees that only the files

1546

@emph{below} a certain hierarchy will be downloaded.

1547

@xref{Directory-Based Limits}, for more details.

1548

@end table

1549

1550

@c man end

1551

1552

@node Recursive Retrieval, Following Links, Invoking, Top

1553

@chapter Recursive Retrieval

1554

@cindex recursion

1555

@cindex retrieving

1556

@cindex recursive retrieval

1557

1558

GNU Wget is capable of traversing parts of the Web (or a single

1559

@sc{http} or @sc{ftp} server), following links and directory structure.

1560

We refer to this as to @dfn{recursive retrieval}, or @dfn{recursion}.

1561

1562

With @sc{http} @sc{url}s, Wget retrieves and parses the @sc{html} from

1563

the given @sc{url}, documents, retrieving the files the @sc{html}

1564

document was referring to, through markup like @code{href}, or

1565

@code{src}. If the freshly downloaded file is also of type

1566

@code{text/html} or @code{application/xhtml+xml}, it will be parsed and

1567

followed further.

1568

1569

Recursive retrieval of @sc{http} and @sc{html} content is

1570

@dfn{breadth-first}. This means that Wget first downloads the requested

1571

@sc{html} document, then the documents linked from that document, then the

1572

documents linked by them, and so on. In other words, Wget first

1573

downloads the documents at depth 1, then those at depth 2, and so on

1574

until the specified maximum depth.

1575

1576

The maximum @dfn{depth} to which the retrieval may descend is specified

1577

with the @samp{-l} option. The default maximum depth is five layers.

1578

1579

When retrieving an @sc{ftp} @sc{url} recursively, Wget will retrieve all

1580

the data from the given directory tree (including the subdirectories up

1581

to the specified depth) on the remote server, creating its mirror image

1582

locally. @sc{ftp} retrieval is also limited by the @code{depth}

1583

parameter. Unlike @sc{http} recursion, @sc{ftp} recursion is performed

1584

depth-first.

1585

1586

By default, Wget will create a local directory tree, corresponding to

1587

the one found on the remote server.

1588

1589

Recursive retrieving can find a number of applications, the most

1590

important of which is mirroring. It is also useful for @sc{www}

1591

presentations, and any other opportunities where slow network

1592

connections should be bypassed by storing the files locally.

1593

1594

You should be warned that recursive downloads can overload the remote

1595

servers. Because of that, many administrators frown upon them and may

1596

ban access from your site if they detect very fast downloads of big

1597

amounts of content. When downloading from Internet servers, consider

1598

using the @samp{-w} option to introduce a delay between accesses to the

1599

server. The download will take a while longer, but the server

1600

administrator will not be alarmed by your rudeness.

1601

1602

Of course, recursive download may cause problems on your machine. If

1603

left to run unchecked, it can easily fill up the disk. If downloading

1604

from local network, it can also take bandwidth on the system, as well as

1605

consume memory and CPU.

1606

1607

Try to specify the criteria that match the kind of download you are

1608

trying to achieve. If you want to download only one page, use

1609

@samp{--page-requisites} without any additional recursion. If you want

1610

to download things under one directory, use @samp{-np} to avoid

1611

downloading things from other directories. If you want to download all

1612

the files from one directory, use @samp{-l 1} to make sure the recursion

1613

depth never exceeds one. @xref{Following Links}, for more information

1614

about this.

1615

1616

Recursive retrieval should be used with care. Don't say you were not

1617

warned.

1618

1619

@node Following Links, Time-Stamping, Recursive Retrieval, Top

1620

@chapter Following Links

1621

@cindex links

1622

@cindex following links

1623

1624

When retrieving recursively, one does not wish to retrieve loads of

1625

unnecessary data. Most of the time the users bear in mind exactly what

1626

they want to download, and want Wget to follow only specific links.

1627

1628

For example, if you wish to download the music archive from

1629

@samp{fly.srk.fer.hr}, you will not want to download all the home pages

1630

that happen to be referenced by an obscure part of the archive.

1631

1632

Wget possesses several mechanisms that allows you to fine-tune which

1633

links it will follow.

1634

1635

@menu

1636

* Spanning Hosts:: (Un)limiting retrieval based on host name.

1637

* Types of Files:: Getting only certain files.

1638

* Directory-Based Limits:: Getting only certain directories.

1639

* Relative Links:: Follow relative links only.

1640

* FTP Links:: Following FTP links.

1641

@end menu

1642

1643

@node Spanning Hosts, Types of Files, Following Links, Following Links

1644

@section Spanning Hosts

1645

@cindex spanning hosts

1646

@cindex hosts, spanning

1647

1648

Wget's recursive retrieval normally refuses to visit hosts different

1649

than the one you specified on the command line. This is a reasonable

1650

default; without it, every retrieval would have the potential to turn

1651

your Wget into a small version of google.

1652

1653

However, visiting different hosts, or @dfn{host spanning,} is sometimes

1654

a useful option. Maybe the images are served from a different server.

1655

Maybe you're mirroring a site that consists of pages interlinked between

1656

three servers. Maybe the server has two equivalent names, and the @sc{html}

1657

pages refer to both interchangeably.

1658

1659

@table @asis

1660

@item Span to any host---@samp{-H}

1661

1662

The @samp{-H} option turns on host spanning, thus allowing Wget's

1663

recursive run to visit any host referenced by a link. Unless sufficient

1664

recursion-limiting criteria are applied depth, these foreign hosts will

1665

typically link to yet more hosts, and so on until Wget ends up sucking

1666

up much more data than you have intended.

1667

1668

@item Limit spanning to certain domains---@samp{-D}

1669

1670

The @samp{-D} option allows you to specify the domains that will be

1671

followed, thus limiting the recursion only to the hosts that belong to

1672

these domains. Obviously, this makes sense only in conjunction with

1673

@samp{-H}. A typical example would be downloading the contents of

1674

@samp{www.server.com}, but allowing downloads from

1675

@samp{images.server.com}, etc.:

1676

1677

@example

1678

wget -rH -Dserver.com http://www.server.com/

1679

@end example

1680

1681

You can specify more than one address by separating them with a comma,

1682

e.g. @samp{-Ddomain1.com,domain2.com}.

1683

1684

@item Keep download off certain domains---@samp{--exclude-domains}

1685

1686

If there are domains you want to exclude specifically, you can do it

1687

with @samp{--exclude-domains}, which accepts the same type of arguments

1688

of @samp{-D}, but will @emph{exclude} all the listed domains. For

1689

example, if you want to download all the hosts from @samp{foo.edu}

1690

domain, with the exception of @samp{sunsite.foo.edu}, you can do it like

1691

this:

1692

1693

@example

1694

wget -rH -Dfoo.edu --exclude-domains sunsite.foo.edu \

1695

http://www.foo.edu/

1696

@end example

1697

1698

@end table

1699

1700

@node Types of Files, Directory-Based Limits, Spanning Hosts, Following Links

1701

@section Types of Files

1702

@cindex types of files

1703

1704

When downloading material from the web, you will often want to restrict

1705

the retrieval to only certain file types. For example, if you are

1706

interested in downloading @sc{gif}s, you will not be overjoyed to get

1707

loads of PostScript documents, and vice versa.

1708

1709

Wget offers two options to deal with this problem. Each option

1710

description lists a short name, a long name, and the equivalent command

1711

in @file{.wgetrc}.

1712

1713

@cindex accept wildcards

1714

@cindex accept suffixes

1715

@cindex wildcards, accept

1716

@cindex suffixes, accept

1717

@table @samp

1718

@item -A @var{acclist}

1719

@itemx --accept @var{acclist}

1720

@itemx accept = @var{acclist}

1721

The argument to @samp{--accept} option is a list of file suffixes or

1722

patterns that Wget will download during recursive retrieval. A suffix

1723

is the ending part of a file, and consists of ``normal'' letters,

1724

e.g. @samp{gif} or @samp{.jpg}. A matching pattern contains shell-like

1725

wildcards, e.g. @samp{books*} or @samp{zelazny*196[0-9]*}.

1726

1727

So, specifying @samp{wget -A gif,jpg} will make Wget download only the

1728

files ending with @samp{gif} or @samp{jpg}, i.e. @sc{gif}s and

1729

@sc{jpeg}s. On the other hand, @samp{wget -A "zelazny*196[0-9]*"} will

1730

download only files beginning with @samp{zelazny} and containing numbers

1731

from 1960 to 1969 anywhere within. Look up the manual of your shell for

1732

a description of how pattern matching works.

1733

1734

Of course, any number of suffixes and patterns can be combined into a

1735

comma-separated list, and given as an argument to @samp{-A}.

1736

1737

@cindex reject wildcards

1738

@cindex reject suffixes

1739

@cindex wildcards, reject

1740

@cindex suffixes, reject

1741

@item -R @var{rejlist}

1742

@itemx --reject @var{rejlist}

1743

@itemx reject = @var{rejlist}

1744

The @samp{--reject} option works the same way as @samp{--accept}, only

1745

its logic is the reverse; Wget will download all files @emph{except} the

1746

ones matching the suffixes (or patterns) in the list.

1747

1748

So, if you want to download a whole page except for the cumbersome

1749

@sc{mpeg}s and @sc{.au} files, you can use @samp{wget -R mpg,mpeg,au}.

1750

Analogously, to download all files except the ones beginning with

1751

@samp{bjork}, use @samp{wget -R "bjork*"}. The quotes are to prevent

1752

expansion by the shell.

1753

@end table

1754

1755

The @samp{-A} and @samp{-R} options may be combined to achieve even

1756

better fine-tuning of which files to retrieve. E.g. @samp{wget -A

1757

"*zelazny*" -R .ps} will download all the files having @samp{zelazny} as

1758

a part of their name, but @emph{not} the PostScript files.

1759

1760

Note that these two options do not affect the downloading of @sc{html}

1761

files; Wget must load all the @sc{html}s to know where to go at

1762

all---recursive retrieval would make no sense otherwise.

1763

1764

@node Directory-Based Limits, Relative Links, Types of Files, Following Links

1765

@section Directory-Based Limits

1766

@cindex directories

1767

@cindex directory limits

1768

1769

Regardless of other link-following facilities, it is often useful to

1770

place the restriction of what files to retrieve based on the directories

1771

those files are placed in. There can be many reasons for this---the

1772

home pages may be organized in a reasonable directory structure; or some

1773

directories may contain useless information, e.g. @file{/cgi-bin} or

1774

@file{/dev} directories.

1775

1776

Wget offers three different options to deal with this requirement. Each

1777

option description lists a short name, a long name, and the equivalent

1778

command in @file{.wgetrc}.

1779

1780

@cindex directories, include

1781

@cindex include directories

1782

@cindex accept directories

1783

@table @samp

1784

@item -I @var{list}

1785

@itemx --include @var{list}

1786

@itemx include_directories = @var{list}

1787

@samp{-I} option accepts a comma-separated list of directories included

1788

in the retrieval. Any other directories will simply be ignored. The

1789

directories are absolute paths.

1790

1791

So, if you wish to download from @samp{http://host/people/bozo/}

1792

following only links to bozo's colleagues in the @file{/people}

1793

directory and the bogus scripts in @file{/cgi-bin}, you can specify:

1794

1795

@example

1796

wget -I /people,/cgi-bin http://host/people/bozo/

1797

@end example

1798

1799

@cindex directories, exclude

1800

@cindex exclude directories

1801

@cindex reject directories

1802

@item -X @var{list}

1803

@itemx --exclude @var{list}

1804

@itemx exclude_directories = @var{list}

1805

@samp{-X} option is exactly the reverse of @samp{-I}---this is a list of

1806

directories @emph{excluded} from the download. E.g. if you do not want

1807

Wget to download things from @file{/cgi-bin} directory, specify @samp{-X

1808

/cgi-bin} on the command line.

1809

1810

The same as with @samp{-A}/@samp{-R}, these two options can be combined

1811

to get a better fine-tuning of downloading subdirectories. E.g. if you

1812

want to load all the files from @file{/pub} hierarchy except for

1813

@file{/pub/worthless}, specify @samp{-I/pub -X/pub/worthless}.

1814

1815

@cindex no parent

1816

@item -np

1817

@itemx --no-parent

1818

@itemx no_parent = on

1819

The simplest, and often very useful way of limiting directories is

1820

disallowing retrieval of the links that refer to the hierarchy

1821

@dfn{above} than the beginning directory, i.e. disallowing ascent to the

1822

parent directory/directories.

1823

1824

The @samp{--no-parent} option (short @samp{-np}) is useful in this case.

1825

Using it guarantees that you will never leave the existing hierarchy.

1826

Supposing you issue Wget with:

1827

1828

@example

1829

wget -r --no-parent http://somehost/~luzer/my-archive/

1830

@end example

1831

1832

You may rest assured that none of the references to

1833

@file{/~his-girls-homepage/} or @file{/~luzer/all-my-mpegs/} will be

1834

followed. Only the archive you are interested in will be downloaded.

1835

Essentially, @samp{--no-parent} is similar to

1836

@samp{-I/~luzer/my-archive}, only it handles redirections in a more

1837

intelligent fashion.

1838

@end table

1839

1840

@node Relative Links, FTP Links, Directory-Based Limits, Following Links

1841

@section Relative Links

1842

@cindex relative links

1843

1844

When @samp{-L} is turned on, only the relative links are ever followed.

1845

Relative links are here defined those that do not refer to the web

1846

server root. For example, these links are relative:

1847

1848

@example

1849

1850

1851

1852

@end example

1853

1854

These links are not relative:

1855

1856

@example

1857

1858

1859

1860

@end example

1861

1862

Using this option guarantees that recursive retrieval will not span

1863

hosts, even without @samp{-H}. In simple cases it also allows downloads

1864

to ``just work'' without having to convert links.

1865

1866

This option is probably not very useful and might be removed in a future

1867

release.

1868

1869

@node FTP Links, , Relative Links, Following Links

1870

@section Following FTP Links

1871

@cindex following ftp links

1872

1873

The rules for @sc{ftp} are somewhat specific, as it is necessary for

1874

them to be. @sc{ftp} links in @sc{html} documents are often included

1875

for purposes of reference, and it is often inconvenient to download them

1876

by default.

1877

1878

To have @sc{ftp} links followed from @sc{html} documents, you need to

1879

specify the @samp{--follow-ftp} option. Having done that, @sc{ftp}

1880

links will span hosts regardless of @samp{-H} setting. This is logical,

1881

as @sc{ftp} links rarely point to the same host where the @sc{http}

1882

server resides. For similar reasons, the @samp{-L} options has no

1883

effect on such downloads. On the other hand, domain acceptance

1884

(@samp{-D}) and suffix rules (@samp{-A} and @samp{-R}) apply normally.

1885

1886

Also note that followed links to @sc{ftp} directories will not be

1887

retrieved recursively further.

1888

1889

@node Time-Stamping, Startup File, Following Links, Top

1890

@chapter Time-Stamping

1891

@cindex time-stamping

1892

@cindex timestamping

1893

@cindex updating the archives

1894

@cindex incremental updating

1895

1896

One of the most important aspects of mirroring information from the

1897

Internet is updating your archives.

1898

1899

Downloading the whole archive again and again, just to replace a few

1900

changed files is expensive, both in terms of wasted bandwidth and money,

1901

and the time to do the update. This is why all the mirroring tools

1902

offer the option of incremental updating.

1903

1904

Such an updating mechanism means that the remote server is scanned in

1905

search of @dfn{new} files. Only those new files will be downloaded in

1906

the place of the old ones.

1907

1908

A file is considered new if one of these two conditions are met:

1909

1910

@enumerate

1911

@item

1912

A file of that name does not already exist locally.

1913

1914

@item

1915

A file of that name does exist, but the remote file was modified more

1916

recently than the local file.

1917

@end enumerate

1918

1919

To implement this, the program needs to be aware of the time of last

1920

modification of both local and remote files. We call this information the

1921

@dfn{time-stamp} of a file.

1922

1923

The time-stamping in GNU Wget is turned on using @samp{--timestamping}

1924

(@samp{-N}) option, or through @code{timestamping = on} directive in

1925

@file{.wgetrc}. With this option, for each file it intends to download,

1926

Wget will check whether a local file of the same name exists. If it

1927

does, and the remote file is older, Wget will not download it.

1928

1929

If the local file does not exist, or the sizes of the files do not

1930

match, Wget will download the remote file no matter what the time-stamps

1931

say.

1932

1933

@menu

1934

* Time-Stamping Usage::

1935

* HTTP Time-Stamping Internals::

1936

* FTP Time-Stamping Internals::

1937

@end menu

1938

1939

@node Time-Stamping Usage, HTTP Time-Stamping Internals, Time-Stamping, Time-Stamping

1940

@section Time-Stamping Usage

1941

@cindex time-stamping usage

1942

@cindex usage, time-stamping

1943

1944

The usage of time-stamping is simple. Say you would like to download a

1945

file so that it keeps its date of modification.

1946

1947

@example

1948

wget -S http://www.gnu.ai.mit.edu/

1949

@end example

1950

1951

A simple @code{ls -l} shows that the time stamp on the local file equals

1952

the state of the @code{Last-Modified} header, as returned by the server.

1953

As you can see, the time-stamping info is preserved locally, even

1954

without @samp{-N} (at least for @sc{http}).

1955

1956

Several days later, you would like Wget to check if the remote file has

1957

changed, and download it if it has.

1958

1959

@example

1960

wget -N http://www.gnu.ai.mit.edu/

1961

@end example

1962

1963

Wget will ask the server for the last-modified date. If the local file

1964

has the same timestamp as the server, or a newer one, the remote file

1965

will not be re-fetched. However, if the remote file is more recent,

1966

Wget will proceed to fetch it.

1967

1968

The same goes for @sc{ftp}. For example:

1969

1970

@example

1971

wget "ftp://ftp.ifi.uio.no/pub/emacs/gnus/*"

1972

@end example

1973

1974

(The quotes around that URL are to prevent the shell from trying to

1975

interpret the @samp{*}.)

1976

1977

After download, a local directory listing will show that the timestamps

1978

match those on the remote server. Reissuing the command with @samp{-N}

1979

will make Wget re-fetch @emph{only} the files that have been modified

1980

since the last download.

1981

1982

If you wished to mirror the GNU archive every week, you would use a

1983

command like the following, weekly:

1984

1985

@example

1986

wget --timestamping -r ftp://ftp.gnu.org/pub/gnu/

1987

@end example

1988

1989

Note that time-stamping will only work for files for which the server

1990

gives a timestamp. For @sc{http}, this depends on getting a

1991

@code{Last-Modified} header. For @sc{ftp}, this depends on getting a

1992

directory listing with dates in a format that Wget can parse

1993

(@pxref{FTP Time-Stamping Internals}).

1994

1995

@node HTTP Time-Stamping Internals, FTP Time-Stamping Internals, Time-Stamping Usage, Time-Stamping

1996

@section HTTP Time-Stamping Internals

1997

@cindex http time-stamping

1998

1999

Time-stamping in @sc{http} is implemented by checking of the

2000

@code{Last-Modified} header. If you wish to retrieve the file

2001

@file{foo.html} through @sc{http}, Wget will check whether

2002

@file{foo.html} exists locally. If it doesn't, @file{foo.html} will be

2003

retrieved unconditionally.

2004

2005

If the file does exist locally, Wget will first check its local

2006

time-stamp (similar to the way @code{ls -l} checks it), and then send a

2007

@code{HEAD} request to the remote server, demanding the information on

2008

the remote file.

2009

2010

The @code{Last-Modified} header is examined to find which file was

2011

modified more recently (which makes it ``newer''). If the remote file

2012

is newer, it will be downloaded; if it is older, Wget will give

2013

up.@footnote{As an additional check, Wget will look at the

2014

@code{Content-Length} header, and compare the sizes; if they are not the

2015

same, the remote file will be downloaded no matter what the time-stamp

2016

says.}

2017

2018

When @samp{--backup-converted} (@samp{-K}) is specified in conjunction

2019

with @samp{-N}, server file @samp{@var{X}} is compared to local file

2020

@samp{@var{X}.orig}, if extant, rather than being compared to local file

2021

@samp{@var{X}}, which will always differ if it's been converted by

2022

@samp{--convert-links} (@samp{-k}).

2023

2024

Arguably, @sc{http} time-stamping should be implemented using the

2025

@code{If-Modified-Since} request.

2026

2027

@node FTP Time-Stamping Internals, , HTTP Time-Stamping Internals, Time-Stamping

2028

@section FTP Time-Stamping Internals

2029

@cindex ftp time-stamping

2030

2031

In theory, @sc{ftp} time-stamping works much the same as @sc{http}, only

2032

@sc{ftp} has no headers---time-stamps must be ferreted out of directory

2033

listings.

2034

2035

If an @sc{ftp} download is recursive or uses globbing, Wget will use the

2036

@sc{ftp} @code{LIST} command to get a file listing for the directory

2037

containing the desired file(s). It will try to analyze the listing,

2038

treating it like Unix @code{ls -l} output, extracting the time-stamps.

2039

The rest is exactly the same as for @sc{http}. Note that when

2040

retrieving individual files from an @sc{ftp} server without using

2041

globbing or recursion, listing files will not be downloaded (and thus

2042

files will not be time-stamped) unless @samp{-N} is specified.

2043

2044

Assumption that every directory listing is a Unix-style listing may

2045

sound extremely constraining, but in practice it is not, as many

2046

non-Unix @sc{ftp} servers use the Unixoid listing format because most

2047

(all?) of the clients understand it. Bear in mind that @sc{rfc959}

2048

defines no standard way to get a file list, let alone the time-stamps.

2049

We can only hope that a future standard will define this.

2050

2051

Another non-standard solution includes the use of @code{MDTM} command

2052

that is supported by some @sc{ftp} servers (including the popular

2053

@code{wu-ftpd}), which returns the exact time of the specified file.

2054

Wget may support this command in the future.

2055

2056

@node Startup File, Examples, Time-Stamping, Top

2057

@chapter Startup File

2058

@cindex startup file

2059

@cindex wgetrc

2060

@cindex .wgetrc

2061

@cindex startup

2062

@cindex .netrc

2063

2064

Once you know how to change default settings of Wget through command

2065

line arguments, you may wish to make some of those settings permanent.

2066

You can do that in a convenient way by creating the Wget startup

2067

file---@file{.wgetrc}.

2068

2069

Besides @file{.wgetrc} is the ``main'' initialization file, it is

2070

convenient to have a special facility for storing passwords. Thus Wget

2071

reads and interprets the contents of @file{$HOME/.netrc}, if it finds

2072

it. You can find @file{.netrc} format in your system manuals.

2073

2074

Wget reads @file{.wgetrc} upon startup, recognizing a limited set of

2075

commands.

2076

2077

@menu

2078

* Wgetrc Location:: Location of various wgetrc files.

2079

* Wgetrc Syntax:: Syntax of wgetrc.

2080

* Wgetrc Commands:: List of available commands.

2081

* Sample Wgetrc:: A wgetrc example.

2082

@end menu

2083

2084

@node Wgetrc Location, Wgetrc Syntax, Startup File, Startup File

2085

@section Wgetrc Location

2086

@cindex wgetrc location

2087

@cindex location of wgetrc

2088

2089

When initializing, Wget will look for a @dfn{global} startup file,

2090

@file{/usr/local/etc/wgetrc} by default (or some prefix other than

2091

@file{/usr/local}, if Wget was not installed there) and read commands

2092

from there, if it exists.

2093

2094

Then it will look for the user's file. If the environmental variable

2095

@code{WGETRC} is set, Wget will try to load that file. Failing that, no

2096

further attempts will be made.

2097

2098

If @code{WGETRC} is not set, Wget will try to load @file{$HOME/.wgetrc}.

2099

2100

The fact that user's settings are loaded after the system-wide ones

2101

means that in case of collision user's wgetrc @emph{overrides} the

2102

system-wide wgetrc (in @file{/usr/local/etc/wgetrc} by default).

2103

Fascist admins, away!

2104

2105

@node Wgetrc Syntax, Wgetrc Commands, Wgetrc Location, Startup File

2106

@section Wgetrc Syntax

2107

@cindex wgetrc syntax

2108

@cindex syntax of wgetrc

2109

2110

The syntax of a wgetrc command is simple:

2111

2112

@example

2113

variable = value

2114

@end example

2115

2116

The @dfn{variable} will also be called @dfn{command}. Valid

2117

@dfn{values} are different for different commands.

2118

2119

The commands are case-insensitive and underscore-insensitive. Thus

2120

@samp{DIr__PrefiX} is the same as @samp{dirprefix}. Empty lines, lines

2121

beginning with @samp{#} and lines containing white-space only are

2122

discarded.

2123

2124

Commands that expect a comma-separated list will clear the list on an

2125

empty command. So, if you wish to reset the rejection list specified in

2126

global @file{wgetrc}, you can do it with:

2127

2128

@example

2129

reject =

2130

@end example

2131

2132

@node Wgetrc Commands, Sample Wgetrc, Wgetrc Syntax, Startup File

2133

@section Wgetrc Commands

2134

@cindex wgetrc commands

2135

2136

The complete set of commands is listed below. Legal values are listed

2137

after the @samp{=}. Simple Boolean values can be set or unset using

2138

@samp{on} and @samp{off} or @samp{1} and @samp{0}. A fancier kind of

2139

Boolean allowed in some cases is the @dfn{lockable Boolean}, which may

2140

be set to @samp{on}, @samp{off}, @samp{always}, or @samp{never}. If an

2141

option is set to @samp{always} or @samp{never}, that value will be

2142

locked in for the duration of the Wget invocation---command-line options

2143

will not override.

2144

2145

Some commands take pseudo-arbitrary values. @var{address} values can be

2146

hostnames or dotted-quad IP addresses. @var{n} can be any positive

2147

integer, or @samp{inf} for infinity, where appropriate. @var{string}

2148

values can be any non-empty string.

2149

2150

Most of these commands have command-line equivalents (@pxref{Invoking}),

2151

though some of the more obscure or rarely used ones do not.

2152

2153

@table @asis

2154

@item accept/reject = @var{string}

2155

Same as @samp{-A}/@samp{-R} (@pxref{Types of Files}).

2156

2157

@item add_hostdir = on/off

2158

Enable/disable host-prefixed file names. @samp{-nH} disables it.

2159

2160

@item continue = on/off

2161

If set to on, force continuation of preexistent partially retrieved

2162

files. See @samp{-c} before setting it.

2163

2164

@item background = on/off

2165

Enable/disable going to background---the same as @samp{-b} (which

2166

enables it).

2167

2168

@item backup_converted = on/off

2169

Enable/disable saving pre-converted files with the suffix

2170

@samp{.orig}---the same as @samp{-K} (which enables it).

2171

2172

@c @item backups = @var{number}

2173

@c #### Document me!

2174

@c

2175

@item base = @var{string}

2176

Consider relative @sc{url}s in @sc{url} input files forced to be

2177

interpreted as @sc{html} as being relative to @var{string}---the same as

2178

@samp{-B}.

2179

2180

@item bind_address = @var{address}

2181

Bind to @var{address}, like the @samp{--bind-address} option.

2182

2183

@item cache = on/off

2184

When set to off, disallow server-caching. See the @samp{-C} option.

2185

2186

@item convert_links = on/off

2187

Convert non-relative links locally. The same as @samp{-k}.

2188

2189

@item cookies = on/off

2190

When set to off, disallow cookies. See the @samp{--cookies} option.

2191

2192

@item load_cookies = @var{file}

2193

Load cookies from @var{file}. See @samp{--load-cookies}.

2194

2195

@item save_cookies = @var{file}

2196

Save cookies to @var{file}. See @samp{--save-cookies}.

2197

2198

@item connect_timeout = @var{n}

2199

Set the connect timeout---the same as @samp{--connect-timeout}.

2200

2201

@item cut_dirs = @var{n}

2202

Ignore @var{n} remote directory components.

2203

2204

@item debug = on/off

2205

Debug mode, same as @samp{-d}.

2206

2207

@item delete_after = on/off

2208

Delete after download---the same as @samp{--delete-after}.

2209

2210

@item dir_prefix = @var{string}

2211

Top of directory tree---the same as @samp{-P}.

2212

2213

@item dirstruct = on/off

2214

Turning dirstruct on or off---the same as @samp{-x} or @samp{-nd},

2215

respectively.

2216

2217

@item dns_cache = on/off

2218

Turn DNS caching on/off. Since DNS caching is on by default, this

2219

option is normally used to turn it off. Same as @samp{--dns-cache}.

2220

2221

@item dns_timeout = @var{n}

2222

Set the DNS timeout---the same as @samp{--dns-timeout}.

2223

2224

@item domains = @var{string}

2225

Same as @samp{-D} (@pxref{Spanning Hosts}).

2226

2227

@item dot_bytes = @var{n}

2228

Specify the number of bytes ``contained'' in a dot, as seen throughout

2229

the retrieval (1024 by default). You can postfix the value with

2230

@samp{k} or @samp{m}, representing kilobytes and megabytes,

2231

respectively. With dot settings you can tailor the dot retrieval to

2232

suit your needs, or you can use the predefined @dfn{styles}

2233

(@pxref{Download Options}).

2234

2235

@item dots_in_line = @var{n}

2236

Specify the number of dots that will be printed in each line throughout

2237

the retrieval (50 by default).

2238

2239

@item dot_spacing = @var{n}

2240

Specify the number of dots in a single cluster (10 by default).

2241

2242

@item exclude_directories = @var{string}

2243

Specify a comma-separated list of directories you wish to exclude from

2244

download---the same as @samp{-X} (@pxref{Directory-Based Limits}).

2245

2246

@item exclude_domains = @var{string}

2247

Same as @samp{--exclude-domains} (@pxref{Spanning Hosts}).

2248

2249

@item follow_ftp = on/off

2250

Follow @sc{ftp} links from @sc{html} documents---the same as

2251

@samp{--follow-ftp}.

2252

2253

@item follow_tags = @var{string}

2254

Only follow certain @sc{html} tags when doing a recursive retrieval, just like

2255

@samp{--follow-tags}.

2256

2257

@item force_html = on/off

2258

If set to on, force the input filename to be regarded as an @sc{html}

2259

document---the same as @samp{-F}.

2260

2261

@item ftp_proxy = @var{string}

2262

Use @var{string} as @sc{ftp} proxy, instead of the one specified in

2263

environment.

2264

2265

@item glob = on/off

2266

Turn globbing on/off---the same as @samp{-g}.

2267

2268

@item header = @var{string}

2269

Define an additional header, like @samp{--header}.

2270

2271

@item html_extension = on/off

2272

Add a @samp{.html} extension to @samp{text/html} or

2273

@samp{application/xhtml+xml} files without it, like

2274

@samp{-E}.

2275

2276

@item http_passwd = @var{string}

2277

Set @sc{http} password.

2278

2279

@item http_proxy = @var{string}

2280

Use @var{string} as @sc{http} proxy, instead of the one specified in

2281

environment.

2282

2283

@item http_user = @var{string}

2284

Set @sc{http} user to @var{string}.

2285

2286

@item ignore_length = on/off

2287

When set to on, ignore @code{Content-Length} header; the same as

2288

@samp{--ignore-length}.

2289

2290

@item ignore_tags = @var{string}

2291

Ignore certain @sc{html} tags when doing a recursive retrieval, just like

2292

@samp{-G} / @samp{--ignore-tags}.

2293

2294

@item include_directories = @var{string}

2295

Specify a comma-separated list of directories you wish to follow when

2296

downloading---the same as @samp{-I}.

2297

2298

@item input = @var{string}

2299

Read the @sc{url}s from @var{string}, like @samp{-i}.

2300

2301

@item kill_longer = on/off

2302

Consider data longer than specified in content-length header as invalid

2303

(and retry getting it). The default behavior is to save as much data

2304

as there is, provided there is more than or equal to the value in

2305

@code{Content-Length}.

2306

2307

@item limit_rate = @var{rate}

2308

Limit the download speed to no more than @var{rate} bytes per second.

2309

The same as @samp{--limit-rate}.

2310

2311

@item logfile = @var{string}

2312

Set logfile---the same as @samp{-o}.

2313

2314

@item login = @var{string}

2315

Your user name on the remote machine, for @sc{ftp}. Defaults to

2316

@samp{anonymous}.

2317

2318

@item mirror = on/off

2319

Turn mirroring on/off. The same as @samp{-m}.

2320

2321

@item netrc = on/off

2322

Turn reading netrc on or off.

2323

2324

@item noclobber = on/off

2325

Same as @samp{-nc}.

2326

2327

@item no_parent = on/off

2328

Disallow retrieving outside the directory hierarchy, like

2329

@samp{--no-parent} (@pxref{Directory-Based Limits}).

2330

2331

@item no_proxy = @var{string}

2332

Use @var{string} as the comma-separated list of domains to avoid in

2333

proxy loading, instead of the one specified in environment.

2334

2335

@item output_document = @var{string}

2336

Set the output filename---the same as @samp{-O}.

2337

2338

@item page_requisites = on/off

2339

Download all ancillary documents necessary for a single @sc{html} page to

2340

display properly---the same as @samp{-p}.

2341

2342

@item passive_ftp = on/off/always/never

2343

Set passive @sc{ftp}---the same as @samp{--passive-ftp}. Some scripts

2344

and @samp{.pm} (Perl module) files download files using @samp{wget

2345

--passive-ftp}. If your firewall does not allow this, you can set

2346

@samp{passive_ftp = never} to override the command-line.

2347

2348

@item passwd = @var{string}

2349

Set your @sc{ftp} password to @var{password}. Without this setting, the

2350

password defaults to @samp{username@@hostname.domainname}.

2351

2352

@item post_data = @var{string}

2353

Use POST as the method for all HTTP requests and send @var{string} in

2354

the request body. The same as @samp{--post-data}.

2355

2356

@item post_file = @var{file}

2357

Use POST as the method for all HTTP requests and send the contents of

2358

@var{file} in the request body. The same as @samp{--post-file}.

2359

2360

@item progress = @var{string}

2361

Set the type of the progress indicator. Legal types are ``dot'' and

2362

``bar''.

2363

2364

@item proxy_user = @var{string}

2365

Set proxy authentication user name to @var{string}, like @samp{--proxy-user}.

2366

2367

@item proxy_passwd = @var{string}

2368

Set proxy authentication password to @var{string}, like @samp{--proxy-passwd}.

2369

2370

@item referer = @var{string}

2371

Set HTTP @samp{Referer:} header just like @samp{--referer}. (Note it

2372

was the folks who wrote the @sc{http} spec who got the spelling of

2373

``referrer'' wrong.)

2374

2375

@item quiet = on/off

2376

Quiet mode---the same as @samp{-q}.

2377

2378

@item quota = @var{quota}

2379

Specify the download quota, which is useful to put in the global

2380

@file{wgetrc}. When download quota is specified, Wget will stop

2381

retrieving after the download sum has become greater than quota. The

2382

quota can be specified in bytes (default), kbytes @samp{k} appended) or

2383

mbytes (@samp{m} appended). Thus @samp{quota = 5m} will set the quota

2384

to 5 mbytes. Note that the user's startup file overrides system

2385

settings.

2386

2387

@item read_timeout = @var{n}

2388

Set the read (and write) timeout---the same as @samp{--read-timeout}.

2389

2390

@item reclevel = @var{n}

2391

Recursion level---the same as @samp{-l}.

2392

2393

@item recursive = on/off

2394

Recursive on/off---the same as @samp{-r}.

2395

2396

@item relative_only = on/off

2397

Follow only relative links---the same as @samp{-L} (@pxref{Relative

2398

Links}).

2399

2400

@item remove_listing = on/off

2401

If set to on, remove @sc{ftp} listings downloaded by Wget. Setting it

2402

to off is the same as @samp{-nr}.

2403

2404

@item restrict_file_names = unix/windows

2405

Restrict the file names generated by Wget from URLs. See

2406

@samp{--restrict-file-names} for a more detailed description.

2407

2408

@item retr_symlinks = on/off

2409

When set to on, retrieve symbolic links as if they were plain files; the

2410

same as @samp{--retr-symlinks}.

2411

2412

@item robots = on/off

2413

Specify whether the norobots convention is respected by Wget, ``on'' by

2414

default. This switch controls both the @file{/robots.txt} and the

2415

@samp{nofollow} aspect of the spec. @xref{Robot Exclusion}, for more

2416

details about this. Be sure you know what you are doing before turning

2417

this off.

2418

2419

@item server_response = on/off

2420

Choose whether or not to print the @sc{http} and @sc{ftp} server

2421

responses---the same as @samp{-S}.

2422

2423

@item span_hosts = on/off

2424

Same as @samp{-H}.

2425

2426

@item strict_comments = on/off

2427

Same as @samp{--strict-comments}.

2428

2429

@item timeout = @var{n}

2430

Set timeout value---the same as @samp{-T}.

2431

2432

@item timestamping = on/off

2433

Turn timestamping on/off. The same as @samp{-N} (@pxref{Time-Stamping}).

2434

2435

@item tries = @var{n}

2436

Set number of retries per @sc{url}---the same as @samp{-t}.

2437

2438

@item use_proxy = on/off

2439

Turn proxy support on/off. The same as @samp{-Y}.

2440

2441

@item verbose = on/off

2442

Turn verbose on/off---the same as @samp{-v}/@samp{-nv}.

2443

2444

@item wait = @var{n}

2445

Wait @var{n} seconds between retrievals---the same as @samp{-w}.

2446

2447

@item waitretry = @var{n}

2448

Wait up to @var{n} seconds between retries of failed retrievals

2449

only---the same as @samp{--waitretry}. Note that this is turned on by

2450

default in the global @file{wgetrc}.

2451

2452

@item randomwait = on/off

2453

Turn random between-request wait times on or off. The same as

2454

@samp{--random-wait}.

2455

@end table

2456

2457

@node Sample Wgetrc, , Wgetrc Commands, Startup File

2458

@section Sample Wgetrc

2459

@cindex sample wgetrc

2460

2461

This is the sample initialization file, as given in the distribution.

2462

It is divided in two section---one for global usage (suitable for global

2463

startup file), and one for local usage (suitable for

2464

@file{$HOME/.wgetrc}). Be careful about the things you change.

2465

2466

Note that almost all the lines are commented out. For a command to have

2467

any effect, you must remove the @samp{#} character at the beginning of

2468

its line.

2469

2470

@example

2471

@include sample.wgetrc.munged_for_texi_inclusion

2472

@end example

2473

2474

@node Examples, Various, Startup File, Top

2475

@chapter Examples

2476

@cindex examples

2477

2478

@c man begin EXAMPLES

2479

The examples are divided into three sections loosely based on their

2480

complexity.

2481

2482

@menu

2483

* Simple Usage:: Simple, basic usage of the program.

2484

* Advanced Usage:: Advanced tips.

2485

* Very Advanced Usage:: The hairy stuff.

2486

@end menu

2487

2488

@node Simple Usage, Advanced Usage, Examples, Examples

2489

@section Simple Usage

2490

2491

@itemize @bullet

2492

@item

2493

Say you want to download a @sc{url}. Just type:

2494

2495

@example

2496

wget http://fly.srk.fer.hr/

2497

@end example

2498

2499

@item

2500

But what will happen if the connection is slow, and the file is lengthy?

2501

The connection will probably fail before the whole file is retrieved,

2502

more than once. In this case, Wget will try getting the file until it

2503

either gets the whole of it, or exceeds the default number of retries

2504

(this being 20). It is easy to change the number of tries to 45, to

2505

insure that the whole file will arrive safely:

2506

2507

@example

2508

wget --tries=45 http://fly.srk.fer.hr/jpg/flyweb.jpg

2509

@end example

2510

2511

@item

2512

Now let's leave Wget to work in the background, and write its progress

2513

to log file @file{log}. It is tiring to type @samp{--tries}, so we

2514

shall use @samp{-t}.

2515

2516

@example

2517

wget -t 45 -o log http://fly.srk.fer.hr/jpg/flyweb.jpg &

2518

@end example

2519

2520

The ampersand at the end of the line makes sure that Wget works in the

2521

background. To unlimit the number of retries, use @samp{-t inf}.

2522

2523

@item

2524

The usage of @sc{ftp} is as simple. Wget will take care of login and

2525

password.

2526

2527

@example

2528

wget ftp://gnjilux.srk.fer.hr/welcome.msg

2529

@end example

2530

2531

@item

2532

If you specify a directory, Wget will retrieve the directory listing,

2533

parse it and convert it to @sc{html}. Try:

2534

2535

@example

2536

wget ftp://ftp.gnu.org/pub/gnu/

2537

links index.html

2538

@end example

2539

@end itemize

2540

2541

@node Advanced Usage, Very Advanced Usage, Simple Usage, Examples

2542

@section Advanced Usage

2543

2544

@itemize @bullet

2545

@item

2546

You have a file that contains the URLs you want to download? Use the

2547

@samp{-i} switch:

2548

2549

@example

2550

wget -i @var{file}

2551

@end example

2552

2553

If you specify @samp{-} as file name, the @sc{url}s will be read from

2554

standard input.

2555

2556

@item

2557

Create a five levels deep mirror image of the GNU web site, with the

2558

same directory structure the original has, with only one try per

2559

document, saving the log of the activities to @file{gnulog}:

2560

2561

@example

2562

wget -r http://www.gnu.org/ -o gnulog

2563

@end example

2564

2565

@item

2566

The same as the above, but convert the links in the @sc{html} files to

2567

point to local files, so you can view the documents off-line:

2568

2569

@example

2570

wget --convert-links -r http://www.gnu.org/ -o gnulog

2571

@end example

2572

2573

@item

2574

Retrieve only one @sc{html} page, but make sure that all the elements needed

2575

for the page to be displayed, such as inline images and external style

2576

sheets, are also downloaded. Also make sure the downloaded page

2577

references the downloaded links.

2578

2579

@example

2580

wget -p --convert-links http://www.server.com/dir/page.html

2581

@end example

2582

2583

The @sc{html} page will be saved to @file{www.server.com/dir/page.html}, and

2584

the images, stylesheets, etc., somewhere under @file{www.server.com/},

2585

depending on where they were on the remote server.

2586

2587

@item

2588

The same as the above, but without the @file{www.server.com/} directory.

2589

In fact, I don't want to have all those random server directories

2590

anyway---just save @emph{all} those files under a @file{download/}

2591

subdirectory of the current directory.

2592

2593

@example

2594

wget -p --convert-links -nH -nd -Pdownload \

2595

http://www.server.com/dir/page.html

2596

@end example

2597

2598

@item

2599

Retrieve the index.html of @samp{www.lycos.com}, showing the original

2600

server headers:

2601

2602

@example

2603

wget -S http://www.lycos.com/

2604

@end example

2605

2606

@item

2607

Save the server headers with the file, perhaps for post-processing.

2608

2609

@example

2610

wget -s http://www.lycos.com/

2611

more index.html

2612

@end example

2613

2614

@item

2615

Retrieve the first two levels of @samp{wuarchive.wustl.edu}, saving them

2616

to @file{/tmp}.

2617

2618

@example

2619

wget -r -l2 -P/tmp ftp://wuarchive.wustl.edu/

2620

@end example

2621

2622

@item

2623

You want to download all the @sc{gif}s from a directory on an @sc{http}

2624

server. You tried @samp{wget http://www.server.com/dir/*.gif}, but that

2625

didn't work because @sc{http} retrieval does not support globbing. In

2626

that case, use:

2627

2628

@example

2629

wget -r -l1 --no-parent -A.gif http://www.server.com/dir/

2630

@end example

2631

2632

More verbose, but the effect is the same. @samp{-r -l1} means to

2633

retrieve recursively (@pxref{Recursive Retrieval}), with maximum depth

2634

of 1. @samp{--no-parent} means that references to the parent directory

2635

are ignored (@pxref{Directory-Based Limits}), and @samp{-A.gif} means to

2636

download only the @sc{gif} files. @samp{-A "*.gif"} would have worked

2637

too.

2638

2639

@item

2640

Suppose you were in the middle of downloading, when Wget was

2641

interrupted. Now you do not want to clobber the files already present.

2642

It would be:

2643

2644

@example

2645

wget -nc -r http://www.gnu.org/

2646

@end example

2647

2648

@item

2649

If you want to encode your own username and password to @sc{http} or

2650

@sc{ftp}, use the appropriate @sc{url} syntax (@pxref{URL Format}).

2651

2652

@example

2653

wget ftp://hniksic:mypassword@@unix.server.com/.emacs

2654

@end example

2655

2656

Note, however, that this usage is not advisable on multi-user systems

2657

because it reveals your password to anyone who looks at the output of

2658

@code{ps}.

2659

2660

@cindex redirecting output

2661

@item

2662

You would like the output documents to go to standard output instead of

2663

to files?

2664

2665

@example

2666

wget -O - http://jagor.srce.hr/ http://www.srce.hr/

2667

@end example

2668

2669

You can also combine the two options and make pipelines to retrieve the

2670

documents from remote hotlists:

2671

2672

@example

2673

wget -O - http://cool.list.com/ | wget --force-html -i -

2674

@end example

2675

@end itemize

2676

2677

@node Very Advanced Usage, , Advanced Usage, Examples

2678

@section Very Advanced Usage

2679

2680

@cindex mirroring

2681

@itemize @bullet

2682

@item

2683

If you wish Wget to keep a mirror of a page (or @sc{ftp}

2684

subdirectories), use @samp{--mirror} (@samp{-m}), which is the shorthand

2685

for @samp{-r -l inf -N}. You can put Wget in the crontab file asking it

2686

to recheck a site each Sunday:

2687

2688

@example

2689

crontab

2690

0 0 * * 0 wget --mirror http://www.gnu.org/ -o /home/me/weeklog

2691

@end example

2692

2693

@item

2694

In addition to the above, you want the links to be converted for local

2695

viewing. But, after having read this manual, you know that link

2696

conversion doesn't play well with timestamping, so you also want Wget to

2697

back up the original @sc{html} files before the conversion. Wget invocation

2698

would look like this:

2699

2700

@example

2701

wget --mirror --convert-links --backup-converted \

2702

http://www.gnu.org/ -o /home/me/weeklog

2703

@end example

2704

2705

@item

2706

But you've also noticed that local viewing doesn't work all that well

2707

when @sc{html} files are saved under extensions other than @samp{.html},

2708

perhaps because they were served as @file{index.cgi}. So you'd like

2709

Wget to rename all the files served with content-type @samp{text/html}

2710

or @samp{application/xhtml+xml} to @file{@var{name}.html}.

2711

2712

@example

2713

wget --mirror --convert-links --backup-converted \

2714

--html-extension -o /home/me/weeklog \

2715

http://www.gnu.org/

2716

@end example

2717

2718

Or, with less typing:

2719

2720

@example

2721

wget -m -k -K -E http://www.gnu.org/ -o /home/me/weeklog

2722

@end example

2723

@end itemize

2724

@c man end

2725

2726

@node Various, Appendices, Examples, Top

2727

@chapter Various

2728

@cindex various

2729

2730

This chapter contains all the stuff that could not fit anywhere else.

2731

2732

@menu

2733

* Proxies:: Support for proxy servers

2734

* Distribution:: Getting the latest version.

2735

* Mailing List:: Wget mailing list for announcements and discussion.

2736

* Reporting Bugs:: How and where to report bugs.

2737

* Portability:: The systems Wget works on.

2738

* Signals:: Signal-handling performed by Wget.

2739

@end menu

2740

2741

@node Proxies, Distribution, Various, Various

2742

@section Proxies

2743

@cindex proxies

2744

2745

@dfn{Proxies} are special-purpose @sc{http} servers designed to transfer

2746

data from remote servers to local clients. One typical use of proxies

2747

is lightening network load for users behind a slow connection. This is

2748

achieved by channeling all @sc{http} and @sc{ftp} requests through the

2749

proxy which caches the transferred data. When a cached resource is

2750

requested again, proxy will return the data from cache. Another use for

2751

proxies is for companies that separate (for security reasons) their

2752

internal networks from the rest of Internet. In order to obtain

2753

information from the Web, their users connect and retrieve remote data

2754

using an authorized proxy.

2755

2756

Wget supports proxies for both @sc{http} and @sc{ftp} retrievals. The

2757

standard way to specify proxy location, which Wget recognizes, is using

2758

the following environment variables:

2759

2760

@table @code

2761

@item http_proxy

2762

This variable should contain the @sc{url} of the proxy for @sc{http}

2763

connections.

2764

2765

@item ftp_proxy

2766

This variable should contain the @sc{url} of the proxy for @sc{ftp}

2767

connections. It is quite common that @sc{http_proxy} and @sc{ftp_proxy}

2768

are set to the same @sc{url}.

2769

2770

@item no_proxy

2771

This variable should contain a comma-separated list of domain extensions

2772

proxy should @emph{not} be used for. For instance, if the value of

2773

@code{no_proxy} is @samp{.mit.edu}, proxy will not be used to retrieve

2774

documents from MIT.

2775

@end table

2776

2777

In addition to the environment variables, proxy location and settings

2778

may be specified from within Wget itself.

2779

2780

@table @samp

2781

@item -Y on/off

2782

@itemx --proxy=on/off

2783

@itemx proxy = on/off

2784

This option may be used to turn the proxy support on or off. Proxy

2785

support is on by default, provided that the appropriate environment

2786

variables are set.

2787

2788

@item http_proxy = @var{URL}

2789

@itemx ftp_proxy = @var{URL}

2790

@itemx no_proxy = @var{string}

2791

These startup file variables allow you to override the proxy settings

2792

specified by the environment.

2793

@end table

2794

2795

Some proxy servers require authorization to enable you to use them. The

2796

authorization consists of @dfn{username} and @dfn{password}, which must

2797

be sent by Wget. As with @sc{http} authorization, several

2798

authentication schemes exist. For proxy authorization only the

2799

@code{Basic} authentication scheme is currently implemented.

2800

2801

You may specify your username and password either through the proxy

2802

@sc{url} or through the command-line options. Assuming that the

2803

company's proxy is located at @samp{proxy.company.com} at port 8001, a

2804

proxy @sc{url} location containing authorization data might look like

2805

this:

2806

2807

@example

2808

http://hniksic:mypassword@@proxy.company.com:8001/

2809

@end example

2810

2811

Alternatively, you may use the @samp{proxy-user} and

2812

@samp{proxy-password} options, and the equivalent @file{.wgetrc}

2813

settings @code{proxy_user} and @code{proxy_passwd} to set the proxy

2814

username and password.

2815

2816

@node Distribution, Mailing List, Proxies, Various

2817

@section Distribution

2818

@cindex latest version

2819

2820

Like all GNU utilities, the latest version of Wget can be found at the

2821

master GNU archive site ftp.gnu.org, and its mirrors. For example,

2822

Wget @value{VERSION} can be found at

2823

@url{ftp://ftp.gnu.org/pub/gnu/wget/wget-@value{VERSION}.tar.gz}

2824

2825

@node Mailing List, Reporting Bugs, Distribution, Various

2826

@section Mailing List

2827

@cindex mailing list

2828

@cindex list

2829

2830

Wget has its own mailing list at @email{wget@@sunsite.dk}, thanks

2831

to Karsten Thygesen. The mailing list is for discussion of Wget

2832

features and web, reporting Wget bugs (those that you think may be of

2833

interest to the public) and mailing announcements. You are welcome to

2834

subscribe. The more people on the list, the better!

2835

2836

To subscribe, simply send mail to @email{wget-subscribe@@sunsite.dk}.

2837

Unsubscribe by mailing to @email{wget-unsubscribe@@sunsite.dk}.

2838

2839

The mailing list is archived at @url{http://fly.srk.fer.hr/archive/wget}.

2840

Alternative archive is available at

2841

@url{http://www.mail-archive.com/wget%40sunsite.auc.dk/}.

2842

2843

@node Reporting Bugs, Portability, Mailing List, Various

2844

@section Reporting Bugs

2845

@cindex bugs

2846

@cindex reporting bugs

2847

@cindex bug reports

2848

2849

@c man begin BUGS

2850

You are welcome to send bug reports about GNU Wget to

2851

@email{bug-wget@@gnu.org}.

2852

2853

Before actually submitting a bug report, please try to follow a few

2854

simple guidelines.

2855

2856

@enumerate

2857

@item

2858

Please try to ascertain that the behavior you see really is a bug. If

2859

Wget crashes, it's a bug. If Wget does not behave as documented,

2860

it's a bug. If things work strange, but you are not sure about the way

2861

they are supposed to work, it might well be a bug.

2862

2863

@item

2864

Try to repeat the bug in as simple circumstances as possible. E.g. if

2865

Wget crashes while downloading @samp{wget -rl0 -kKE -t5 -Y0

2866

http://yoyodyne.com -o /tmp/log}, you should try to see if the crash is

2867

repeatable, and if will occur with a simpler set of options. You might

2868

even try to start the download at the page where the crash occurred to

2869

see if that page somehow triggered the crash.

2870

2871

Also, while I will probably be interested to know the contents of your

2872

@file{.wgetrc} file, just dumping it into the debug message is probably

2873

a bad idea. Instead, you should first try to see if the bug repeats

2874

with @file{.wgetrc} moved out of the way. Only if it turns out that

2875

@file{.wgetrc} settings affect the bug, mail me the relevant parts of

2876

the file.

2877

2878

@item

2879

Please start Wget with @samp{-d} option and send the log (or the

2880

relevant parts of it). If Wget was compiled without debug support,

2881

recompile it. It is @emph{much} easier to trace bugs with debug support

2882

on.

2883

2884

@item

2885

If Wget has crashed, try to run it in a debugger, e.g. @code{gdb `which

2886

wget` core} and type @code{where} to get the backtrace.

2887

@end enumerate

2888

@c man end

2889

2890

@node Portability, Signals, Reporting Bugs, Various

2891

@section Portability

2892

@cindex portability

2893

@cindex operating systems

2894

2895

Since Wget uses GNU Autoconf for building and configuring, and avoids

2896

using ``special'' ultra--mega--cool features of any particular Unix, it

2897

should compile (and work) on all common Unix flavors.

2898

2899

Various Wget versions have been compiled and tested under many kinds of

2900

Unix systems, including Solaris, Linux, SunOS, OSF (aka Digital Unix),

2901

Ultrix, *BSD, IRIX, and others; refer to the file @file{MACHINES} in the

2902

distribution directory for a comprehensive list. If you compile it on

2903

an architecture not listed there, please let me know so I can update it.

2904

2905

Wget should also compile on the other Unix systems, not listed in

2906

@file{MACHINES}. If it doesn't, please let me know.

2907

2908

Thanks to kind contributors, this version of Wget compiles and works on

2909

Microsoft Windows 95 and Windows NT platforms. It has been compiled

2910

successfully using MS Visual C++ 4.0, Watcom, and Borland C compilers,

2911

with Winsock as networking software. Naturally, it is crippled of some

2912

features available on Unix, but it should work as a substitute for

2913

people stuck with Windows. Note that the Windows port is

2914

@strong{neither tested nor maintained} by me---all questions and

2915

problems should be reported to Wget mailing list at

2916

@email{wget@@sunsite.dk} where the maintainers will look at them.

2917

2918

@node Signals, , Portability, Various

2919

@section Signals

2920

@cindex signal handling

2921

@cindex hangup

2922

2923

Since the purpose of Wget is background work, it catches the hangup

2924

signal (@code{SIGHUP}) and ignores it. If the output was on standard

2925

output, it will be redirected to a file named @file{wget-log}.

2926

Otherwise, @code{SIGHUP} is ignored. This is convenient when you wish

2927

to redirect the output of Wget after having started it.

2928

2929

@example

2930

$ wget http://www.ifi.uio.no/~larsi/gnus.tar.gz &

2931

$ kill -HUP %% # Redirect the output to wget-log

2932

@end example

2933

2934

Other than that, Wget will not try to interfere with signals in any way.

2935

@kbd{C-c}, @code{kill -TERM} and @code{kill -KILL} should kill it alike.

2936

2937

@node Appendices, Copying, Various, Top

2938

@chapter Appendices

2939

2940

This chapter contains some references I consider useful.

2941

2942

@menu

2943

* Robot Exclusion:: Wget's support for RES.

2944

* Security Considerations:: Security with Wget.

2945

* Contributors:: People who helped.

2946

@end menu

2947

2948

@node Robot Exclusion, Security Considerations, Appendices, Appendices

2949

@section Robot Exclusion

2950

@cindex robot exclusion

2951

@cindex robots.txt

2952

@cindex server maintenance

2953

2954

It is extremely easy to make Wget wander aimlessly around a web site,

2955

sucking all the available data in progress. @samp{wget -r @var{site}},

2956

and you're set. Great? Not for the server admin.

2957

2958

As long as Wget is only retrieving static pages, and doing it at a

2959

reasonable rate (see the @samp{--wait} option), there's not much of a

2960

problem. The trouble is that Wget can't tell the difference between the

2961

smallest static page and the most demanding CGI. A site I know has a

2962

section handled by a CGI Perl script that converts Info files to @sc{html} on

2963

the fly. The script is slow, but works well enough for human users

2964

viewing an occasional Info file. However, when someone's recursive Wget

2965

download stumbles upon the index page that links to all the Info files

2966

through the script, the system is brought to its knees without providing

2967

anything useful to the user (This task of converting Info files could be

2968

done locally and access to Info documentation for all installed GNU

2969

software on a system is available from the @code{info} command).

2970

2971

To avoid this kind of accident, as well as to preserve privacy for

2972

documents that need to be protected from well-behaved robots, the

2973

concept of @dfn{robot exclusion} was invented. The idea is that

2974

the server administrators and document authors can specify which

2975

portions of the site they wish to protect from robots and those

2976

they will permit access.

2977

2978

The most popular mechanism, and the @i{de facto} standard supported by

2979

all the major robots, is the ``Robots Exclusion Standard'' (RES) written

2980

by Martijn Koster et al. in 1994. It specifies the format of a text

2981

file containing directives that instruct the robots which URL paths to

2982

avoid. To be found by the robots, the specifications must be placed in

2983

@file{/robots.txt} in the server root, which the robots are expected to

2984

download and parse.

2985

2986

Although Wget is not a web robot in the strictest sense of the word, it

2987

can downloads large parts of the site without the user's intervention to

2988

download an individual page. Because of that, Wget honors RES when

2989

downloading recursively. For instance, when you issue:

2990

2991

@example

2992

wget -r http://www.server.com/

2993

@end example

2994

2995

First the index of @samp{www.server.com} will be downloaded. If Wget

2996

finds that it wants to download more documents from that server, it will

2997

request @samp{http://www.server.com/robots.txt} and, if found, use it

2998

for further downloads. @file{robots.txt} is loaded only once per each

2999

server.

3000

3001

Until version 1.8, Wget supported the first version of the standard,

3002

written by Martijn Koster in 1994 and available at

3003

@url{http://www.robotstxt.org/wc/norobots.html}. As of version 1.8,

3004

Wget has supported the additional directives specified in the internet

3005

draft @samp{<draft-koster-robots-00.txt>} titled ``A Method for Web

3006

Robots Control''. The draft, which has as far as I know never made to

3007

an @sc{rfc}, is available at

3008

@url{http://www.robotstxt.org/wc/norobots-rfc.txt}.

3009

3010

This manual no longer includes the text of the Robot Exclusion Standard.

3011

3012

The second, less known mechanism, enables the author of an individual

3013

document to specify whether they want the links from the file to be

3014

followed by a robot. This is achieved using the @code{META} tag, like

3015

this:

3016

3017

@example

3018

3019

@end example

3020

3021

This is explained in some detail at

3022

@url{http://www.robotstxt.org/wc/meta-user.html}. Wget supports this

3023

method of robot exclusion in addition to the usual @file{/robots.txt}

3024

exclusion.

3025

3026

If you know what you are doing and really really wish to turn off the

3027

robot exclusion, set the @code{robots} variable to @samp{off} in your

3028

@file{.wgetrc}. You can achieve the same effect from the command line

3029

using the @code{-e} switch, e.g. @samp{wget -e robots=off @var{url}...}.

3030

3031

@node Security Considerations, Contributors, Robot Exclusion, Appendices

3032

@section Security Considerations

3033

@cindex security

3034

3035

When using Wget, you must be aware that it sends unencrypted passwords

3036

through the network, which may present a security problem. Here are the

3037

main issues, and some solutions.

3038

3039

@enumerate

3040

@item The passwords on the command line are visible using @code{ps}.

3041

The best way around it is to use @code{wget -i -} and feed the @sc{url}s

3042

to Wget's standard input, each on a separate line, terminated by

3043

@kbd{C-d}. Another workaround is to use @file{.netrc} to store

3044

passwords; however, storing unencrypted passwords is also considered a

3045

security risk.

3046

3047

@item

3048

Using the insecure @dfn{basic} authentication scheme, unencrypted

3049

passwords are transmitted through the network routers and gateways.

3050

3051

@item

3052

The @sc{ftp} passwords are also in no way encrypted. There is no good

3053

solution for this at the moment.

3054

3055

@item

3056

Although the ``normal'' output of Wget tries to hide the passwords,

3057

debugging logs show them, in all forms. This problem is avoided by

3058

being careful when you send debug logs (yes, even when you send them to

3059

me).

3060

@end enumerate

3061

3062

@node Contributors, , Security Considerations, Appendices

3063

@section Contributors

3064

@cindex contributors

3065

3066

@iftex

3067

GNU Wget was written by Hrvoje Nik@v{s}i@'{c} @email{hniksic@@xemacs.org}.

3068

@end iftex

3069

@ifnottex

3070

GNU Wget was written by Hrvoje Niksic @email{hniksic@@xemacs.org}.

3071

@end ifnottex

3072

However, its development could never have gone as far as it has, were it

3073

not for the help of many people, either with bug reports, feature

3074

proposals, patches, or letters saying ``Thanks!''.

3075

3076

Special thanks goes to the following people (no particular order):

3077

3078

@itemize @bullet

3079

@item

3080

Karsten Thygesen---donated system resources such as the mailing list,

3081

web space, and @sc{ftp} space, along with a lot of time to make these

3082

actually work.

3083

3084

@item

3085

Shawn McHorse---bug reports and patches.

3086

3087

@item

3088

Kaveh R. Ghazi---on-the-fly @code{ansi2knr}-ization. Lots of

3089

portability fixes.

3090

3091

@item

3092

Gordon Matzigkeit---@file{.netrc} support.

3093

3094

@item

3095

@iftex

3096

Zlatko @v{C}alu@v{s}i@'{c}, Tomislav Vujec and Dra@v{z}en

3097

Ka@v{c}ar---feature suggestions and ``philosophical'' discussions.

3098

@end iftex

3099

@ifnottex

3100

Zlatko Calusic, Tomislav Vujec and Drazen Kacar---feature suggestions

3101

and ``philosophical'' discussions.

3102

@end ifnottex

3103

3104

@item

3105

Darko Budor---initial port to Windows.

3106

3107

@item

3108

Antonio Rosella---help and suggestions, plus the Italian translation.

3109

3110

@item

3111

@iftex

3112

Tomislav Petrovi@'{c}, Mario Miko@v{c}evi@'{c}---many bug reports and

3113

suggestions.

3114

@end iftex

3115

@ifnottex

3116

Tomislav Petrovic, Mario Mikocevic---many bug reports and suggestions.

3117

@end ifnottex

3118

3119

@item

3120

@iftex

3121

Fran@,{c}ois Pinard---many thorough bug reports and discussions.

3122

@end iftex

3123

@ifnottex

3124

Francois Pinard---many thorough bug reports and discussions.

3125

@end ifnottex

3126

3127

@item

3128

Karl Eichwalder---lots of help with internationalization and other

3129

things.

3130

3131

@item

3132

Junio Hamano---donated support for Opie and @sc{http} @code{Digest}

3133

authentication.

3134

3135

@item

3136

The people who provided donations for development, including Brian

3137

Gough.

3138

@end itemize

3139

3140

The following people have provided patches, bug/build reports, useful

3141

suggestions, beta testing services, fan mail and all the other things

3142

that make maintenance so much fun:

3143

3144

Ian Abbott

3145

Tim Adam,

3146

Adrian Aichner,

3147

Martin Baehr,

3148

Dieter Baron,

3149

Roger Beeman,

3150

Dan Berger,

3151

T. Bharath,

3152

Paul Bludov,

3153

Daniel Bodea,

3154

Mark Boyns,

3155

John Burden,

3156

Wanderlei Cavassin,

3157

Gilles Cedoc,

3158

Tim Charron,

3159

Noel Cragg,

3160

@iftex

3161

Kristijan @v{C}onka@v{s},

3162

@end iftex

3163

@ifnottex

3164

Kristijan Conkas,

3165

@end ifnottex

3166

John Daily,

3167

Ahmon Dancy,

3168

Andrew Davison,

3169

Andrew Deryabin,

3170

Ulrich Drepper,

3171

Marc Duponcheel,

3172

@iftex

3173

Damir D@v{z}eko,

3174

@end iftex

3175

@ifnottex

3176

Damir Dzeko,

3177

@end ifnottex

3178

Alan Eldridge,

3179

@iftex

3180

Aleksandar Erkalovi@'{c},

3181

@end iftex

3182

@ifnottex

3183

Aleksandar Erkalovic,

3184

@end ifnottex

3185

Andy Eskilsson,

3186

Christian Fraenkel,

3187

Masashi Fujita,

3188

Howard Gayle,

3189

Marcel Gerrits,

3190

Lemble Gregory,

3191

Hans Grobler,

3192

Mathieu Guillaume,

3193

Dan Harkless,

3194

Aaron Hawley,

3195

Herold Heiko,

3196

Jochen Hein,

3197

Karl Heuer,

3198

HIROSE Masaaki,

3199

Gregor Hoffleit,

3200

Erik Magnus Hulthen,

3201

Richard Huveneers,

3202

Jonas Jensen,

3203

Simon Josefsson,

3204

@iftex

3205

Mario Juri@'{c},

3206

@end iftex

3207

@ifnottex

3208

Mario Juric,

3209

@end ifnottex

3210

@iftex

3211

Hack Kampbj@o rn,

3212

@end iftex

3213

@ifnottex

3214

Hack Kampbjorn,

3215

@end ifnottex

3216

Const Kaplinsky,

3217

@iftex

3218

Goran Kezunovi@'{c},

3219

@end iftex

3220

@ifnottex

3221

Goran Kezunovic,

3222

@end ifnottex

3223

Robert Kleine,

3224

KOJIMA Haime,

3225

Fila Kolodny,

3226

Alexander Kourakos,

3227

Martin Kraemer,

3228

@tex

3229

$\Sigma\acute{\iota}\mu o\varsigma\;

3230

\Xi\varepsilon\nu\iota\tau\acute{\epsilon}\lambda\lambda\eta\varsigma$

3231

(Simos KSenitellis),

3232

@end tex

3233

@ifnottex

3234

Simos KSenitellis,

3235

@end ifnottex

3236

Hrvoje Lacko,

3237

Daniel S. Lewart,

3238

@iftex

3239

Nicol@'{a}s Lichtmeier,

3240

@end iftex

3241

@ifnottex

3242

Nicolas Lichtmeier,

3243

@end ifnottex

3244

Dave Love,

3245

Alexander V. Lukyanov,

3246

Thomas Lussnig,

3247

Aurelien Marchand,

3248

Jordan Mendelson,

3249

Lin Zhe Min,

3250

Tim Mooney,

3251

Simon Munton,

3252

Charlie Negyesi,

3253

R. K. Owen,

3254

Andrew Pollock,

3255

Steve Pothier,

3256

@iftex

3257

Jan P@v{r}ikryl,

3258

@end iftex

3259

@ifnottex

3260

Jan Prikryl,

3261

@end ifnottex

3262

Marin Purgar,

3263

@iftex

3264

Csaba R@'{a}duly,

3265

@end iftex

3266

@ifnottex

3267

Csaba Raduly,

3268

@end ifnottex

3269

Keith Refson,

3270

Bill Richardson,

3271

Tyler Riddle,

3272

Tobias Ringstrom,

3273

@c Texinfo doesn't grok @'{@i}, so we have to use TeX itself.

3274

@tex

3275

Juan Jos\'{e} Rodr\'{\i}gues,

3276

@end tex

3277

@ifnottex

3278

Juan Jose Rodrigues,

3279

@end ifnottex

3280

Maciej W. Rozycki,

3281

Edward J. Sabol,

3282

Heinz Salzmann,

3283

Robert Schmidt,

3284

Andreas Schwab,

3285

Chris Seawood,

3286

Toomas Soome,

3287

Tage Stabell-Kulo,

3288

Sven Sternberger,

3289

Markus Strasser,

3290

John Summerfield,

3291

Szakacsits Szabolcs,

3292

Mike Thomas,

3293

Philipp Thomas,

3294

Mauro Tortonesi,

3295

Dave Turner,

3296

Gisle Vanem,

3297

Russell Vincent,

3298

Charles G Waldman,

3299

Douglas E. Wegscheid,

3300

Jasmin Zainul,

3301

@iftex

3302

Bojan @v{Z}drnja,

3303

@end iftex

3304

@ifnottex

3305

Bojan Zdrnja,

3306

@end ifnottex

3307

Kristijan Zimmer.

3308

3309

Apologies to all who I accidentally left out, and many thanks to all the

3310

subscribers of the Wget mailing list.

3311

3312

@node Copying, Concept Index, Appendices, Top

3313

@chapter Copying

3314

@cindex copying

3315

@cindex GPL

3316

@cindex GFDL

3317

@cindex free software

3318

3319

GNU Wget is licensed under the GNU GPL, which makes it @dfn{free

3320

software}.

3321

3322

Please note that ``free'' in ``free software'' refers to liberty, not

3323

price. As some GNU project advocates like to point out, think of ``free

3324

speech'' rather than ``free beer''. The exact and legally binding

3325

distribution terms are spelled out below; in short, you have the right

3326

(freedom) to run and change Wget and distribute it to other people, and

3327

even---if you want---charge money for doing either. The important

3328

restriction is that you have to grant your recipients the same rights

3329

and impose the same restrictions.

3330

3331

This method of licensing software is also known as @dfn{open source}

3332

because, among other things, it makes sure that all recipients will

3333

receive the source code along with the program, and be able to improve

3334

it. The GNU project prefers the term ``free software'' for reasons

3335

outlined at

3336

@url{http://www.gnu.org/philosophy/free-software-for-freedom.html}.

3337

3338

The exact license terms are defined by this paragraph and the GNU

3339

General Public License it refers to:

3340

3341

@quotation

3342

GNU Wget is free software; you can redistribute it and/or modify it

3343

under the terms of the GNU General Public License as published by the

3344

Free Software Foundation; either version 2 of the License, or (at your

3345

option) any later version.

3346

3347

GNU Wget is distributed in the hope that it will be useful, but WITHOUT

3348

ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or

3349

FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License

3350

for more details.

3351

3352

A copy of the GNU General Public License is included as part of this

3353

manual; if you did not receive it, write to the Free Software

3354

Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

3355

@end quotation

3356

3357

In addition to this, this manual is free in the same sense:

3358

3359

@quotation

3360

Permission is granted to copy, distribute and/or modify this document

3361

under the terms of the GNU Free Documentation License, Version 1.1 or

3362

any later version published by the Free Software Foundation; with the

3363

Invariant Sections being ``GNU General Public License'' and ``GNU Free

3364

Documentation License'', with no Front-Cover Texts, and with no

3365

Back-Cover Texts. A copy of the license is included in the section

3366

entitled ``GNU Free Documentation License''.

3367

@end quotation

3368

3369

@c #### Maybe we should wrap these licenses in ifinfo? Stallman says

3370

@c that the GFDL needs to be present in the manual, and to me it would

3371

@c suck to include the license for the manual and not the license for

3372

@c the program.

3373

3374

The full texts of the GNU General Public License and of the GNU Free

3375

Documentation License are available below.

3376

3377

@menu

3378

* GNU General Public License::

3379

* GNU Free Documentation License::

3380

@end menu

3381

3382

@node GNU General Public License, GNU Free Documentation License, Copying, Copying

3383

@section GNU General Public License

3384

@center Version 2, June 1991

3385

3386

@display

3387

3388

675 Mass Ave, Cambridge, MA 02139, USA

3389

3390

Everyone is permitted to copy and distribute verbatim copies

3391

of this license document, but changing it is not allowed.

3392

@end display

3393

3394

@unnumberedsec Preamble

3395

3396

The licenses for most software are designed to take away your

3397

freedom to share and change it. By contrast, the GNU General Public

3398

License is intended to guarantee your freedom to share and change free

3399

software---to make sure the software is free for all its users. This

3400

General Public License applies to most of the Free Software

3401

Foundation's software and to any other program whose authors commit to

3402

using it. (Some other Free Software Foundation software is covered by

3403

the GNU Library General Public License instead.) You can apply it to

3404

your programs, too.

3405

3406

When we speak of free software, we are referring to freedom, not

3407

price. Our General Public Licenses are designed to make sure that you

3408

have the freedom to distribute copies of free software (and charge for

3409

this service if you wish), that you receive source code or can get it

3410

if you want it, that you can change the software or use pieces of it

3411

in new free programs; and that you know you can do these things.

3412

3413

To protect your rights, we need to make restrictions that forbid

3414

anyone to deny you these rights or to ask you to surrender the rights.

3415

These restrictions translate to certain responsibilities for you if you

3416

distribute copies of the software, or if you modify it.

3417

3418

For example, if you distribute copies of such a program, whether

3419

gratis or for a fee, you must give the recipients all the rights that

3420

you have. You must make sure that they, too, receive or can get the

3421

source code. And you must show them these terms so they know their

3422

rights.

3423

3424

We protect your rights with two steps: (1) copyright the software, and

3425

(2) offer you this license which gives you legal permission to copy,

3426

distribute and/or modify the software.

3427

3428

Also, for each author's protection and ours, we want to make certain

3429

that everyone understands that there is no warranty for this free

3430

software. If the software is modified by someone else and passed on, we

3431

want its recipients to know that what they have is not the original, so

3432

that any problems introduced by others will not reflect on the original

3433

authors' reputations.

3434

3435

Finally, any free program is threatened constantly by software

3436

patents. We wish to avoid the danger that redistributors of a free

3437

program will individually obtain patent licenses, in effect making the

3438

program proprietary. To prevent this, we have made it clear that any

3439

patent must be licensed for everyone's free use or not licensed at all.

3440

3441

The precise terms and conditions for copying, distribution and

3442

modification follow.

3443

3444

@iftex

3445

@unnumberedsec TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

3446

@end iftex

3447

@ifnottex

3448

@center TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

3449

@end ifnottex

3450

3451

@enumerate

3452

@item

3453

This License applies to any program or other work which contains

3454

a notice placed by the copyright holder saying it may be distributed

3455

under the terms of this General Public License. The ``Program'', below,

3456

refers to any such program or work, and a ``work based on the Program''

3457

means either the Program or any derivative work under copyright law:

3458

that is to say, a work containing the Program or a portion of it,

3459

either verbatim or with modifications and/or translated into another

3460

language. (Hereinafter, translation is included without limitation in

3461

the term ``modification''.) Each licensee is addressed as ``you''.

3462

3463

Activities other than copying, distribution and modification are not

3464

covered by this License; they are outside its scope. The act of

3465

running the Program is not restricted, and the output from the Program

3466

is covered only if its contents constitute a work based on the

3467

Program (independent of having been made by running the Program).

3468

Whether that is true depends on what the Program does.

3469

3470

@item

3471

You may copy and distribute verbatim copies of the Program's

3472

source code as you receive it, in any medium, provided that you

3473

conspicuously and appropriately publish on each copy an appropriate

3474

copyright notice and disclaimer of warranty; keep intact all the

3475

notices that refer to this License and to the absence of any warranty;

3476

and give any other recipients of the Program a copy of this License

3477

along with the Program.

3478

3479

You may charge a fee for the physical act of transferring a copy, and

3480

you may at your option offer warranty protection in exchange for a fee.

3481

3482

@item

3483

You may modify your copy or copies of the Program or any portion

3484

of it, thus forming a work based on the Program, and copy and

3485

distribute such modifications or work under the terms of Section 1

3486

above, provided that you also meet all of these conditions:

3487

3488

@enumerate a

3489

@item

3490

You must cause the modified files to carry prominent notices

3491

stating that you changed the files and the date of any change.

3492

3493

@item

3494

You must cause any work that you distribute or publish, that in

3495

whole or in part contains or is derived from the Program or any

3496

part thereof, to be licensed as a whole at no charge to all third

3497

parties under the terms of this License.

3498

3499

@item

3500

If the modified program normally reads commands interactively

3501

when run, you must cause it, when started running for such

3502

interactive use in the most ordinary way, to print or display an

3503

announcement including an appropriate copyright notice and a

3504

notice that there is no warranty (or else, saying that you provide

3505

a warranty) and that users may redistribute the program under

3506

these conditions, and telling the user how to view a copy of this

3507

License. (Exception: if the Program itself is interactive but

3508

does not normally print such an announcement, your work based on

3509

the Program is not required to print an announcement.)

3510

@end enumerate

3511

3512

These requirements apply to the modified work as a whole. If

3513

identifiable sections of that work are not derived from the Program,

3514

and can be reasonably considered independent and separate works in

3515

themselves, then this License, and its terms, do not apply to those

3516

sections when you distribute them as separate works. But when you

3517

distribute the same sections as part of a whole which is a work based

3518

on the Program, the distribution of the whole must be on the terms of

3519

this License, whose permissions for other licensees extend to the

3520

entire whole, and thus to each and every part regardless of who wrote it.

3521

3522

Thus, it is not the intent of this section to claim rights or contest

3523

your rights to work written entirely by you; rather, the intent is to

3524

exercise the right to control the distribution of derivative or

3525

collective works based on the Program.

3526

3527

In addition, mere aggregation of another work not based on the Program

3528

with the Program (or with a work based on the Program) on a volume of

3529

a storage or distribution medium does not bring the other work under

3530

the scope of this License.

3531

3532

@item

3533

You may copy and distribute the Program (or a work based on it,

3534

under Section 2) in object code or executable form under the terms of

3535

Sections 1 and 2 above provided that you also do one of the following:

3536

3537

@enumerate a

3538

@item

3539

Accompany it with the complete corresponding machine-readable

3540

source code, which must be distributed under the terms of Sections

3541

1 and 2 above on a medium customarily used for software interchange; or,

3542

3543

@item

3544

Accompany it with a written offer, valid for at least three

3545

years, to give any third party, for a charge no more than your

3546

cost of physically performing source distribution, a complete

3547

machine-readable copy of the corresponding source code, to be

3548

distributed under the terms of Sections 1 and 2 above on a medium

3549

customarily used for software interchange; or,

3550

3551

@item

3552

Accompany it with the information you received as to the offer

3553

to distribute corresponding source code. (This alternative is

3554

allowed only for noncommercial distribution and only if you

3555

received the program in object code or executable form with such

3556

an offer, in accord with Subsection b above.)

3557

@end enumerate

3558

3559

The source code for a work means the preferred form of the work for

3560

making modifications to it. For an executable work, complete source

3561

code means all the source code for all modules it contains, plus any

3562

associated interface definition files, plus the scripts used to

3563

control compilation and installation of the executable. However, as a

3564

special exception, the source code distributed need not include

3565

anything that is normally distributed (in either source or binary

3566

form) with the major components (compiler, kernel, and so on) of the

3567

operating system on which the executable runs, unless that component

3568

itself accompanies the executable.

3569

3570

If distribution of executable or object code is made by offering

3571

access to copy from a designated place, then offering equivalent

3572

access to copy the source code from the same place counts as

3573

distribution of the source code, even though third parties are not

3574

compelled to copy the source along with the object code.

3575

3576

@item

3577

You may not copy, modify, sublicense, or distribute the Program

3578

except as expressly provided under this License. Any attempt

3579

otherwise to copy, modify, sublicense or distribute the Program is

3580

void, and will automatically terminate your rights under this License.

3581

However, parties who have received copies, or rights, from you under

3582

this License will not have their licenses terminated so long as such

3583

parties remain in full compliance.

3584

3585

@item

3586

You are not required to accept this License, since you have not

3587

signed it. However, nothing else grants you permission to modify or

3588

distribute the Program or its derivative works. These actions are

3589

prohibited by law if you do not accept this License. Therefore, by

3590

modifying or distributing the Program (or any work based on the

3591

Program), you indicate your acceptance of this License to do so, and

3592

all its terms and conditions for copying, distributing or modifying

3593

the Program or works based on it.

3594

3595

@item

3596

Each time you redistribute the Program (or any work based on the

3597

Program), the recipient automatically receives a license from the

3598

original licensor to copy, distribute or modify the Program subject to

3599

these terms and conditions. You may not impose any further

3600

restrictions on the recipients' exercise of the rights granted herein.

3601

You are not responsible for enforcing compliance by third parties to

3602

this License.

3603

3604

@item

3605

If, as a consequence of a court judgment or allegation of patent

3606

infringement or for any other reason (not limited to patent issues),

3607

conditions are imposed on you (whether by court order, agreement or

3608

otherwise) that contradict the conditions of this License, they do not

3609

excuse you from the conditions of this License. If you cannot

3610

distribute so as to satisfy simultaneously your obligations under this

3611

License and any other pertinent obligations, then as a consequence you

3612

may not distribute the Program at all. For example, if a patent

3613

license would not permit royalty-free redistribution of the Program by

3614

all those who receive copies directly or indirectly through you, then

3615

the only way you could satisfy both it and this License would be to

3616

refrain entirely from distribution of the Program.

3617

3618

If any portion of this section is held invalid or unenforceable under

3619

any particular circumstance, the balance of the section is intended to

3620

apply and the section as a whole is intended to apply in other

3621

circumstances.

3622

3623

It is not the purpose of this section to induce you to infringe any

3624

patents or other property right claims or to contest validity of any

3625

such claims; this section has the sole purpose of protecting the

3626

integrity of the free software distribution system, which is

3627

implemented by public license practices. Many people have made

3628

generous contributions to the wide range of software distributed

3629

through that system in reliance on consistent application of that

3630

system; it is up to the author/donor to decide if he or she is willing

3631

to distribute software through any other system and a licensee cannot

3632

impose that choice.

3633

3634

This section is intended to make thoroughly clear what is believed to

3635

be a consequence of the rest of this License.

3636

3637

@item

3638

If the distribution and/or use of the Program is restricted in

3639

certain countries either by patents or by copyrighted interfaces, the

3640

original copyright holder who places the Program under this License

3641

may add an explicit geographical distribution limitation excluding

3642

those countries, so that distribution is permitted only in or among

3643

countries not thus excluded. In such case, this License incorporates

3644

the limitation as if written in the body of this License.

3645

3646

@item

3647

The Free Software Foundation may publish revised and/or new versions

3648

of the General Public License from time to time. Such new versions will

3649

be similar in spirit to the present version, but may differ in detail to

3650

address new problems or concerns.

3651

3652

Each version is given a distinguishing version number. If the Program

3653

specifies a version number of this License which applies to it and ``any

3654

later version'', you have the option of following the terms and conditions

3655

either of that version or of any later version published by the Free

3656

Software Foundation. If the Program does not specify a version number of

3657

this License, you may choose any version ever published by the Free Software

3658

Foundation.

3659

3660

@item

3661

If you wish to incorporate parts of the Program into other free

3662

programs whose distribution conditions are different, write to the author

3663

to ask for permission. For software which is copyrighted by the Free

3664

Software Foundation, write to the Free Software Foundation; we sometimes

3665

make exceptions for this. Our decision will be guided by the two goals

3666

of preserving the free status of all derivatives of our free software and

3667

of promoting the sharing and reuse of software generally.

3668

3669

@iftex

3670

@heading NO WARRANTY

3671

@end iftex

3672

@ifnottex

3673

@center NO WARRANTY

3674

@end ifnottex

3675

@cindex no warranty

3676

3677

@item

3678

BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY

3679

FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN

3680

OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES

3681

PROVIDE THE PROGRAM ``AS IS'' WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED

3682

OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF

3683

MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS

3684

TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE

3685

PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,

3686

REPAIR OR CORRECTION.

3687

3688

@item

3689

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING

3690

WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR

3691

REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,

3692

INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING

3693

OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED

3694

TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY

3695

YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER

3696

PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE

3697

POSSIBILITY OF SUCH DAMAGES.

3698

@end enumerate

3699

3700

@iftex

3701

@heading END OF TERMS AND CONDITIONS

3702

@end iftex

3703

@ifnottex

3704

@center END OF TERMS AND CONDITIONS

3705

@end ifnottex

3706

3707

@page

3708

@unnumberedsec How to Apply These Terms to Your New Programs

3709

3710

If you develop a new program, and you want it to be of the greatest

3711

possible use to the public, the best way to achieve this is to make it

3712

free software which everyone can redistribute and change under these terms.

3713

3714

To do so, attach the following notices to the program. It is safest

3715

to attach them to the start of each source file to most effectively

3716

convey the exclusion of warranty; and each file should have at least

3717

the ``copyright'' line and a pointer to where the full notice is found.

3718

3719

@smallexample

3720

@var{one line to give the program's name and an idea of what it does.}

3721

Copyright (C) 19@var{yy} @var{name of author}

3722

3723

This program is free software; you can redistribute it and/or

3724

modify it under the terms of the GNU General Public License

3725

as published by the Free Software Foundation; either version 2

3726

of the License, or (at your option) any later version.

3727

3728

This program is distributed in the hope that it will be useful,

3729

but WITHOUT ANY WARRANTY; without even the implied warranty of

3730

MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

3731

GNU General Public License for more details.

3732

3733

You should have received a copy of the GNU General Public License

3734

along with this program; if not, write to the Free Software

3735

Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

3736

@end smallexample

3737

3738

Also add information on how to contact you by electronic and paper mail.

3739

3740

If the program is interactive, make it output a short notice like this

3741

when it starts in an interactive mode:

3742

3743

@smallexample

3744

Gnomovision version 69, Copyright (C) 19@var{yy} @var{name of author}

3745

Gnomovision comes with ABSOLUTELY NO WARRANTY; for details

3746

type `show w'. This is free software, and you are welcome

3747

to redistribute it under certain conditions; type `show c'

3748

for details.

3749

@end smallexample

3750

3751

The hypothetical commands @samp{show w} and @samp{show c} should show

3752

the appropriate parts of the General Public License. Of course, the

3753

commands you use may be called something other than @samp{show w} and

3754

@samp{show c}; they could even be mouse-clicks or menu items---whatever

3755

suits your program.

3756

3757

You should also get your employer (if you work as a programmer) or your

3758

school, if any, to sign a ``copyright disclaimer'' for the program, if

3759

necessary. Here is a sample; alter the names:

3760

3761

@smallexample

3762

@group

3763

Yoyodyne, Inc., hereby disclaims all copyright

3764

interest in the program `Gnomovision'

3765

(which makes passes at compilers) written

3766

by James Hacker.

3767

3768

@var{signature of Ty Coon}, 1 April 1989

3769

Ty Coon, President of Vice

3770

@end group

3771

@end smallexample

3772

3773

This General Public License does not permit incorporating your program into

3774

proprietary programs. If your program is a subroutine library, you may

3775

consider it more useful to permit linking proprietary applications with the

3776

library. If this is what you want to do, use the GNU Library General

3777

Public License instead of this License.

3778

3779

@node GNU Free Documentation License, , GNU General Public License, Copying

3780

@section GNU Free Documentation License

3781

@center Version 1.1, March 2000

3782

3783

@display

3784

3785

59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

3786

3787

Everyone is permitted to copy and distribute verbatim copies

3788

of this license document, but changing it is not allowed.

3789

@end display

3790

@sp 1

3791

@enumerate 0

3792

@item

3793

PREAMBLE

3794

3795

The purpose of this License is to make a manual, textbook, or other

3796

written document ``free'' in the sense of freedom: to assure everyone

3797

the effective freedom to copy and redistribute it, with or without

3798

modifying it, either commercially or noncommercially. Secondarily,

3799

this License preserves for the author and publisher a way to get

3800

credit for their work, while not being considered responsible for

3801

modifications made by others.

3802

3803

This License is a kind of ``copyleft'', which means that derivative

3804

works of the document must themselves be free in the same sense. It

3805

complements the GNU General Public License, which is a copyleft

3806

license designed for free software.

3807

3808

We have designed this License in order to use it for manuals for free

3809

software, because free software needs free documentation: a free

3810

program should come with manuals providing the same freedoms that the

3811

software does. But this License is not limited to software manuals;

3812

it can be used for any textual work, regardless of subject matter or

3813

whether it is published as a printed book. We recommend this License

3814

principally for works whose purpose is instruction or reference.

3815

3816

@sp 1

3817

@item

3818

APPLICABILITY AND DEFINITIONS

3819

3820

This License applies to any manual or other work that contains a

3821

notice placed by the copyright holder saying it can be distributed

3822

under the terms of this License. The ``Document'', below, refers to any

3823

such manual or work. Any member of the public is a licensee, and is

3824

addressed as ``you''.

3825

3826

A ``Modified Version'' of the Document means any work containing the

3827

Document or a portion of it, either copied verbatim, or with

3828

modifications and/or translated into another language.

3829

3830

A ``Secondary Section'' is a named appendix or a front-matter section of

3831

the Document that deals exclusively with the relationship of the

3832

publishers or authors of the Document to the Document's overall subject

3833

(or to related matters) and contains nothing that could fall directly

3834

within that overall subject. (For example, if the Document is in part a

3835

textbook of mathematics, a Secondary Section may not explain any

3836

mathematics.) The relationship could be a matter of historical

3837

connection with the subject or with related matters, or of legal,

3838

commercial, philosophical, ethical or political position regarding

3839

them.

3840

3841

The ``Invariant Sections'' are certain Secondary Sections whose titles

3842

are designated, as being those of Invariant Sections, in the notice

3843

that says that the Document is released under this License.

3844

3845

The ``Cover Texts'' are certain short passages of text that are listed,

3846

as Front-Cover Texts or Back-Cover Texts, in the notice that says that

3847

the Document is released under this License.

3848

3849

A ``Transparent'' copy of the Document means a machine-readable copy,

3850

represented in a format whose specification is available to the

3851

general public, whose contents can be viewed and edited directly and

3852

straightforwardly with generic text editors or (for images composed of

3853

pixels) generic paint programs or (for drawings) some widely available

3854

drawing editor, and that is suitable for input to text formatters or

3855

for automatic translation to a variety of formats suitable for input

3856

to text formatters. A copy made in an otherwise Transparent file

3857

format whose markup has been designed to thwart or discourage

3858

subsequent modification by readers is not Transparent. A copy that is

3859

not ``Transparent'' is called ``Opaque''.

3860

3861

Examples of suitable formats for Transparent copies include plain

3862

@sc{ascii} without markup, Texinfo input format, LaTeX input format, @sc{sgml}

3863

or @sc{xml} using a publicly available @sc{dtd}, and standard-conforming simple

3864

@sc{html} designed for human modification. Opaque formats include

3865

PostScript, @sc{pdf}, proprietary formats that can be read and edited only

3866

by proprietary word processors, @sc{sgml} or @sc{xml} for which the @sc{dtd} and/or

3867

processing tools are not generally available, and the

3868

machine-generated @sc{html} produced by some word processors for output

3869

purposes only.

3870

3871

The ``Title Page'' means, for a printed book, the title page itself,

3872

plus such following pages as are needed to hold, legibly, the material

3873

this License requires to appear in the title page. For works in

3874

formats which do not have any title page as such, ``Title Page'' means

3875

the text near the most prominent appearance of the work's title,

3876

preceding the beginning of the body of the text.

3877

@sp 1

3878

@item

3879

VERBATIM COPYING

3880

3881

You may copy and distribute the Document in any medium, either

3882

commercially or noncommercially, provided that this License, the

3883

copyright notices, and the license notice saying this License applies

3884

to the Document are reproduced in all copies, and that you add no other

3885

conditions whatsoever to those of this License. You may not use

3886

technical measures to obstruct or control the reading or further

3887

copying of the copies you make or distribute. However, you may accept

3888

compensation in exchange for copies. If you distribute a large enough

3889

number of copies you must also follow the conditions in section 3.

3890

3891

You may also lend copies, under the same conditions stated above, and

3892

you may publicly display copies.

3893

@sp 1

3894

@item

3895

COPYING IN QUANTITY

3896

3897

If you publish printed copies of the Document numbering more than 100,

3898

and the Document's license notice requires Cover Texts, you must enclose

3899

the copies in covers that carry, clearly and legibly, all these Cover

3900

Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on

3901

the back cover. Both covers must also clearly and legibly identify

3902

you as the publisher of these copies. The front cover must present

3903

the full title with all words of the title equally prominent and

3904

visible. You may add other material on the covers in addition.

3905

Copying with changes limited to the covers, as long as they preserve

3906

the title of the Document and satisfy these conditions, can be treated

3907

as verbatim copying in other respects.

3908

3909

If the required texts for either cover are too voluminous to fit

3910

legibly, you should put the first ones listed (as many as fit

3911

reasonably) on the actual cover, and continue the rest onto adjacent

3912

pages.

3913

3914

If you publish or distribute Opaque copies of the Document numbering

3915

more than 100, you must either include a machine-readable Transparent

3916

copy along with each Opaque copy, or state in or with each Opaque copy

3917

a publicly-accessible computer-network location containing a complete

3918

Transparent copy of the Document, free of added material, which the

3919

general network-using public has access to download anonymously at no

3920

charge using public-standard network protocols. If you use the latter

3921

option, you must take reasonably prudent steps, when you begin

3922

distribution of Opaque copies in quantity, to ensure that this

3923

Transparent copy will remain thus accessible at the stated location

3924

until at least one year after the last time you distribute an Opaque

3925

copy (directly or through your agents or retailers) of that edition to

3926

the public.

3927

3928

It is requested, but not required, that you contact the authors of the

3929

Document well before redistributing any large number of copies, to give

3930

them a chance to provide you with an updated version of the Document.

3931

@sp 1

3932

@item

3933

MODIFICATIONS

3934

3935

You may copy and distribute a Modified Version of the Document under

3936

the conditions of sections 2 and 3 above, provided that you release

3937

the Modified Version under precisely this License, with the Modified

3938

Version filling the role of the Document, thus licensing distribution

3939

and modification of the Modified Version to whoever possesses a copy

3940

of it. In addition, you must do these things in the Modified Version:

3941

3942

A. Use in the Title Page (and on the covers, if any) a title distinct

3943

from that of the Document, and from those of previous versions

3944

(which should, if there were any, be listed in the History section

3945

of the Document). You may use the same title as a previous version

3946

if the original publisher of that version gives permission.@*

3947

B. List on the Title Page, as authors, one or more persons or entities

3948

responsible for authorship of the modifications in the Modified

3949

Version, together with at least five of the principal authors of the

3950

Document (all of its principal authors, if it has less than five).@*

3951

C. State on the Title page the name of the publisher of the

3952

Modified Version, as the publisher.@*

3953

D. Preserve all the copyright notices of the Document.@*

3954

E. Add an appropriate copyright notice for your modifications

3955

adjacent to the other copyright notices.@*

3956

F. Include, immediately after the copyright notices, a license notice

3957

giving the public permission to use the Modified Version under the

3958

terms of this License, in the form shown in the Addendum below.@*

3959

G. Preserve in that license notice the full lists of Invariant Sections

3960

and required Cover Texts given in the Document's license notice.@*

3961

H. Include an unaltered copy of this License.@*

3962

I. Preserve the section entitled ``History'', and its title, and add to

3963

it an item stating at least the title, year, new authors, and

3964

publisher of the Modified Version as given on the Title Page. If

3965

there is no section entitled ``History'' in the Document, create one

3966

stating the title, year, authors, and publisher of the Document as

3967

given on its Title Page, then add an item describing the Modified

3968

Version as stated in the previous sentence.@*

3969

J. Preserve the network location, if any, given in the Document for

3970

public access to a Transparent copy of the Document, and likewise

3971

the network locations given in the Document for previous versions

3972

it was based on. These may be placed in the ``History'' section.

3973

You may omit a network location for a work that was published at

3974

least four years before the Document itself, or if the original

3975

publisher of the version it refers to gives permission.@*

3976

K. In any section entitled ``Acknowledgements'' or ``Dedications'',

3977

preserve the section's title, and preserve in the section all the

3978

substance and tone of each of the contributor acknowledgements

3979

and/or dedications given therein.@*

3980

L. Preserve all the Invariant Sections of the Document,

3981

unaltered in their text and in their titles. Section numbers

3982

or the equivalent are not considered part of the section titles.@*

3983

M. Delete any section entitled ``Endorsements''. Such a section

3984

may not be included in the Modified Version.@*

3985

N. Do not retitle any existing section as ``Endorsements''

3986

or to conflict in title with any Invariant Section.@*

3987

@sp 1

3988

If the Modified Version includes new front-matter sections or

3989

appendices that qualify as Secondary Sections and contain no material

3990

copied from the Document, you may at your option designate some or all

3991

of these sections as invariant. To do this, add their titles to the

3992

list of Invariant Sections in the Modified Version's license notice.

3993

These titles must be distinct from any other section titles.

3994

3995

You may add a section entitled ``Endorsements'', provided it contains

3996

nothing but endorsements of your Modified Version by various

3997

parties--for example, statements of peer review or that the text has

3998

been approved by an organization as the authoritative definition of a

3999

standard.

4000

4001

You may add a passage of up to five words as a Front-Cover Text, and a

4002

passage of up to 25 words as a Back-Cover Text, to the end of the list

4003

of Cover Texts in the Modified Version. Only one passage of

4004

Front-Cover Text and one of Back-Cover Text may be added by (or

4005

through arrangements made by) any one entity. If the Document already

4006

includes a cover text for the same cover, previously added by you or

4007

by arrangement made by the same entity you are acting on behalf of,

4008

you may not add another; but you may replace the old one, on explicit

4009

permission from the previous publisher that added the old one.

4010

4011

The author(s) and publisher(s) of the Document do not by this License

4012

give permission to use their names for publicity for or to assert or

4013

imply endorsement of any Modified Version.

4014

@sp 1

4015

@item

4016

COMBINING DOCUMENTS

4017

4018

You may combine the Document with other documents released under this

4019

License, under the terms defined in section 4 above for modified

4020

versions, provided that you include in the combination all of the

4021

Invariant Sections of all of the original documents, unmodified, and

4022

list them all as Invariant Sections of your combined work in its

4023

license notice.

4024

4025

The combined work need only contain one copy of this License, and

4026

multiple identical Invariant Sections may be replaced with a single

4027

copy. If there are multiple Invariant Sections with the same name but

4028

different contents, make the title of each such section unique by

4029

adding at the end of it, in parentheses, the name of the original

4030

author or publisher of that section if known, or else a unique number.

4031

Make the same adjustment to the section titles in the list of

4032

Invariant Sections in the license notice of the combined work.

4033

4034

In the combination, you must combine any sections entitled ``History''

4035

in the various original documents, forming one section entitled

4036

``History''; likewise combine any sections entitled ``Acknowledgements'',

4037

and any sections entitled ``Dedications''. You must delete all sections

4038

entitled ``Endorsements.''

4039

@sp 1

4040

@item

4041

COLLECTIONS OF DOCUMENTS

4042

4043

You may make a collection consisting of the Document and other documents

4044

released under this License, and replace the individual copies of this

4045

License in the various documents with a single copy that is included in

4046

the collection, provided that you follow the rules of this License for

4047

verbatim copying of each of the documents in all other respects.

4048

4049

You may extract a single document from such a collection, and distribute

4050

it individually under this License, provided you insert a copy of this

4051

License into the extracted document, and follow this License in all

4052

other respects regarding verbatim copying of that document.

4053

@sp 1

4054

@item

4055

AGGREGATION WITH INDEPENDENT WORKS

4056

4057

A compilation of the Document or its derivatives with other separate

4058

and independent documents or works, in or on a volume of a storage or

4059

distribution medium, does not as a whole count as a Modified Version

4060

of the Document, provided no compilation copyright is claimed for the

4061

compilation. Such a compilation is called an ``aggregate'', and this

4062

License does not apply to the other self-contained works thus compiled

4063

with the Document, on account of their being thus compiled, if they

4064

are not themselves derivative works of the Document.

4065

4066

If the Cover Text requirement of section 3 is applicable to these

4067

copies of the Document, then if the Document is less than one quarter

4068

of the entire aggregate, the Document's Cover Texts may be placed on

4069

covers that surround only the Document within the aggregate.

4070

Otherwise they must appear on covers around the whole aggregate.

4071

@sp 1

4072

@item

4073

TRANSLATION

4074

4075

Translation is considered a kind of modification, so you may

4076

distribute translations of the Document under the terms of section 4.

4077

Replacing Invariant Sections with translations requires special

4078

permission from their copyright holders, but you may include

4079

translations of some or all Invariant Sections in addition to the

4080

original versions of these Invariant Sections. You may include a

4081

translation of this License provided that you also include the

4082

original English version of this License. In case of a disagreement

4083

between the translation and the original English version of this

4084

License, the original English version will prevail.

4085

@sp 1

4086

@item

4087

TERMINATION

4088

4089

You may not copy, modify, sublicense, or distribute the Document except

4090

as expressly provided for under this License. Any other attempt to

4091

copy, modify, sublicense or distribute the Document is void, and will

4092

automatically terminate your rights under this License. However,

4093

parties who have received copies, or rights, from you under this

4094

License will not have their licenses terminated so long as such

4095

parties remain in full compliance.

4096

@sp 1

4097

@item

4098

FUTURE REVISIONS OF THIS LICENSE

4099

4100

The Free Software Foundation may publish new, revised versions

4101

of the GNU Free Documentation License from time to time. Such new

4102

versions will be similar in spirit to the present version, but may

4103

differ in detail to address new problems or concerns. See

4104

http://www.gnu.org/copyleft/.

4105

4106

Each version of the License is given a distinguishing version number.

4107

If the Document specifies that a particular numbered version of this

4108

License ``or any later version'' applies to it, you have the option of

4109

following the terms and conditions either of that specified version or

4110

of any later version that has been published (not as a draft) by the

4111

Free Software Foundation. If the Document does not specify a version

4112

number of this License, you may choose any version ever published (not

4113

as a draft) by the Free Software Foundation.

4114

4115

@end enumerate

4116

4117

@unnumberedsec ADDENDUM: How to use this License for your documents

4118

4119

To use this License in a document you have written, include a copy of

4120

the License in the document and put the following copyright and

4121

license notices just after the title page:

4122

4123

@smallexample

4124

@group

4125

4126

Copyright (C) @var{year} @var{your name}.

4127

Permission is granted to copy, distribute and/or modify this document

4128

under the terms of the GNU Free Documentation License, Version 1.1

4129

or any later version published by the Free Software Foundation;

4130

with the Invariant Sections being @var{list their titles}, with the

4131

Front-Cover Texts being @var{list}, and with the Back-Cover Texts being @var{list}.

4132

A copy of the license is included in the section entitled ``GNU

4133

Free Documentation License''.

4134

@end group

4135

@end smallexample

4136

If you have no Invariant Sections, write ``with no Invariant Sections''

4137

instead of saying which ones are invariant. If you have no

4138

Front-Cover Texts, write ``no Front-Cover Texts'' instead of

4139

``Front-Cover Texts being @var{list}''; likewise for Back-Cover Texts.

4140

4141

If your document contains nontrivial examples of program code, we

4142

recommend releasing these examples in parallel under your choice of

4143

free software license, such as the GNU General Public License,

4144

to permit their use in free software.

4145

4146

4147

@node Concept Index, , Copying, Top

4148

@unnumbered Concept Index

4149

@printindex cp

4150

4151

@contents

4152

4153

@bye