~ntt-pf-lab/nova/monkey_patch_notification

<?xml version="1.0" encoding="utf-8"?><!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'><html lang="en" xmlns="http://www.w3.org/1999/xhtml">

<head>

<title>Twisted Documentation:

Using the Twisted Web Client

</title>

</head>

Using the Twisted Web Client

</h1>

Overview

</a></li><ul><li><a href="#auto1">

Prerequisites

</a></li></ul><li><a href="#auto2">

The Agent

</a></li><ul><li><a href="#auto3">Issuing Requests</a></li><li><a href="#auto4">

Receiving Responses

</a></li></ul><li><a href="#auto5">

Conclusion

</a></li></ol></div>

<h2>

Overview

This document describes how to use the HTTP client included in Twisted

Web. After reading it, you should be able to make HTTP requests using

Twisted Web. You will be able to specify the request method, headers,

and body and you will be able to retrieve the response code, headers, and

body.

<h3>

Prerequisites

This document assumes that you are familiar with <a href="../../core/howto/defer.html" shape="rect">Deferreds and Failures</a>, and <a href="../../core/howto/producers.html" shape="rect">producers and consumers</a>.

It also assumes you are familiar with the basic concepts of HTTP, such

as requests and responses, methods, headers, and message bodies.

<h2>

The Agent

<h3>Issuing Requests<a name="auto3"/></h3>

The <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.web.client.Agent.html" title="twisted.web.client.Agent">twisted.web.client.Agent</a></code> class is the entry

point into the client API. Requests are issued using the <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.web.client.Agent.request.html" title="twisted.web.client.Agent.request">request</a></code> method, which

takes as parameters a request method, a request URI, the request headers,

and an object which can produce the request body (if there is to be one).

The agent is responsible for connection setup. Because of this, it

requires a reactor as an argument to its initializer. An example of

creating an agent and issuing a request using it might look like this:

<div class="py-listing"><pre> 1

from twisted.internet import reactor

from twisted.web.client import Agent

from twisted.web.http_headers import Headers

agent = Agent(reactor)

d = agent.request(

'GET',

'http://example.com/',

Headers({'User-Agent': ['Twisted Web Client Example']}),

None)

def cbResponse(ignored):

print 'Response received'

100

d.addCallback(cbResponse)

101

102

def cbShutdown(ignored):

103

reactor.stop()

104

d.addBoth(cbShutdown)

105

106

reactor.run()

107

</pre><div class="caption">

108

Issue a request with an Agent

109

- <a href="listings/client/request.py">listings/client/request.py</a></div></div>

110

111

112

As may be obvious, this issues a new GET request for /

113

to the web server on <code>example.com</code>. <code>Agent</code> is

114

responsible for resolving the hostname into an IP address and connecting

115

to it on port 80. It is also responsible for cleaning up the connection

116

afterwards. This code sends a request which includes one custom header,

117

User-Agent. The last argument passed to <code>Agent.request</code> is

118

<code>None</code>, though, so the request has no body.

119

120

121

122

Sending a request which does include a body requires passing an object

123

providing <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.web.iweb.IBodyProducer.html" title="twisted.web.iweb.IBodyProducer">twisted.web.iweb.IBodyProducer</a></code>

124

to <code>Agent.request</code>. This interface extends the more general

125

<code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.internet.interfaces.IPushProducer.html" title="twisted.internet.interfaces.IPushProducer">IPushProducer</a></code>

126

by adding a new <code>length</code> attribute and adding several

127

constraints to the way the producer and consumer interact.

128

129

130

<ul>

131

<li>

132

The length attribute must be a non-negative integer or the constant

133

<code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.web.iweb.UNKNOWN_LENGTH.html" title="twisted.web.iweb.UNKNOWN_LENGTH">twisted.web.iweb.UNKNOWN_LENGTH</a></code>. If the

134

length is known, it will be used to specify the value for the

135

Content-Length header in the request. If the length is

136

unknown the attribute should be set to <code>UNKNOWN_LENGTH</code>.

137

Since more servers support Content-Length, if a length can be

138

provided it should be.

139

</li>

140

141

<li>

142

An additional method is required on <code>IEntityBodyProvider</code>

143

implementations: <code>startProducing</code>. This method is used to

144

associate a consumer with the producer. It should return a

145

<code>Deferred</code> which fires when all data has been produced.

146

</li>

147

148

<li>

149

<code>IEntityBodyProvider</code> implementations should never call the

150

consumer's <code>unregisterProducer</code> method. Instead, when it

151

has produced all of the data it is going to produce, it should only

152

fire the <code>Deferred</code> returned by <code>startProducing</code>.

153

</li>

154

</ul>

155

156

157

For additional details about the requirements of <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.web.iweb.IBodyProducer.html" title="twisted.web.iweb.IBodyProducer">IBodyProducer</a></code> implementations, see

158

the API documentation.

159

160

161

162

Here's a simple <code>IEntityBodyProvider</code> implementation which

163

writes an in-memory string to the consumer:

164

165

166

<div class="py-listing"><pre> 1

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

from zope.interface import implements

188

189

from twisted.internet.defer import succeed

190

from twisted.web.iweb import IBodyProducer

191

192

class StringProducer(object):

193

implements(IBodyProducer)

194

195

def __init__(self, body):

196

self.body = body

197

self.length = len(body)

198

199

def startProducing(self, consumer):

200

consumer.write(self.body)

201

return succeed(None)

202

203

def pauseProducing(self):

204

pass

205

206

def stopProducing(self):

207

pass

208

</pre><div class="caption">

209

A string-based body producer.

210

- <a href="listings/client/stringprod.py">listings/client/stringprod.py</a></div></div>

211

212

213

This producer can be used to issue a request with a body:

214

215

216

<div class="py-listing"><pre> 1

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

from twisted.web.http_headers import Headers

243

244

from stringprod import StringProducer

245

246

agent = Agent(reactor)

247

body = StringProducer("hello, world")

248

d = agent.request(

249

'GET',

250

'http://example.com/',

251

Headers({'User-Agent': ['Twisted Web Client Example'],

252

'Content-Type': ['text/x-greeting']}),

253

body)

254

255

def cbResponse(ignored):

256

print 'Response received'

257

d.addCallback(cbResponse)

258

259

def cbShutdown(ignored):

260

reactor.stop()

261

d.addBoth(cbShutdown)

262

263

reactor.run()

264

</pre><div class="caption">

265

Issue a request with a body.

266

- <a href="listings/client/sendbody.py">listings/client/sendbody.py</a></div></div>

267

268

<h3>

269

Receiving Responses

270

271

272

273

So far, the examples have demonstrated how to issue a request. However,

274

they have ignored the response, except for showing that it is a

275

<code>Deferred</code> which seems to fire when the response has been

276

received. Next we'll cover what that response is and how to interpret

277

it.

278

279

280

281

<code>Agent.request</code>, as with most <code>Deferred</code>-returning

282

APIs, can return a <code>Deferred</code> which fires with a

283

<code>Failure</code>. If the request fails somehow, this will be

284

reflected with a failure. This may be due to a problem looking up the

285

host IP address, or it may be because the HTTP server is not accepting

286

connections, or it may be because of a problem parsing the response, or

287

any other problem which arises which prevents the response from being

288

received. It does not include responses with an error status.

289

290

291

292

If the request succeeds, though, the <code>Deferred</code> will fire with

293

a <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.web.client.Response.html" title="twisted.web.client.Response">Response</a></code>. This

294

happens as soon as all the response headers have been received. It

295

happens before any of the response body, if there is one, is processed.

296

The <code>Response</code> object has several attributes giving the

297

response information: its code, version, phrase, and headers, as well as

298

the length of the body to expect. The <code>Response</code> object also

299

has a method which makes the response body available: <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.web._newclient.Response.deliverBody.deliverBody.html" title="twisted.web._newclient.Response.deliverBody.deliverBody">deliverBody</a></code>.

300

Using the attributes of the response object and this method, here's an

301

example which displays part of the response to a request:

302

303

304

<div class="py-listing"><pre> 1

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

from pprint import pformat

352

353

from twisted.internet import reactor

354

355

from twisted.internet.protocol import Protocol

356

357

from twisted.web.http_headers import Headers

358

359

class BeginningPrinter(Protocol):

360

def __init__(self, finished):

361

self.finished = finished

362

self.remaining = 1024 * 10

363

364

def dataReceived(self, bytes):

365

if self.remaining:

366

display = bytes[:self.remaining]

367

print 'Some data received:'

368

print display

369

self.remaining -= len(display)

370

371

def connectionLost(self, reason):

372

print 'Finished receiving body:', reason.getErrorMessage()

373

self.finished.callback(None)

374

375

agent = Agent(reactor)

376

d = agent.request(

377

'GET',

378

'http://example.com/',

379

Headers({'User-Agent': ['Twisted Web Client Example']}),

380

None)

381

382

def cbRequest(response):

383

print 'Response version:', response.version

384

print 'Response code:', response.code

385

print 'Response phrase:', response.phrase

386

print 'Response headers:'

387

print pformat(list(response.headers.getAllRawHeaders()))

388

finished = Deferred()

389

response.deliverBody(BeginningPrinter(finished))

390

return finished

391

d.addCallback(cbRequest)

392

393

def cbShutdown(ignored):

394

reactor.stop()

395

d.addBoth(cbShutdown)

396

397

reactor.run()

398

</pre><div class="caption">

399

Inspect the response.

400

- <a href="listings/client/response.py">listings/client/response.py</a></div></div>

401

402

403

The <code>BeginningPrinter</code> protocol in this example is passed to

404

<code>Response.deliverBody</code> and the response body is then delivered

405

to its <code>dataReceived</code> method as it arrives. When the body has

406

been completely delivered, the protocol's <code>connectionLost</code>

407

method is called. It is important to inspect the <code>Failure</code>

408

passed to <code>connectionLost</code>. If the response body has been

409

completely received, the failure will wrap a <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.web.client.ResponseDone.html" title="twisted.web.client.ResponseDone">twisted.web.client.ResponseDone</a></code> exception. This

410

indicates that it is known that all data has been received. It

411

is also possible for the failure to wrap a <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.web.http.PotentialDataLoss.html" title="twisted.web.http.PotentialDataLoss">twisted.web.http.PotentialDataLoss</a></code> exception: this

412

indicates that the server framed the response such that there is no way

413

to know when the entire response body has been received. Only

414

HTTP/1.0 servers should behave this way. Finally, it is possible for

415

the exception to be of another type, indicating guaranteed data loss for

416

some reason (a lost connection, a memory error, etc).

417

418

419

420

Just as protocols associated with a TCP connection are given a transport,

421

so will be a protocol passed to <code>deliverBody</code>. Since it makes

422

no sense to write more data to the connection at this stage of the

423

request, though, the transport only provides <code class="API"><a href="http://twistedmatrix.com/documents/10.0.0/api/twisted.internet.interfaces.IPushProducer.html" title="twisted.internet.interfaces.IPushProducer">IPushProducer</a></code>. This allows the

424

protocol to control the flow of the response data: a call to the

425

transport's <code>pauseProducing</code> method will pause delivery; a

426

later call to <code>resumeProducing</code> will resume it. If it is

427

decided that the rest of the response body is not desired,

428

<code>stopProducing</code> can be used to stop delivery permanently;

429

after this, the protocol's <code>connectionLost</code> method will be

430

called.

431

432

433

434

An important thing to keep in mind is that the body will only be read

435

from the connection after <code>Response.deliverBody</code> is called.

436

This also means that the connection will remain open until this is done

437

(and the body read). So, in general, any response with a body

438

must have that body read using <code>deliverBody</code>. If the

439

application is not interested in the body, it should issue a

440

HEAD request or use a protocol which immediately calls

441

<code>stopProducing</code> on its transport.

442

443

444

<h2>

445

Conclusion

446

447

448

449

You should now understand the basics of the Twisted Web HTTP client. In

450

particular, you should understand:

451

452

453

<ul>

454

<li>

455

How to issue requests with arbitrary methods, headers, and bodies.

456

</li>

457

<li>

458

How to access the response version, code, phrase, headers, and body.

459

</li>

460

<li>

461

How to control the streaming of the response body.

462

</li>

463

</ul>

464

</div>

465

466

<a href="index.html">Index</a>

467

Version: 10.0.0

468

</body>

469

</html>

b'\\ No newline at end of file'

Older »