1
***********************************************************
2
HOWTO Fetch Internet Resources Using The urllib Package
3
***********************************************************
5
:Author: `Michael Foord <http://www.voidspace.org.uk/python/index.shtml>`_
9
There is a French translation of an earlier revision of this
10
HOWTO, available at `urllib2 - Le Manuel manquant
11
<http://www.voidspace.org.uk/python/articles/urllib2_francais.shtml>`_.
18
.. sidebar:: Related Articles
20
You may also find useful the following article on fetching web resources
23
* `Basic Authentication <http://www.voidspace.org.uk/python/articles/authentication.shtml>`_
25
A tutorial on *Basic Authentication*, with examples in Python.
27
**urllib.request** is a `Python <http://www.python.org>`_ module for fetching URLs
28
(Uniform Resource Locators). It offers a very simple interface, in the form of
29
the *urlopen* function. This is capable of fetching URLs using a variety of
30
different protocols. It also offers a slightly more complex interface for
31
handling common situations - like basic authentication, cookies, proxies and so
32
on. These are provided by objects called handlers and openers.
34
urllib.request supports fetching URLs for many "URL schemes" (identified by the string
35
before the ":" in URL - for example "ftp" is the URL scheme of
36
"ftp://python.org/") using their associated network protocols (e.g. FTP, HTTP).
37
This tutorial focuses on the most common case, HTTP.
39
For straightforward situations *urlopen* is very easy to use. But as soon as you
40
encounter errors or non-trivial cases when opening HTTP URLs, you will need some
41
understanding of the HyperText Transfer Protocol. The most comprehensive and
42
authoritative reference to HTTP is :rfc:`2616`. This is a technical document and
43
not intended to be easy to read. This HOWTO aims to illustrate using *urllib*,
44
with enough detail about HTTP to help you through. It is not intended to replace
45
the :mod:`urllib.request` docs, but is supplementary to them.
51
The simplest way to use urllib.request is as follows::
54
response = urllib.request.urlopen('http://python.org/')
55
html = response.read()
57
Many uses of urllib will be that simple (note that instead of an 'http:' URL we
58
could have used an URL starting with 'ftp:', 'file:', etc.). However, it's the
59
purpose of this tutorial to explain the more complicated cases, concentrating on
62
HTTP is based on requests and responses - the client makes requests and servers
63
send responses. urllib.request mirrors this with a ``Request`` object which represents
64
the HTTP request you are making. In its simplest form you create a Request
65
object that specifies the URL you want to fetch. Calling ``urlopen`` with this
66
Request object returns a response object for the URL requested. This response is
67
a file-like object, which means you can for example call ``.read()`` on the
72
req = urllib.request.Request('http://www.voidspace.org.uk')
73
response = urllib.request.urlopen(req)
74
the_page = response.read()
76
Note that urllib.request makes use of the same Request interface to handle all URL
77
schemes. For example, you can make an FTP request like so::
79
req = urllib.request.Request('ftp://example.com/')
81
In the case of HTTP, there are two extra things that Request objects allow you
82
to do: First, you can pass data to be sent to the server. Second, you can pass
83
extra information ("metadata") *about* the data or the about request itself, to
84
the server - this information is sent as HTTP "headers". Let's look at each of
90
Sometimes you want to send data to a URL (often the URL will refer to a CGI
91
(Common Gateway Interface) script [#]_ or other web application). With HTTP,
92
this is often done using what's known as a **POST** request. This is often what
93
your browser does when you submit a HTML form that you filled in on the web. Not
94
all POSTs have to come from forms: you can use a POST to transmit arbitrary data
95
to your own application. In the common case of HTML forms, the data needs to be
96
encoded in a standard way, and then passed to the Request object as the ``data``
97
argument. The encoding is done using a function from the :mod:`urllib.parse`
101
import urllib.request
103
url = 'http://www.someserver.com/cgi-bin/register.cgi'
104
values = {'name' : 'Michael Foord',
105
'location' : 'Northampton',
106
'language' : 'Python' }
108
data = urllib.parse.urlencode(values)
109
req = urllib.request.Request(url, data)
110
response = urllib.request.urlopen(req)
111
the_page = response.read()
113
Note that other encodings are sometimes required (e.g. for file upload from HTML
114
forms - see `HTML Specification, Form Submission
115
<http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13>`_ for more
118
If you do not pass the ``data`` argument, urllib uses a **GET** request. One
119
way in which GET and POST requests differ is that POST requests often have
120
"side-effects": they change the state of the system in some way (for example by
121
placing an order with the website for a hundredweight of tinned spam to be
122
delivered to your door). Though the HTTP standard makes it clear that POSTs are
123
intended to *always* cause side-effects, and GET requests *never* to cause
124
side-effects, nothing prevents a GET request from having side-effects, nor a
125
POST requests from having no side-effects. Data can also be passed in an HTTP
126
GET request by encoding it in the URL itself.
128
This is done as follows::
130
>>> import urllib.request
131
>>> import urllib.parse
133
>>> data['name'] = 'Somebody Here'
134
>>> data['location'] = 'Northampton'
135
>>> data['language'] = 'Python'
136
>>> url_values = urllib.parse.urlencode(data)
137
>>> print(url_values)
138
name=Somebody+Here&language=Python&location=Northampton
139
>>> url = 'http://www.example.com/example.cgi'
140
>>> full_url = url + '?' + url_values
141
>>> data = urllib.request.open(full_url)
143
Notice that the full URL is created by adding a ``?`` to the URL, followed by
149
We'll discuss here one particular HTTP header, to illustrate how to add headers
150
to your HTTP request.
152
Some websites [#]_ dislike being browsed by programs, or send different versions
153
to different browsers [#]_ . By default urllib identifies itself as
154
``Python-urllib/x.y`` (where ``x`` and ``y`` are the major and minor version
155
numbers of the Python release,
156
e.g. ``Python-urllib/2.5``), which may confuse the site, or just plain
157
not work. The way a browser identifies itself is through the
158
``User-Agent`` header [#]_. When you create a Request object you can
159
pass a dictionary of headers in. The following example makes the same
160
request as above, but identifies itself as a version of Internet
164
import urllib.request
166
url = 'http://www.someserver.com/cgi-bin/register.cgi'
167
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
168
values = {'name' : 'Michael Foord',
169
'location' : 'Northampton',
170
'language' : 'Python' }
171
headers = { 'User-Agent' : user_agent }
173
data = urllib.parse.urlencode(values)
174
req = urllib.request.Request(url, data, headers)
175
response = urllib.request.urlopen(req)
176
the_page = response.read()
178
The response also has two useful methods. See the section on `info and geturl`_
179
which comes after we have a look at what happens when things go wrong.
185
*urlopen* raises :exc:`URLError` when it cannot handle a response (though as usual
186
with Python APIs, builtin exceptions such as
187
:exc:`ValueError`, :exc:`TypeError` etc. may also
190
:exc:`HTTPError` is the subclass of :exc:`URLError` raised in the specific case of
193
The exception classes are exported from the :mod:`urllib.error` module.
198
Often, URLError is raised because there is no network connection (no route to
199
the specified server), or the specified server doesn't exist. In this case, the
200
exception raised will have a 'reason' attribute, which is a tuple containing an
201
error code and a text error message.
205
>>> req = urllib.request.Request('http://www.pretend_server.org')
206
>>> try: urllib.request.urlopen(req)
207
>>> except urllib.error.URLError, e:
210
(4, 'getaddrinfo failed')
216
Every HTTP response from the server contains a numeric "status code". Sometimes
217
the status code indicates that the server is unable to fulfil the request. The
218
default handlers will handle some of these responses for you (for example, if
219
the response is a "redirection" that requests the client fetch the document from
220
a different URL, urllib will handle that for you). For those it can't handle,
221
urlopen will raise an :exc:`HTTPError`. Typical errors include '404' (page not
222
found), '403' (request forbidden), and '401' (authentication required).
224
See section 10 of RFC 2616 for a reference on all the HTTP error codes.
226
The :exc:`HTTPError` instance raised will have an integer 'code' attribute, which
227
corresponds to the error sent by the server.
232
Because the default handlers handle redirects (codes in the 300 range), and
233
codes in the 100-299 range indicate success, you will usually only see error
234
codes in the 400-599 range.
236
:attr:`http.server.BaseHTTPRequestHandler.responses` is a useful dictionary of
237
response codes in that shows all the response codes used by RFC 2616. The
238
dictionary is reproduced here for convenience ::
240
# Table mapping response codes to messages; entries have the
241
# form {code: (shortmessage, longmessage)}.
243
100: ('Continue', 'Request received, please continue'),
244
101: ('Switching Protocols',
245
'Switching to new protocol; obey Upgrade header'),
247
200: ('OK', 'Request fulfilled, document follows'),
248
201: ('Created', 'Document created, URL follows'),
250
'Request accepted, processing continues off-line'),
251
203: ('Non-Authoritative Information', 'Request fulfilled from cache'),
252
204: ('No Content', 'Request fulfilled, nothing follows'),
253
205: ('Reset Content', 'Clear input form for further input.'),
254
206: ('Partial Content', 'Partial content follows.'),
256
300: ('Multiple Choices',
257
'Object has several resources -- see URI list'),
258
301: ('Moved Permanently', 'Object moved permanently -- see URI list'),
259
302: ('Found', 'Object moved temporarily -- see URI list'),
260
303: ('See Other', 'Object moved -- see Method and URL list'),
261
304: ('Not Modified',
262
'Document has not changed since given time'),
264
'You must use proxy specified in Location to access this '
266
307: ('Temporary Redirect',
267
'Object moved temporarily -- see URI list'),
270
'Bad request syntax or unsupported method'),
271
401: ('Unauthorized',
272
'No permission -- see authorization schemes'),
273
402: ('Payment Required',
274
'No payment -- see charging schemes'),
276
'Request forbidden -- authorization will not help'),
277
404: ('Not Found', 'Nothing matches the given URI'),
278
405: ('Method Not Allowed',
279
'Specified method is invalid for this server.'),
280
406: ('Not Acceptable', 'URI not available in preferred format.'),
281
407: ('Proxy Authentication Required', 'You must authenticate with '
282
'this proxy before proceeding.'),
283
408: ('Request Timeout', 'Request timed out; try again later.'),
284
409: ('Conflict', 'Request conflict.'),
286
'URI no longer exists and has been permanently removed.'),
287
411: ('Length Required', 'Client must specify Content-Length.'),
288
412: ('Precondition Failed', 'Precondition in headers is false.'),
289
413: ('Request Entity Too Large', 'Entity is too large.'),
290
414: ('Request-URI Too Long', 'URI is too long.'),
291
415: ('Unsupported Media Type', 'Entity body in unsupported format.'),
292
416: ('Requested Range Not Satisfiable',
293
'Cannot satisfy request range.'),
294
417: ('Expectation Failed',
295
'Expect condition could not be satisfied.'),
297
500: ('Internal Server Error', 'Server got itself in trouble'),
298
501: ('Not Implemented',
299
'Server does not support this operation'),
300
502: ('Bad Gateway', 'Invalid responses from another server/proxy.'),
301
503: ('Service Unavailable',
302
'The server cannot process the request due to a high load'),
303
504: ('Gateway Timeout',
304
'The gateway server did not receive a timely response'),
305
505: ('HTTP Version Not Supported', 'Cannot fulfill request.'),
308
When an error is raised the server responds by returning an HTTP error code
309
*and* an error page. You can use the :exc:`HTTPError` instance as a response on the
310
page returned. This means that as well as the code attribute, it also has read,
311
geturl, and info, methods as returned by the ``urllib.response`` module::
313
>>> req = urllib.request.Request('http://www.python.org/fish.html')
315
>>> urllib.request.urlopen(req)
316
>>> except urllib.error.URLError, e:
321
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
322
"http://www.w3.org/TR/html4/loose.dtd">
323
<?xml-stylesheet href="./css/ht2html.css"
325
<html><head><title>Error 404: File Not Found</title>
331
So if you want to be prepared for :exc:`HTTPError` *or* :exc:`URLError` there are two
332
basic approaches. I prefer the second approach.
340
from urllib.request import Request, urlopen
341
from urllib.error import URLError, HTTPError
342
req = Request(someurl)
344
response = urlopen(req)
346
print('The server couldn\'t fulfill the request.')
347
print('Error code: ', e.code)
349
print('We failed to reach a server.')
350
print('Reason: ', e.reason)
357
The ``except HTTPError`` *must* come first, otherwise ``except URLError``
358
will *also* catch an :exc:`HTTPError`.
365
from urllib.request import Request, urlopen
366
from urllib.error import URLError
367
req = Request(someurl)
369
response = urlopen(req)
371
if hasattr(e, 'reason'):
372
print('We failed to reach a server.')
373
print('Reason: ', e.reason)
374
elif hasattr(e, 'code'):
375
print('The server couldn\'t fulfill the request.')
376
print('Error code: ', e.code)
384
The response returned by urlopen (or the :exc:`HTTPError` instance) has two
385
useful methods :meth:`info` and :meth:`geturl` and is defined in the module
386
:mod:`urllib.response`..
388
**geturl** - this returns the real URL of the page fetched. This is useful
389
because ``urlopen`` (or the opener object used) may have followed a
390
redirect. The URL of the page fetched may not be the same as the URL requested.
392
**info** - this returns a dictionary-like object that describes the page
393
fetched, particularly the headers sent by the server. It is currently an
394
:class:`http.client.HTTPMessage` instance.
396
Typical headers include 'Content-length', 'Content-type', and so on. See the
397
`Quick Reference to HTTP Headers <http://www.cs.tut.fi/~jkorpela/http.html>`_
398
for a useful listing of HTTP headers with brief explanations of their meaning
405
When you fetch a URL you use an opener (an instance of the perhaps
406
confusingly-named :class:`urllib.request.OpenerDirector`). Normally we have been using
407
the default opener - via ``urlopen`` - but you can create custom
408
openers. Openers use handlers. All the "heavy lifting" is done by the
409
handlers. Each handler knows how to open URLs for a particular URL scheme (http,
410
ftp, etc.), or how to handle an aspect of URL opening, for example HTTP
411
redirections or HTTP cookies.
413
You will want to create openers if you want to fetch URLs with specific handlers
414
installed, for example to get an opener that handles cookies, or to get an
415
opener that does not handle redirections.
417
To create an opener, instantiate an ``OpenerDirector``, and then call
418
``.add_handler(some_handler_instance)`` repeatedly.
420
Alternatively, you can use ``build_opener``, which is a convenience function for
421
creating opener objects with a single function call. ``build_opener`` adds
422
several handlers by default, but provides a quick way to add more and/or
423
override the default handlers.
425
Other sorts of handlers you might want to can handle proxies, authentication,
426
and other common but slightly specialised situations.
428
``install_opener`` can be used to make an ``opener`` object the (global) default
429
opener. This means that calls to ``urlopen`` will use the opener you have
432
Opener objects have an ``open`` method, which can be called directly to fetch
433
urls in the same way as the ``urlopen`` function: there's no need to call
434
``install_opener``, except as a convenience.
440
To illustrate creating and installing a handler we will use the
441
``HTTPBasicAuthHandler``. For a more detailed discussion of this subject --
442
including an explanation of how Basic Authentication works - see the `Basic
443
Authentication Tutorial
444
<http://www.voidspace.org.uk/python/articles/authentication.shtml>`_.
446
When authentication is required, the server sends a header (as well as the 401
447
error code) requesting authentication. This specifies the authentication scheme
448
and a 'realm'. The header looks like : ``Www-authenticate: SCHEME
453
Www-authenticate: Basic realm="cPanel Users"
456
The client should then retry the request with the appropriate name and password
457
for the realm included as a header in the request. This is 'basic
458
authentication'. In order to simplify this process we can create an instance of
459
``HTTPBasicAuthHandler`` and an opener to use this handler.
461
The ``HTTPBasicAuthHandler`` uses an object called a password manager to handle
462
the mapping of URLs and realms to passwords and usernames. If you know what the
463
realm is (from the authentication header sent by the server), then you can use a
464
``HTTPPasswordMgr``. Frequently one doesn't care what the realm is. In that
465
case, it is convenient to use ``HTTPPasswordMgrWithDefaultRealm``. This allows
466
you to specify a default username and password for a URL. This will be supplied
467
in the absence of you providing an alternative combination for a specific
468
realm. We indicate this by providing ``None`` as the realm argument to the
469
``add_password`` method.
471
The top-level URL is the first URL that requires authentication. URLs "deeper"
472
than the URL you pass to .add_password() will also match. ::
474
# create a password manager
475
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
477
# Add the username and password.
478
# If we knew the realm, we could use it instead of None.
479
top_level_url = "http://example.com/foo/"
480
password_mgr.add_password(None, top_level_url, username, password)
482
handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
484
# create "opener" (OpenerDirector instance)
485
opener = urllib.request.build_opener(handler)
487
# use the opener to fetch a URL
490
# Install the opener.
491
# Now all calls to urllib.request.urlopen use our opener.
492
urllib.request.install_opener(opener)
496
In the above example we only supplied our ``HHTPBasicAuthHandler`` to
497
``build_opener``. By default openers have the handlers for normal situations
498
-- ``ProxyHandler``, ``UnknownHandler``, ``HTTPHandler``,
499
``HTTPDefaultErrorHandler``, ``HTTPRedirectHandler``, ``FTPHandler``,
500
``FileHandler``, ``HTTPErrorProcessor``.
502
``top_level_url`` is in fact *either* a full URL (including the 'http:' scheme
503
component and the hostname and optionally the port number)
504
e.g. "http://example.com/" *or* an "authority" (i.e. the hostname,
505
optionally including the port number) e.g. "example.com" or "example.com:8080"
506
(the latter example includes a port number). The authority, if present, must
507
NOT contain the "userinfo" component - for example "joe@password:example.com" is
514
**urllib** will auto-detect your proxy settings and use those. This is through
515
the ``ProxyHandler`` which is part of the normal handler chain. Normally that's
516
a good thing, but there are occasions when it may not be helpful [#]_. One way
517
to do this is to setup our own ``ProxyHandler``, with no proxies defined. This
518
is done using similar steps to setting up a `Basic Authentication`_ handler : ::
520
>>> proxy_support = urllib.request.ProxyHandler({})
521
>>> opener = urllib.request.build_opener(proxy_support)
522
>>> urllib.request.install_opener(opener)
526
Currently ``urllib.request`` *does not* support fetching of ``https`` locations
527
through a proxy. However, this can be enabled by extending urllib.request as
528
shown in the recipe [#]_.
534
The Python support for fetching resources from the web is layered. urllib uses
535
the :mod:`http.client` library, which in turn uses the socket library.
537
As of Python 2.3 you can specify how long a socket should wait for a response
538
before timing out. This can be useful in applications which have to fetch web
539
pages. By default the socket module has *no timeout* and can hang. Currently,
540
the socket timeout is not exposed at the http.client or urllib.request levels.
541
However, you can set the default timeout globally for all sockets using ::
544
import urllib.request
548
socket.setdefaulttimeout(timeout)
550
# this call to urllib.request.urlopen now uses the default timeout
551
# we have set in the socket module
552
req = urllib.request.Request('http://www.voidspace.org.uk')
553
response = urllib.request.urlopen(req)
562
This document was reviewed and revised by John Lee.
564
.. [#] For an introduction to the CGI protocol see
565
`Writing Web Applications in Python <http://www.pyzine.com/Issue008/Section_Articles/article_CGIOne.html>`_.
566
.. [#] Like Google for example. The *proper* way to use google from a program
567
is to use `PyGoogle <http://pygoogle.sourceforge.net>`_ of course. See
568
`Voidspace Google <http://www.voidspace.org.uk/python/recipebook.shtml#google>`_
569
for some examples of using the Google API.
570
.. [#] Browser sniffing is a very bad practise for website design - building
571
sites using web standards is much more sensible. Unfortunately a lot of
572
sites still send different versions to different browsers.
573
.. [#] The user agent for MSIE 6 is
574
*'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)'*
575
.. [#] For details of more HTTP request headers, see
576
`Quick Reference to HTTP Headers`_.
577
.. [#] In my case I have to use a proxy to access the internet at work. If you
578
attempt to fetch *localhost* URLs through this proxy it blocks them. IE
579
is set to use the proxy, which urllib picks up on. In order to test
580
scripts with a localhost server, I have to prevent urllib from using
582
.. [#] urllib opener for SSL proxy (CONNECT method): `ASPN Cookbook Recipe
583
<http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/456195>`_.
added
removed
Doc/howto/urllib2.rst
