1
********************************
2
Functional Programming HOWTO
3
********************************
5
:Author: A. M. Kuchling
8
(This is a first draft. Please send comments/error reports/suggestions to
11
In this document, we'll take a tour of Python's features suitable for
12
implementing programs in a functional style. After an introduction to the
13
concepts of functional programming, we'll look at language features such as
14
:term:`iterator`\s and :term:`generator`\s and relevant library modules such as
15
:mod:`itertools` and :mod:`functools`.
21
This section explains the basic concept of functional programming; if you're
22
just interested in learning about Python language features, skip to the next
25
Programming languages support decomposing problems in several different ways:
27
* Most programming languages are **procedural**: programs are lists of
28
instructions that tell the computer what to do with the program's input. C,
29
Pascal, and even Unix shells are procedural languages.
31
* In **declarative** languages, you write a specification that describes the
32
problem to be solved, and the language implementation figures out how to
33
perform the computation efficiently. SQL is the declarative language you're
34
most likely to be familiar with; a SQL query describes the data set you want
35
to retrieve, and the SQL engine decides whether to scan tables or use indexes,
36
which subclauses should be performed first, etc.
38
* **Object-oriented** programs manipulate collections of objects. Objects have
39
internal state and support methods that query or modify this internal state in
40
some way. Smalltalk and Java are object-oriented languages. C++ and Python
41
are languages that support object-oriented programming, but don't force the
42
use of object-oriented features.
44
* **Functional** programming decomposes a problem into a set of functions.
45
Ideally, functions only take inputs and produce outputs, and don't have any
46
internal state that affects the output produced for a given input. Well-known
47
functional languages include the ML family (Standard ML, OCaml, and other
48
variants) and Haskell.
50
The designers of some computer languages choose to emphasize one
51
particular approach to programming. This often makes it difficult to
52
write programs that use a different approach. Other languages are
53
multi-paradigm languages that support several different approaches.
54
Lisp, C++, and Python are multi-paradigm; you can write programs or
55
libraries that are largely procedural, object-oriented, or functional
56
in all of these languages. In a large program, different sections
57
might be written using different approaches; the GUI might be
58
object-oriented while the processing logic is procedural or
59
functional, for example.
61
In a functional program, input flows through a set of functions. Each function
62
operates on its input and produces some output. Functional style discourages
63
functions with side effects that modify internal state or make other changes
64
that aren't visible in the function's return value. Functions that have no side
65
effects at all are called **purely functional**. Avoiding side effects means
66
not using data structures that get updated as a program runs; every function's
67
output must only depend on its input.
69
Some languages are very strict about purity and don't even have assignment
70
statements such as ``a=3`` or ``c = a + b``, but it's difficult to avoid all
71
side effects. Printing to the screen or writing to a disk file are side
72
effects, for example. For example, in Python a call to the :func:`print` or
73
:func:`time.sleep` function both return no useful value; they're only called for
74
their side effects of sending some text to the screen or pausing execution for a
77
Python programs written in functional style usually won't go to the extreme of
78
avoiding all I/O or all assignments; instead, they'll provide a
79
functional-appearing interface but will use non-functional features internally.
80
For example, the implementation of a function will still use assignments to
81
local variables, but won't modify global variables or have other side effects.
83
Functional programming can be considered the opposite of object-oriented
84
programming. Objects are little capsules containing some internal state along
85
with a collection of method calls that let you modify this state, and programs
86
consist of making the right set of state changes. Functional programming wants
87
to avoid state changes as much as possible and works with data flowing between
88
functions. In Python you might combine the two approaches by writing functions
89
that take and return instances representing objects in your application (e-mail
90
messages, transactions, etc.).
92
Functional design may seem like an odd constraint to work under. Why should you
93
avoid objects and side effects? There are theoretical and practical advantages
94
to the functional style:
99
* Ease of debugging and testing.
105
A theoretical benefit is that it's easier to construct a mathematical proof that
106
a functional program is correct.
108
For a long time researchers have been interested in finding ways to
109
mathematically prove programs correct. This is different from testing a program
110
on numerous inputs and concluding that its output is usually correct, or reading
111
a program's source code and concluding that the code looks right; the goal is
112
instead a rigorous proof that a program produces the right result for all
115
The technique used to prove programs correct is to write down **invariants**,
116
properties of the input data and of the program's variables that are always
117
true. For each line of code, you then show that if invariants X and Y are true
118
**before** the line is executed, the slightly different invariants X' and Y' are
119
true **after** the line is executed. This continues until you reach the end of
120
the program, at which point the invariants should match the desired conditions
121
on the program's output.
123
Functional programming's avoidance of assignments arose because assignments are
124
difficult to handle with this technique; assignments can break invariants that
125
were true before the assignment without producing any new invariants that can be
128
Unfortunately, proving programs correct is largely impractical and not relevant
129
to Python software. Even trivial programs require proofs that are several pages
130
long; the proof of correctness for a moderately complicated program would be
131
enormous, and few or none of the programs you use daily (the Python interpreter,
132
your XML parser, your web browser) could be proven correct. Even if you wrote
133
down or generated a proof, there would then be the question of verifying the
134
proof; maybe there's an error in it, and you wrongly believe you've proved the
141
A more practical benefit of functional programming is that it forces you to
142
break apart your problem into small pieces. Programs are more modular as a
143
result. It's easier to specify and write a small function that does one thing
144
than a large function that performs a complicated transformation. Small
145
functions are also easier to read and to check for errors.
148
Ease of debugging and testing
149
-----------------------------
151
Testing and debugging a functional-style program is easier.
153
Debugging is simplified because functions are generally small and clearly
154
specified. When a program doesn't work, each function is an interface point
155
where you can check that the data are correct. You can look at the intermediate
156
inputs and outputs to quickly isolate the function that's responsible for a bug.
158
Testing is easier because each function is a potential subject for a unit test.
159
Functions don't depend on system state that needs to be replicated before
160
running a test; instead you only have to synthesize the right input and then
161
check that the output matches expectations.
167
As you work on a functional-style program, you'll write a number of functions
168
with varying inputs and outputs. Some of these functions will be unavoidably
169
specialized to a particular application, but others will be useful in a wide
170
variety of programs. For example, a function that takes a directory path and
171
returns all the XML files in the directory, or a function that takes a filename
172
and returns its contents, can be applied to many different situations.
174
Over time you'll form a personal library of utilities. Often you'll assemble
175
new programs by arranging existing functions in a new configuration and writing
176
a few functions specialized for the current task.
182
I'll start by looking at a Python language feature that's an important
183
foundation for writing functional-style programs: iterators.
185
An iterator is an object representing a stream of data; this object returns the
186
data one element at a time. A Python iterator must support a method called
187
``__next__()`` that takes no arguments and always returns the next element of
188
the stream. If there are no more elements in the stream, ``__next__()`` must
189
raise the ``StopIteration`` exception. Iterators don't have to be finite,
190
though; it's perfectly reasonable to write an iterator that produces an infinite
193
The built-in :func:`iter` function takes an arbitrary object and tries to return
194
an iterator that will return the object's contents or elements, raising
195
:exc:`TypeError` if the object doesn't support iteration. Several of Python's
196
built-in data types support iteration, the most common being lists and
197
dictionaries. An object is called an **iterable** object if you can get an
200
You can experiment with the iteration interface manually:
205
<...iterator object at ...>
213
Traceback (most recent call last):
214
File "<stdin>", line 1, in ?
218
Python expects iterable objects in several different contexts, the most
219
important being the ``for`` statement. In the statement ``for X in Y``, Y must
220
be an iterator or some object for which ``iter()`` can create an iterator.
221
These two statements are equivalent::
230
Iterators can be materialized as lists or tuples by using the :func:`list` or
231
:func:`tuple` constructor functions:
234
>>> iterator = iter(L)
235
>>> t = tuple(iterator)
239
Sequence unpacking also supports iterators: if you know an iterator will return
240
N elements, you can unpack them into an N-tuple:
243
>>> iterator = iter(L)
248
Built-in functions such as :func:`max` and :func:`min` can take a single
249
iterator argument and will return the largest or smallest element. The ``"in"``
250
and ``"not in"`` operators also support iterators: ``X in iterator`` is true if
251
X is found in the stream returned by the iterator. You'll run into obvious
252
problems if the iterator is infinite; ``max()``, ``min()``, and ``"not in"``
253
will never return, and if the element X never appears in the stream, the
254
``"in"`` operator won't return either.
256
Note that you can only go forward in an iterator; there's no way to get the
257
previous element, reset the iterator, or make a copy of it. Iterator objects
258
can optionally provide these additional capabilities, but the iterator protocol
259
only specifies the ``next()`` method. Functions may therefore consume all of
260
the iterator's output, and if you need to do something different with the same
261
stream, you'll have to create a new iterator.
265
Data Types That Support Iterators
266
---------------------------------
268
We've already seen how lists and tuples support iterators. In fact, any Python
269
sequence type, such as strings, will automatically support creation of an
272
Calling :func:`iter` on a dictionary returns an iterator that will loop over the
275
.. not a doctest since dict ordering varies across Pythons
279
>>> m = {'Jan': 1, 'Feb': 2, 'Mar': 3, 'Apr': 4, 'May': 5, 'Jun': 6,
280
... 'Jul': 7, 'Aug': 8, 'Sep': 9, 'Oct': 10, 'Nov': 11, 'Dec': 12}
282
... print(key, m[key])
296
Note that the order is essentially random, because it's based on the hash
297
ordering of the objects in the dictionary.
299
Applying :func:`iter` to a dictionary always loops over the keys, but
300
dictionaries have methods that return other iterators. If you want to iterate
301
over values or key/value pairs, you can explicitly call the
302
:meth:`values` or :meth:`items` methods to get an appropriate iterator.
304
The :func:`dict` constructor can accept an iterator that returns a finite stream
305
of ``(key, value)`` tuples:
307
>>> L = [('Italy', 'Rome'), ('France', 'Paris'), ('US', 'Washington DC')]
309
{'Italy': 'Rome', 'US': 'Washington DC', 'France': 'Paris'}
311
Files also support iteration by calling the ``readline()`` method until there
312
are no more lines in the file. This means you can read each line of a file like
316
# do something for each line
319
Sets can take their contents from an iterable and let you iterate over the set's
322
S = {2, 3, 5, 7, 11, 13}
328
Generator expressions and list comprehensions
329
=============================================
331
Two common operations on an iterator's output are 1) performing some operation
332
for every element, 2) selecting a subset of elements that meet some condition.
333
For example, given a list of strings, you might want to strip off trailing
334
whitespace from each line or extract all the strings containing a given
337
List comprehensions and generator expressions (short form: "listcomps" and
338
"genexps") are a concise notation for such operations, borrowed from the
339
functional programming language Haskell (http://www.haskell.org). You can strip
340
all the whitespace from a stream of strings with the following code::
342
line_list = [' line 1\n', 'line 2 \n', ...]
344
# Generator expression -- returns iterator
345
stripped_iter = (line.strip() for line in line_list)
347
# List comprehension -- returns list
348
stripped_list = [line.strip() for line in line_list]
350
You can select only certain elements by adding an ``"if"`` condition::
352
stripped_list = [line.strip() for line in line_list
355
With a list comprehension, you get back a Python list; ``stripped_list`` is a
356
list containing the resulting lines, not an iterator. Generator expressions
357
return an iterator that computes the values as necessary, not needing to
358
materialize all the values at once. This means that list comprehensions aren't
359
useful if you're working with iterators that return an infinite stream or a very
360
large amount of data. Generator expressions are preferable in these situations.
362
Generator expressions are surrounded by parentheses ("()") and list
363
comprehensions are surrounded by square brackets ("[]"). Generator expressions
366
( expression for expr in sequence1
368
for expr2 in sequence2
370
for expr3 in sequence3 ...
372
for exprN in sequenceN
375
Again, for a list comprehension only the outside brackets are different (square
376
brackets instead of parentheses).
378
The elements of the generated output will be the successive values of
379
``expression``. The ``if`` clauses are all optional; if present, ``expression``
380
is only evaluated and added to the result when ``condition`` is true.
382
Generator expressions always have to be written inside parentheses, but the
383
parentheses signalling a function call also count. If you want to create an
384
iterator that will be immediately passed to a function you can write::
386
obj_total = sum(obj.count for obj in list_all_objects())
388
The ``for...in`` clauses contain the sequences to be iterated over. The
389
sequences do not have to be the same length, because they are iterated over from
390
left to right, **not** in parallel. For each element in ``sequence1``,
391
``sequence2`` is looped over from the beginning. ``sequence3`` is then looped
392
over for each resulting pair of elements from ``sequence1`` and ``sequence2``.
394
To put it another way, a list comprehension or generator expression is
395
equivalent to the following Python code::
397
for expr1 in sequence1:
399
continue # Skip this element
400
for expr2 in sequence2:
402
continue # Skip this element
404
for exprN in sequenceN:
406
continue # Skip this element
408
# Output the value of
411
This means that when there are multiple ``for...in`` clauses but no ``if``
412
clauses, the length of the resulting output will be equal to the product of the
413
lengths of all the sequences. If you have two lists of length 3, the output
414
list is 9 elements long:
417
:options: +NORMALIZE_WHITESPACE
421
>>> [(x,y) for x in seq1 for y in seq2]
422
[('a', 1), ('a', 2), ('a', 3),
423
('b', 1), ('b', 2), ('b', 3),
424
('c', 1), ('c', 2), ('c', 3)]
426
To avoid introducing an ambiguity into Python's grammar, if ``expression`` is
427
creating a tuple, it must be surrounded with parentheses. The first list
428
comprehension below is a syntax error, while the second one is correct::
431
[ x,y for x in seq1 for y in seq2]
433
[ (x,y) for x in seq1 for y in seq2]
439
Generators are a special class of functions that simplify the task of writing
440
iterators. Regular functions compute a value and return it, but generators
441
return an iterator that returns a stream of values.
443
You're doubtless familiar with how regular function calls work in Python or C.
444
When you call a function, it gets a private namespace where its local variables
445
are created. When the function reaches a ``return`` statement, the local
446
variables are destroyed and the value is returned to the caller. A later call
447
to the same function creates a new private namespace and a fresh set of local
448
variables. But, what if the local variables weren't thrown away on exiting a
449
function? What if you could later resume the function where it left off? This
450
is what generators provide; they can be thought of as resumable functions.
452
Here's the simplest example of a generator function:
456
def generate_ints(N):
460
Any function containing a ``yield`` keyword is a generator function; this is
461
detected by Python's :term:`bytecode` compiler which compiles the function
462
specially as a result.
464
When you call a generator function, it doesn't return a single value; instead it
465
returns a generator object that supports the iterator protocol. On executing
466
the ``yield`` expression, the generator outputs the value of ``i``, similar to a
467
``return`` statement. The big difference between ``yield`` and a ``return``
468
statement is that on reaching a ``yield`` the generator's state of execution is
469
suspended and local variables are preserved. On the next call to the
470
generator's ``.__next__()`` method, the function will resume executing.
472
Here's a sample usage of the ``generate_ints()`` generator:
474
>>> gen = generate_ints(3)
476
<generator object at ...>
484
Traceback (most recent call last):
485
File "stdin", line 1, in ?
486
File "stdin", line 2, in generate_ints
489
You could equally write ``for i in generate_ints(5)``, or ``a,b,c =
492
Inside a generator function, the ``return`` statement can only be used without a
493
value, and signals the end of the procession of values; after executing a
494
``return`` the generator cannot return any further values. ``return`` with a
495
value, such as ``return 5``, is a syntax error inside a generator function. The
496
end of the generator's results can also be indicated by raising
497
``StopIteration`` manually, or by just letting the flow of execution fall off
498
the bottom of the function.
500
You could achieve the effect of generators manually by writing your own class
501
and storing all the local variables of the generator as instance variables. For
502
example, returning a list of integers could be done by setting ``self.count`` to
503
0, and having the ``__next__()`` method increment ``self.count`` and return it.
504
However, for a moderately complicated generator, writing a corresponding class
507
The test suite included with Python's library, ``test_generators.py``, contains
508
a number of more interesting examples. Here's one generator that implements an
509
in-order traversal of a tree using generators recursively. ::
511
# A recursive generator that generates Tree leaves in in-order.
514
for x in inorder(t.left):
519
for x in inorder(t.right):
522
Two other examples in ``test_generators.py`` produce solutions for the N-Queens
523
problem (placing N queens on an NxN chess board so that no queen threatens
524
another) and the Knight's Tour (finding a route that takes a knight to every
525
square of an NxN chessboard without visiting any square twice).
529
Passing values into a generator
530
-------------------------------
532
In Python 2.4 and earlier, generators only produced output. Once a generator's
533
code was invoked to create an iterator, there was no way to pass any new
534
information into the function when its execution is resumed. You could hack
535
together this ability by making the generator look at a global variable or by
536
passing in some mutable object that callers then modify, but these approaches
539
In Python 2.5 there's a simple way to pass values into a generator.
540
:keyword:`yield` became an expression, returning a value that can be assigned to
541
a variable or otherwise operated on::
545
I recommend that you **always** put parentheses around a ``yield`` expression
546
when you're doing something with the returned value, as in the above example.
547
The parentheses aren't always necessary, but it's easier to always add them
548
instead of having to remember when they're needed.
550
(PEP 342 explains the exact rules, which are that a ``yield``-expression must
551
always be parenthesized except when it occurs at the top-level expression on the
552
right-hand side of an assignment. This means you can write ``val = yield i``
553
but have to use parentheses when there's an operation, as in ``val = (yield i)
556
Values are sent into a generator by calling its ``send(value)`` method. This
557
method resumes the generator's code and the ``yield`` expression returns the
558
specified value. If the regular ``__next__()`` method is called, the ``yield``
561
Here's a simple counter that increments by 1 and allows changing the value of
562
the internal counter.
566
def counter (maximum):
570
# If value provided, change counter
576
And here's an example of changing the counter:
588
Traceback (most recent call last):
589
File "t.py", line 15, in ?
593
Because ``yield`` will often be returning ``None``, you should always check for
594
this case. Don't just use its value in expressions unless you're sure that the
595
``send()`` method will be the only method used resume your generator function.
597
In addition to ``send()``, there are two other new methods on generators:
599
* ``throw(type, value=None, traceback=None)`` is used to raise an exception
600
inside the generator; the exception is raised by the ``yield`` expression
601
where the generator's execution is paused.
603
* ``close()`` raises a :exc:`GeneratorExit` exception inside the generator to
604
terminate the iteration. On receiving this exception, the generator's code
605
must either raise :exc:`GeneratorExit` or :exc:`StopIteration`; catching the
606
exception and doing anything else is illegal and will trigger a
607
:exc:`RuntimeError`. ``close()`` will also be called by Python's garbage
608
collector when the generator is garbage-collected.
610
If you need to run cleanup code when a :exc:`GeneratorExit` occurs, I suggest
611
using a ``try: ... finally:`` suite instead of catching :exc:`GeneratorExit`.
613
The cumulative effect of these changes is to turn generators from one-way
614
producers of information into both producers and consumers.
616
Generators also become **coroutines**, a more generalized form of subroutines.
617
Subroutines are entered at one point and exited at another point (the top of the
618
function, and a ``return`` statement), but coroutines can be entered, exited,
619
and resumed at many different points (the ``yield`` statements).
625
Let's look in more detail at built-in functions often used with iterators.
627
Two of Python's built-in functions, :func:`map` and :func:`filter` duplicate the
628
features of generator expressions:
630
``map(f, iterA, iterB, ...)`` returns an iterator over the sequence
631
``f(iterA[0], iterB[0]), f(iterA[1], iterB[1]), f(iterA[2], iterB[2]), ...``.
637
>>> list(map(upper, ['sentence', 'fragment']))
638
['SENTENCE', 'FRAGMENT']
639
>>> [upper(s) for s in ['sentence', 'fragment']]
640
['SENTENCE', 'FRAGMENT']
642
You can of course achieve the same effect with a list comprehension.
644
``filter(predicate, iter)`` returns an iterator over all the sequence elements
645
that meet a certain condition, and is similarly duplicated by list
646
comprehensions. A **predicate** is a function that returns the truth value of
647
some condition; for use with :func:`filter`, the predicate must take a single
651
... return (x % 2) == 0
653
>>> list(filter(is_even, range(10)))
657
This can also be written as a list comprehension:
659
>>> list(x for x in range(10) if is_even(x))
663
``enumerate(iter)`` counts off the elements in the iterable, returning 2-tuples
664
containing the count and each element. ::
666
>>> for item in enumerate(['subject', 'verb', 'object']):
672
:func:`enumerate` is often used when looping through a list and recording the
673
indexes at which certain conditions are met::
675
f = open('data.txt', 'r')
676
for i, line in enumerate(f):
677
if line.strip() == '':
678
print('Blank line at line #%i' % i)
680
``sorted(iterable, [key=None], [reverse=False])`` collects all the elements of
681
the iterable into a list, sorts the list, and returns the sorted result. The
682
``key``, and ``reverse`` arguments are passed through to the constructed list's
683
``.sort()`` method. ::
686
>>> # Generate 8 random numbers between [0, 10000)
687
>>> rand_list = random.sample(range(10000), 8)
689
[769, 7953, 9828, 6431, 8442, 9878, 6213, 2207]
690
>>> sorted(rand_list)
691
[769, 2207, 6213, 6431, 7953, 8442, 9828, 9878]
692
>>> sorted(rand_list, reverse=True)
693
[9878, 9828, 8442, 7953, 6431, 6213, 2207, 769]
695
(For a more detailed discussion of sorting, see the Sorting mini-HOWTO in the
696
Python wiki at http://wiki.python.org/moin/HowTo/Sorting.)
699
The ``any(iter)`` and ``all(iter)`` built-ins look at the truth values of an
700
iterable's contents. :func:`any` returns True if any element in the iterable is
701
a true value, and :func:`all` returns True if all of the elements are true
718
``zip(iterA, iterB, ...)`` takes one element from each iterable and
719
returns them in a tuple::
721
zip(['a', 'b', 'c'], (1, 2, 3)) =>
722
('a', 1), ('b', 2), ('c', 3)
724
It doesn't construct an in-memory list and exhaust all the input iterators
725
before returning; instead tuples are constructed and returned only if they're
726
requested. (The technical term for this behaviour is `lazy evaluation
727
<http://en.wikipedia.org/wiki/Lazy_evaluation>`__.)
729
This iterator is intended to be used with iterables that are all of the same
730
length. If the iterables are of different lengths, the resulting stream will be
731
the same length as the shortest iterable. ::
733
zip(['a', 'b'], (1, 2, 3)) =>
736
You should avoid doing this, though, because an element may be taken from the
737
longer iterators and discarded. This means you can't go on to use the iterators
738
further because you risk skipping a discarded element.
744
The :mod:`itertools` module contains a number of commonly-used iterators as well
745
as functions for combining several iterators. This section will introduce the
746
module's contents by showing small examples.
748
The module's functions fall into a few broad classes:
750
* Functions that create a new iterator based on an existing iterator.
751
* Functions for treating an iterator's elements as function arguments.
752
* Functions for selecting portions of an iterator's output.
753
* A function for grouping an iterator's output.
755
Creating new iterators
756
----------------------
758
``itertools.count(n)`` returns an infinite stream of integers, increasing by 1
759
each time. You can optionally supply the starting number, which defaults to 0::
762
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...
763
itertools.count(10) =>
764
10, 11, 12, 13, 14, 15, 16, 17, 18, 19, ...
766
``itertools.cycle(iter)`` saves a copy of the contents of a provided iterable
767
and returns a new iterator that returns its elements from first to last. The
768
new iterator will repeat these elements infinitely. ::
770
itertools.cycle([1,2,3,4,5]) =>
771
1, 2, 3, 4, 5, 1, 2, 3, 4, 5, ...
773
``itertools.repeat(elem, [n])`` returns the provided element ``n`` times, or
774
returns the element endlessly if ``n`` is not provided. ::
776
itertools.repeat('abc') =>
777
abc, abc, abc, abc, abc, abc, abc, abc, abc, abc, ...
778
itertools.repeat('abc', 5) =>
779
abc, abc, abc, abc, abc
781
``itertools.chain(iterA, iterB, ...)`` takes an arbitrary number of iterables as
782
input, and returns all the elements of the first iterator, then all the elements
783
of the second, and so on, until all of the iterables have been exhausted. ::
785
itertools.chain(['a', 'b', 'c'], (1, 2, 3)) =>
788
``itertools.islice(iter, [start], stop, [step])`` returns a stream that's a
789
slice of the iterator. With a single ``stop`` argument, it will return the
790
first ``stop`` elements. If you supply a starting index, you'll get
791
``stop-start`` elements, and if you supply a value for ``step``, elements will
792
be skipped accordingly. Unlike Python's string and list slicing, you can't use
793
negative values for ``start``, ``stop``, or ``step``. ::
795
itertools.islice(range(10), 8) =>
796
0, 1, 2, 3, 4, 5, 6, 7
797
itertools.islice(range(10), 2, 8) =>
799
itertools.islice(range(10), 2, 8, 2) =>
802
``itertools.tee(iter, [n])`` replicates an iterator; it returns ``n``
803
independent iterators that will all return the contents of the source iterator.
804
If you don't supply a value for ``n``, the default is 2. Replicating iterators
805
requires saving some of the contents of the source iterator, so this can consume
806
significant memory if the iterator is large and one of the new iterators is
807
consumed more than the others. ::
809
itertools.tee( itertools.count() ) =>
813
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...
816
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...
819
Calling functions on elements
820
-----------------------------
822
The ``operator`` module contains a set of functions corresponding to Python's
823
operators. Some examples are ``operator.add(a, b)`` (adds two values),
824
``operator.ne(a, b)`` (same as ``a!=b``), and ``operator.attrgetter('id')``
825
(returns a callable that fetches the ``"id"`` attribute).
827
``itertools.starmap(func, iter)`` assumes that the iterable will return a stream
828
of tuples, and calls ``f()`` using these tuples as the arguments::
830
itertools.starmap(os.path.join,
831
[('/usr', 'bin', 'java'), ('/bin', 'python'),
832
('/usr', 'bin', 'perl'),('/usr', 'bin', 'ruby')])
834
/usr/bin/java, /bin/python, /usr/bin/perl, /usr/bin/ruby
840
Another group of functions chooses a subset of an iterator's elements based on a
843
``itertools.filterfalse(predicate, iter)`` is the opposite, returning all
844
elements for which the predicate returns false::
846
itertools.filterfalse(is_even, itertools.count()) =>
847
1, 3, 5, 7, 9, 11, 13, 15, ...
849
``itertools.takewhile(predicate, iter)`` returns elements for as long as the
850
predicate returns true. Once the predicate returns false, the iterator will
851
signal the end of its results.
858
itertools.takewhile(less_than_10, itertools.count()) =>
859
0, 1, 2, 3, 4, 5, 6, 7, 8, 9
861
itertools.takewhile(is_even, itertools.count()) =>
864
``itertools.dropwhile(predicate, iter)`` discards elements while the predicate
865
returns true, and then returns the rest of the iterable's results.
869
itertools.dropwhile(less_than_10, itertools.count()) =>
870
10, 11, 12, 13, 14, 15, 16, 17, 18, 19, ...
872
itertools.dropwhile(is_even, itertools.count()) =>
873
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ...
879
The last function I'll discuss, ``itertools.groupby(iter, key_func=None)``, is
880
the most complicated. ``key_func(elem)`` is a function that can compute a key
881
value for each element returned by the iterable. If you don't supply a key
882
function, the key is simply each element itself.
884
``groupby()`` collects all the consecutive elements from the underlying iterable
885
that have the same key value, and returns a stream of 2-tuples containing a key
886
value and an iterator for the elements with that key.
890
city_list = [('Decatur', 'AL'), ('Huntsville', 'AL'), ('Selma', 'AL'),
891
('Anchorage', 'AK'), ('Nome', 'AK'),
892
('Flagstaff', 'AZ'), ('Phoenix', 'AZ'), ('Tucson', 'AZ'),
896
def get_state (city_state):
899
itertools.groupby(city_list, get_state) =>
902
('AZ', iterator-3), ...
906
('Decatur', 'AL'), ('Huntsville', 'AL'), ('Selma', 'AL')
908
('Anchorage', 'AK'), ('Nome', 'AK')
910
('Flagstaff', 'AZ'), ('Phoenix', 'AZ'), ('Tucson', 'AZ')
912
``groupby()`` assumes that the underlying iterable's contents will already be
913
sorted based on the key. Note that the returned iterators also use the
914
underlying iterable, so you have to consume the results of iterator-1 before
915
requesting iterator-2 and its corresponding key.
921
The :mod:`functools` module in Python 2.5 contains some higher-order functions.
922
A **higher-order function** takes one or more functions as input and returns a
923
new function. The most useful tool in this module is the
924
:func:`functools.partial` function.
926
For programs written in a functional style, you'll sometimes want to construct
927
variants of existing functions that have some of the parameters filled in.
928
Consider a Python function ``f(a, b, c)``; you may wish to create a new function
929
``g(b, c)`` that's equivalent to ``f(1, b, c)``; you're filling in a value for
930
one of ``f()``'s parameters. This is called "partial function application".
932
The constructor for ``partial`` takes the arguments ``(function, arg1, arg2,
933
... kwarg1=value1, kwarg2=value2)``. The resulting object is callable, so you
934
can just call it to invoke ``function`` with the filled-in arguments.
936
Here's a small but realistic example::
940
def log (message, subsystem):
941
"Write the contents of 'message' to the specified subsystem."
942
print('%s: %s' % (subsystem, message))
945
server_log = functools.partial(log, subsystem='server')
946
server_log('Unable to open socket')
948
``functools.reduce(func, iter, [initial_value])`` cumulatively performs an
949
operation on all the iterable's elements and, therefore, can't be applied to
950
infinite iterables. (Note it is not in :mod:`builtins`, but in the
951
:mod:`functools` module.) ``func`` must be a function that takes two elements
952
and returns a single value. :func:`functools.reduce` takes the first two
953
elements A and B returned by the iterator and calculates ``func(A, B)``. It
954
then requests the third element, C, calculates ``func(func(A, B), C)``, combines
955
this result with the fourth element returned, and continues until the iterable
956
is exhausted. If the iterable returns no values at all, a :exc:`TypeError`
957
exception is raised. If the initial value is supplied, it's used as a starting
958
point and ``func(initial_value, A)`` is the first calculation. ::
960
>>> import operator, functools
961
>>> functools.reduce(operator.concat, ['A', 'BB', 'C'])
963
>>> functools.reduce(operator.concat, [])
964
Traceback (most recent call last):
966
TypeError: reduce() of empty sequence with no initial value
967
>>> functools.reduce(operator.mul, [1,2,3], 1)
969
>>> functools.reduce(operator.mul, [], 1)
972
If you use :func:`operator.add` with :func:`functools.reduce`, you'll add up all the
973
elements of the iterable. This case is so common that there's a special
974
built-in called :func:`sum` to compute it:
977
>>> functools.reduce(operator.add, [1,2,3,4], 0)
984
For many uses of :func:`functools.reduce`, though, it can be clearer to just write the
985
obvious :keyword:`for` loop::
989
product = functools.reduce(operator.mul, [1,2,3], 1)
1000
The :mod:`operator` module was mentioned earlier. It contains a set of
1001
functions corresponding to Python's operators. These functions are often useful
1002
in functional-style code because they save you from writing trivial functions
1003
that perform a single operation.
1005
Some of the functions in this module are:
1007
* Math operations: ``add()``, ``sub()``, ``mul()``, ``floordiv()``, ``abs()``, ...
1008
* Logical operations: ``not_()``, ``truth()``.
1009
* Bitwise operations: ``and_()``, ``or_()``, ``invert()``.
1010
* Comparisons: ``eq()``, ``ne()``, ``lt()``, ``le()``, ``gt()``, and ``ge()``.
1011
* Object identity: ``is_()``, ``is_not()``.
1013
Consult the operator module's documentation for a complete list.
1017
The functional module
1018
---------------------
1020
Collin Winter's `functional module <http://oakwinter.com/code/functional/>`__
1021
provides a number of more advanced tools for functional programming. It also
1022
reimplements several Python built-ins, trying to make them more intuitive to
1023
those used to functional programming in other languages.
1025
This section contains an introduction to some of the most important functions in
1026
``functional``; full documentation can be found at `the project's website
1027
<http://oakwinter.com/code/functional/documentation/>`__.
1029
``compose(outer, inner, unpack=False)``
1031
The ``compose()`` function implements function composition. In other words, it
1032
returns a wrapper around the ``outer`` and ``inner`` callables, such that the
1033
return value from ``inner`` is fed directly to ``outer``. That is, ::
1041
>>> compose(double, add)(5, 6)
1046
>>> double(add(5, 6))
1049
The ``unpack`` keyword is provided to work around the fact that Python functions
1050
are not always `fully curried <http://en.wikipedia.org/wiki/Currying>`__. By
1051
default, it is expected that the ``inner`` function will return a single object
1052
and that the ``outer`` function will take a single argument. Setting the
1053
``unpack`` argument causes ``compose`` to expect a tuple from ``inner`` which
1054
will be expanded before being passed to ``outer``. Put simply, ::
1064
compose(f, g, unpack=True)(5, 6)
1070
Even though ``compose()`` only accepts two functions, it's trivial to build up a
1071
version that will compose any number of functions. We'll use
1072
:func:`functools.reduce`, ``compose()`` and ``partial()`` (the last of which is
1073
provided by both ``functional`` and ``functools``). ::
1075
from functional import compose, partial
1079
multi_compose = partial(functools.reduce, compose)
1082
We can also use ``map()``, ``compose()`` and ``partial()`` to craft a version of
1083
``"".join(...)`` that converts its arguments to string::
1085
from functional import compose, partial
1087
join = compose("".join, partial(map, str))
1092
``flip()`` wraps the callable in ``func`` and causes it to receive its
1093
non-keyword arguments in reverse order. ::
1095
>>> def triple(a, b, c):
1096
... return (a, b, c)
1101
>>> flipped_triple = flip(triple)
1102
>>> flipped_triple(5, 6, 7)
1105
``foldl(func, start, iterable)``
1107
``foldl()`` takes a binary function, a starting value (usually some kind of
1108
'zero'), and an iterable. The function is applied to the starting value and the
1109
first element of the list, then the result of that and the second element of the
1110
list, then the result of that and the third element of the list, and so on.
1112
This means that a call such as::
1114
foldl(f, 0, [1, 2, 3])
1121
``foldl()`` is roughly equivalent to the following recursive function::
1123
def foldl(func, start, seq):
1127
return foldl(func, func(start, seq[0]), seq[1:])
1129
Speaking of equivalence, the above ``foldl`` call can be expressed in terms of
1130
the built-in :func:`functools.reduce` like so::
1133
functools.reduce(f, [1, 2, 3], 0)
1136
We can use ``foldl()``, ``operator.concat()`` and ``partial()`` to write a
1137
cleaner, more aesthetically-pleasing version of Python's ``"".join(...)``
1140
from functional import foldl, partial from operator import concat
1142
join = partial(foldl, concat, "")
1145
Small functions and the lambda expression
1146
=========================================
1148
When writing functional-style programs, you'll often need little functions that
1149
act as predicates or that combine elements in some way.
1151
If there's a Python built-in or a module function that's suitable, you don't
1152
need to define a new function at all::
1154
stripped_lines = [line.strip() for line in lines]
1155
existing_files = filter(os.path.exists, file_list)
1157
If the function you need doesn't exist, you need to write it. One way to write
1158
small functions is to use the ``lambda`` statement. ``lambda`` takes a number
1159
of parameters and an expression combining these parameters, and creates a small
1160
function that returns the value of the expression::
1162
lowercase = lambda x: x.lower()
1164
print_assign = lambda name, value: name + '=' + str(value)
1166
adder = lambda x, y: x+y
1168
An alternative is to just use the ``def`` statement and define a function in the
1174
def print_assign(name, value):
1175
return name + '=' + str(value)
1180
Which alternative is preferable? That's a style question; my usual course is to
1181
avoid using ``lambda``.
1183
One reason for my preference is that ``lambda`` is quite limited in the
1184
functions it can define. The result has to be computable as a single
1185
expression, which means you can't have multiway ``if... elif... else``
1186
comparisons or ``try... except`` statements. If you try to do too much in a
1187
``lambda`` statement, you'll end up with an overly complicated expression that's
1188
hard to read. Quick, what's the following code doing?
1193
total = functools.reduce(lambda a, b: (0, a[1] + b[1]), items)[1]
1195
You can figure it out, but it takes time to disentangle the expression to figure
1196
out what's going on. Using a short nested ``def`` statements makes things a
1201
return 0, a[1] + b[1]
1203
total = functools.reduce(combine, items)[1]
1205
But it would be best of all if I had simply used a ``for`` loop::
1211
Or the :func:`sum` built-in and a generator expression::
1213
total = sum(b for a,b in items)
1215
Many uses of :func:`functools.reduce` are clearer when written as ``for`` loops.
1217
Fredrik Lundh once suggested the following set of rules for refactoring uses of
1220
1) Write a lambda function.
1221
2) Write a comment explaining what the heck that lambda does.
1222
3) Study the comment for a while, and think of a name that captures the essence
1224
4) Convert the lambda to a def statement, using that name.
1225
5) Remove the comment.
1227
I really like these rules, but you're free to disagree
1228
about whether this lambda-free style is better.
1231
Revision History and Acknowledgements
1232
=====================================
1234
The author would like to thank the following people for offering suggestions,
1235
corrections and assistance with various drafts of this article: Ian Bicking,
1236
Nick Coghlan, Nick Efford, Raymond Hettinger, Jim Jewett, Mike Krell, Leandro
1237
Lameiro, Jussi Salmela, Collin Winter, Blake Winton.
1239
Version 0.1: posted June 30 2006.
1241
Version 0.11: posted July 1 2006. Typo fixes.
1243
Version 0.2: posted July 10 2006. Merged genexp and listcomp sections into one.
1246
Version 0.21: Added more references suggested on the tutor mailing list.
1248
Version 0.30: Adds a section on the ``functional`` module written by Collin
1249
Winter; adds short section on the operator module; a few other edits.
1258
**Structure and Interpretation of Computer Programs**, by Harold Abelson and
1259
Gerald Jay Sussman with Julie Sussman. Full text at
1260
http://mitpress.mit.edu/sicp/. In this classic textbook of computer science,
1261
chapters 2 and 3 discuss the use of sequences and streams to organize the data
1262
flow inside a program. The book uses Scheme for its examples, but many of the
1263
design approaches described in these chapters are applicable to functional-style
1266
http://www.defmacro.org/ramblings/fp.html: A general introduction to functional
1267
programming that uses Java examples and has a lengthy historical introduction.
1269
http://en.wikipedia.org/wiki/Functional_programming: General Wikipedia entry
1270
describing functional programming.
1272
http://en.wikipedia.org/wiki/Coroutine: Entry for coroutines.
1274
http://en.wikipedia.org/wiki/Currying: Entry for the concept of currying.
1279
http://gnosis.cx/TPiP/: The first chapter of David Mertz's book
1280
:title-reference:`Text Processing in Python` discusses functional programming
1281
for text processing, in the section titled "Utilizing Higher-Order Functions in
1284
Mertz also wrote a 3-part series of articles on functional programming
1285
for IBM's DeveloperWorks site; see
1286
`part 1 <http://www-128.ibm.com/developerworks/library/l-prog.html>`__,
1287
`part 2 <http://www-128.ibm.com/developerworks/library/l-prog2.html>`__, and
1288
`part 3 <http://www-128.ibm.com/developerworks/linux/library/l-prog3.html>`__,
1291
Python documentation
1292
--------------------
1294
Documentation for the :mod:`itertools` module.
1296
Documentation for the :mod:`operator` module.
1298
:pep:`289`: "Generator Expressions"
1300
:pep:`342`: "Coroutines via Enhanced Generators" describes the new generator
1301
features in Python 2.5.
1306
-----------------------------
1310
XXX Need a large example.
1312
But will an example add much? I'll post a first draft and see
1313
what the comments say.
1320
Programs built out of functions
1321
Functions are strictly input-output, no internal state
1322
Opposed to OO programming, where objects have state
1326
Assignment is difficult to reason about
1327
Not very relevant to Python
1329
Small functions that do one thing
1331
Easy to test due to lack of state
1332
Easy to verify output from intermediate steps
1334
You assemble a toolbox of functions that can be mixed
1337
Need a significant example
1341
The itertools module
1343
Small functions and the lambda statement
1350
Handy little function for printing part of an iterator -- used
1351
while writing this document.
1355
slice = itertools.islice(it, 10)
1356
for elem in slice[:-1]:
1357
sys.stdout.write(str(elem))
1358
sys.stdout.write(', ')
added
removed
Doc/howto/functional.rst
