~dkuhlman/python-training-materials/Materials

<h1>8.4. <a class="reference internal" href="#module-heapq" title="heapq: Heap queue algorithm (a.k.a. priority queue)."><tt class="xref py py-mod docutils literal">heapq</tt></a> — Heap queue algorithm<a class="headerlink" href="#module-heapq" title="Permalink to this headline">¶</a></h1>

New in version 2.3.

</div>

Source code: <a class="reference external" href="https://hg.python.org/cpython/file/2.7/Lib/heapq.py">Lib/heapq.py</a>

This module provides an implementation of the heap queue algorithm, also known

as the priority queue algorithm.

Heaps are binary trees for which every parent node has a value less than or

equal to any of its children. This implementation uses arrays for which

<tt class="docutils literal">heap[k] <= heap[2*k+1]</tt> and <tt class="docutils literal">heap[k] <= heap[2*k+2]</tt> for all k, counting

elements from zero. For the sake of comparison, non-existing elements are

considered to be infinite. The interesting property of a heap is that its

smallest element is always the root, <tt class="docutils literal">heap[0]</tt>.

The API below differs from textbook heap algorithms in two aspects: (a) We use

zero-based indexing. This makes the relationship between the index for a node

and the indexes for its children slightly less obvious, but is more suitable

since Python uses zero-based indexing. (b) Our pop method returns the smallest

item, not the largest (called a “min heap” in textbooks; a “max heap” is more

common in texts because of its suitability for in-place sorting).

These two make it possible to view the heap as a regular Python list without

surprises: <tt class="docutils literal">heap[0]</tt> is the smallest item, and <tt class="docutils literal">heap.sort()</tt> maintains the

100

heap invariant!

101

To create a heap, use a list initialized to <tt class="docutils literal">[]</tt>, or you can transform a

102

populated list into a heap via function <a class="reference internal" href="#heapq.heapify" title="heapq.heapify"><tt class="xref py py-func docutils literal">heapify()</tt></a>.

103

The following functions are provided:

104

105

106

<tt class="descclassname">heapq.</tt><tt class="descname">heappush</tt><big>(</big>heap, item<big>)</big><a class="headerlink" href="#heapq.heappush" title="Permalink to this definition">¶</a></dt>

107

<dd>Push the value item onto the heap, maintaining the heap invariant.

108

</dd></dl>

109

110

111

112

<tt class="descclassname">heapq.</tt><tt class="descname">heappop</tt><big>(</big>heap<big>)</big><a class="headerlink" href="#heapq.heappop" title="Permalink to this definition">¶</a></dt>

113

<dd>Pop and return the smallest item from the heap, maintaining the heap

114

invariant. If the heap is empty, <a class="reference internal" href="exceptions.html#exceptions.IndexError" title="exceptions.IndexError"><tt class="xref py py-exc docutils literal">IndexError</tt></a> is raised. To access the

115

smallest item without popping it, use <tt class="docutils literal">heap[0]</tt>.

116

</dd></dl>

117

118

119

120

<tt class="descclassname">heapq.</tt><tt class="descname">heappushpop</tt><big>(</big>heap, item<big>)</big><a class="headerlink" href="#heapq.heappushpop" title="Permalink to this definition">¶</a></dt>

121

<dd>Push item on the heap, then pop and return the smallest item from the

122

heap. The combined action runs more efficiently than <a class="reference internal" href="#heapq.heappush" title="heapq.heappush"><tt class="xref py py-func docutils literal">heappush()</tt></a>

123

followed by a separate call to <a class="reference internal" href="#heapq.heappop" title="heapq.heappop"><tt class="xref py py-func docutils literal">heappop()</tt></a>.

124

125

New in version 2.6.

126

</div>

127

</dd></dl>

128

129

130

131

<tt class="descclassname">heapq.</tt><tt class="descname">heapify</tt><big>(</big>x<big>)</big><a class="headerlink" href="#heapq.heapify" title="Permalink to this definition">¶</a></dt>

132

<dd>Transform list x into a heap, in-place, in linear time.

133

</dd></dl>

134

135

136

137

<tt class="descclassname">heapq.</tt><tt class="descname">heapreplace</tt><big>(</big>heap, item<big>)</big><a class="headerlink" href="#heapq.heapreplace" title="Permalink to this definition">¶</a></dt>

138

<dd>Pop and return the smallest item from the heap, and also push the new item.

139

The heap size doesn’t change. If the heap is empty, <a class="reference internal" href="exceptions.html#exceptions.IndexError" title="exceptions.IndexError"><tt class="xref py py-exc docutils literal">IndexError</tt></a> is raised.

140

This one step operation is more efficient than a <a class="reference internal" href="#heapq.heappop" title="heapq.heappop"><tt class="xref py py-func docutils literal">heappop()</tt></a> followed by

141

<a class="reference internal" href="#heapq.heappush" title="heapq.heappush"><tt class="xref py py-func docutils literal">heappush()</tt></a> and can be more appropriate when using a fixed-size heap.

142

The pop/push combination always returns an element from the heap and replaces

143

it with item.

144

The value returned may be larger than the item added. If that isn’t

145

desired, consider using <a class="reference internal" href="#heapq.heappushpop" title="heapq.heappushpop"><tt class="xref py py-func docutils literal">heappushpop()</tt></a> instead. Its push/pop

146

combination returns the smaller of the two values, leaving the larger value

147

on the heap.

148

</dd></dl>

149

150

The module also offers three general purpose functions based on heaps.

151

152

153

<tt class="descclassname">heapq.</tt><tt class="descname">merge</tt><big>(</big>*iterables<big>)</big><a class="headerlink" href="#heapq.merge" title="Permalink to this definition">¶</a></dt>

154

<dd>Merge multiple sorted inputs into a single sorted output (for example, merge

155

timestamped entries from multiple log files). Returns an <a class="reference internal" href="../glossary.html#term-iterator">iterator</a>

156

over the sorted values.

157

Similar to <tt class="docutils literal">sorted(itertools.chain(*iterables))</tt> but returns an iterable, does

158

not pull the data into memory all at once, and assumes that each of the input

159

streams is already sorted (smallest to largest).

160

161

New in version 2.6.

162

</div>

163

</dd></dl>

164

165

166

167

<tt class="descclassname">heapq.</tt><tt class="descname">nlargest</tt><big>(</big>n, iterable[, key]<big>)</big><a class="headerlink" href="#heapq.nlargest" title="Permalink to this definition">¶</a></dt>

168

<dd>Return a list with the n largest elements from the dataset defined by

169

iterable. key, if provided, specifies a function of one argument that is

170

used to extract a comparison key from each element in the iterable:

171

<tt class="docutils literal">key=str.lower</tt> Equivalent to: <tt class="docutils literal">sorted(iterable, key=key,

172

reverse=True)[:n]</tt>

173

174

New in version 2.4.

175

</div>

176

177

Changed in version 2.5: Added the optional key argument.

178

</div>

179

</dd></dl>

180

181

182

183

<tt class="descclassname">heapq.</tt><tt class="descname">nsmallest</tt><big>(</big>n, iterable[, key]<big>)</big><a class="headerlink" href="#heapq.nsmallest" title="Permalink to this definition">¶</a></dt>

184

<dd>Return a list with the n smallest elements from the dataset defined by

185

iterable. key, if provided, specifies a function of one argument that is

186

used to extract a comparison key from each element in the iterable:

187

<tt class="docutils literal">key=str.lower</tt> Equivalent to: <tt class="docutils literal">sorted(iterable, key=key)[:n]</tt>

188

189

New in version 2.4.

190

</div>

191

192

Changed in version 2.5: Added the optional key argument.

193

</div>

194

</dd></dl>

195

196

The latter two functions perform best for smaller values of n. For larger

197

values, it is more efficient to use the <a class="reference internal" href="functions.html#sorted" title="sorted"><tt class="xref py py-func docutils literal">sorted()</tt></a> function. Also, when

198

<tt class="docutils literal">n==1</tt>, it is more efficient to use the built-in <a class="reference internal" href="functions.html#min" title="min"><tt class="xref py py-func docutils literal">min()</tt></a> and <a class="reference internal" href="functions.html#max" title="max"><tt class="xref py py-func docutils literal">max()</tt></a>

199

functions. If repeated usage of these functions is required, consider turning

200

the iterable into an actual heap.

201

202

<h2>8.4.1. Basic Examples<a class="headerlink" href="#basic-examples" title="Permalink to this headline">¶</a></h2>

203

A <a class="reference external" href="http://en.wikipedia.org/wiki/Heapsort">heapsort</a> can be implemented by

204

pushing all values onto a heap and then popping off the smallest values one at a

205

time:

206

<div class="highlight-python"><div class="highlight"><pre>>>> def heapsort(iterable):

207

... h = []

208

... for value in iterable:

209

... heappush(h, value)

210

... return [heappop(h) for i in range(len(h))]

211

...

212

>>> heapsort([1, 3, 5, 7, 9, 2, 4, 6, 8, 0])

213

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

214

</pre></div>

215

</div>

216

This is similar to <tt class="docutils literal">sorted(iterable)</tt>, but unlike <a class="reference internal" href="functions.html#sorted" title="sorted"><tt class="xref py py-func docutils literal">sorted()</tt></a>, this

217

implementation is not stable.

218

Heap elements can be tuples. This is useful for assigning comparison values

219

(such as task priorities) alongside the main record being tracked:

220

<div class="highlight-python"><div class="highlight"><pre>>>> h = []

221

>>> heappush(h, (5, 'write code'))

222

>>> heappush(h, (7, 'release product'))

223

>>> heappush(h, (1, 'write spec'))

224

>>> heappush(h, (3, 'create tests'))

225

>>> heappop(h)

226

(1, 'write spec')

227

</pre></div>

228

</div>

229

</div>

230

231

<h2>8.4.2. Priority Queue Implementation Notes<a class="headerlink" href="#priority-queue-implementation-notes" title="Permalink to this headline">¶</a></h2>

232

A <a class="reference external" href="http://en.wikipedia.org/wiki/Priority_queue">priority queue</a> is common use

233

for a heap, and it presents several implementation challenges:

234

235

<li>Sort stability: how do you get two tasks with equal priorities to be returned

236

in the order they were originally added?</li>

237

<li>In the future with Python 3, tuple comparison breaks for (priority, task)

238

pairs if the priorities are equal and the tasks do not have a default

239

comparison order.</li>

240

<li>If the priority of a task changes, how do you move it to a new position in

241

the heap?</li>

242

<li>Or if a pending task needs to be deleted, how do you find it and remove it

243

from the queue?</li>

244

</ul>

245

A solution to the first two challenges is to store entries as 3-element list

246

including the priority, an entry count, and the task. The entry count serves as

247

a tie-breaker so that two tasks with the same priority are returned in the order

248

they were added. And since no two entry counts are the same, the tuple

249

comparison will never attempt to directly compare two tasks.

250

The remaining challenges revolve around finding a pending task and making

251

changes to its priority or removing it entirely. Finding a task can be done

252

with a dictionary pointing to an entry in the queue.

253

Removing the entry or changing its priority is more difficult because it would

254

break the heap structure invariants. So, a possible solution is to mark the

255

existing entry as removed and add a new entry with the revised priority:

256

<div class="highlight-python"><div class="highlight"><pre>pq = [] # list of entries arranged in a heap

257

entry_finder = {} # mapping of tasks to entries

258

REMOVED = '<removed-task>' # placeholder for a removed task

259

counter = itertools.count() # unique sequence count

260

261

def add_task(task, priority=0):

262

'Add a new task or update the priority of an existing task'

263

if task in entry_finder:

264

remove_task(task)

265

count = next(counter)

266

entry = [priority, count, task]

267

entry_finder[task] = entry

268

heappush(pq, entry)

269

270

def remove_task(task):

271

'Mark an existing task as REMOVED. Raise KeyError if not found.'

272

entry = entry_finder.pop(task)

273

entry[-1] = REMOVED

274

275

def pop_task():

276

'Remove and return the lowest priority task. Raise KeyError if empty.'

277

while pq:

278

priority, count, task = heappop(pq)

279

if task is not REMOVED:

280

del entry_finder[task]

281

return task

282

raise KeyError('pop from an empty priority queue')

283

</pre></div>

284

</div>

285

</div>

286

287

<h2>8.4.3. Theory<a class="headerlink" href="#theory" title="Permalink to this headline">¶</a></h2>

288

Heaps are arrays for which <tt class="docutils literal">a[k] <= a[2*k+1]</tt> and <tt class="docutils literal">a[k] <= a[2*k+2]</tt> for all

289

k, counting elements from 0. For the sake of comparison, non-existing

290

elements are considered to be infinite. The interesting property of a heap is

291

that <tt class="docutils literal">a[0]</tt> is always its smallest element.

292

The strange invariant above is meant to be an efficient memory representation

293

for a tournament. The numbers below are k, not <tt class="docutils literal">a[k]</tt>:

294

<div class="highlight-python"><div class="highlight"><pre> 0

295

296

1 2

297

298

3 4 5 6

299

300

7 8 9 10 11 12 13 14

301

302

15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

303

</pre></div>

304

</div>

305

In the tree above, each cell k is topping <tt class="docutils literal">2*k+1</tt> and <tt class="docutils literal">2*k+2</tt>. In an usual

306

binary tournament we see in sports, each cell is the winner over the two cells

307

it tops, and we can trace the winner down the tree to see all opponents s/he

308

had. However, in many computer applications of such tournaments, we do not need

309

to trace the history of a winner. To be more memory efficient, when a winner is

310

promoted, we try to replace it by something else at a lower level, and the rule

311

becomes that a cell and the two cells it tops contain three different items, but

312

the top cell “wins” over the two topped cells.

313

If this heap invariant is protected at all time, index 0 is clearly the overall

314

winner. The simplest algorithmic way to remove it and find the “next” winner is

315

to move some loser (let’s say cell 30 in the diagram above) into the 0 position,

316

and then percolate this new 0 down the tree, exchanging values, until the

317

invariant is re-established. This is clearly logarithmic on the total number of

318

items in the tree. By iterating over all items, you get an O(n log n) sort.

319

A nice feature of this sort is that you can efficiently insert new items while

320

the sort is going on, provided that the inserted items are not “better” than the

321

last 0’th element you extracted. This is especially useful in simulation

322

contexts, where the tree holds all incoming events, and the “win” condition

323

means the smallest scheduled time. When an event schedules other events for

324

execution, they are scheduled into the future, so they can easily go into the

325

heap. So, a heap is a good structure for implementing schedulers (this is what

326

I used for my MIDI sequencer :-).

327

Various structures for implementing schedulers have been extensively studied,

328

and heaps are good for this, as they are reasonably speedy, the speed is almost

329

constant, and the worst case is not much different than the average case.

330

However, there are other representations which are more efficient overall, yet

331

the worst cases might be terrible.

332

Heaps are also very useful in big disk sorts. You most probably all know that a

333

big sort implies producing “runs” (which are pre-sorted sequences, whose size is

334

usually related to the amount of CPU memory), followed by a merging passes for

335

these runs, which merging is often very cleverly organised <a class="footnote-reference" href="#id2" id="id1">[1]</a>. It is very

336

important that the initial sort produces the longest runs possible. Tournaments

337

are a good way to achieve that. If, using all the memory available to hold a

338

tournament, you replace and percolate items that happen to fit the current run,

339

you’ll produce runs which are twice the size of the memory for random input, and

340

much better for input fuzzily ordered.

341

Moreover, if you output the 0’th item on disk and get an input which may not fit

342

in the current tournament (because the value “wins” over the last output value),

343

it cannot fit in the heap, so the size of the heap decreases. The freed memory

344

could be cleverly reused immediately for progressively building a second heap,

345

which grows at exactly the same rate the first heap is melting. When the first

346

heap completely vanishes, you switch heaps and start a new run. Clever and

347

quite effective!

348

In a word, heaps are useful memory structures to know. I use them in a few

349

applications, and I think it is good to keep a ‘heap’ module around. :-)

350

Footnotes

351

352

353

354

<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>The disk balancing algorithms which are current, nowadays, are more annoying

355

than clever, and this is a consequence of the seeking capabilities of the disks.

356

On devices which cannot seek, like big tape drives, the story was quite

357

different, and one had to be very clever to ensure (far in advance) that each

358

tape movement will be the most effective possible (that is, will best

359

participate at “progressing” the merge). Some tapes were even able to read

360

backwards, and this was also used to avoid the rewinding time. Believe me, real

361

good tape sorts were quite spectacular to watch! From all times, sorting has

362

always been a Great Art! :-)</td></tr>

363

</tbody>

364

</table>

365

</div>

366

</div>

367

368

369

</div>

370

</div>

371

</div>

372

373

374

<h3><a href="../contents.html">Table Of Contents</a></h3>

375

<ul>

376

<li><a class="reference internal" href="#">8.4. <tt class="docutils literal">heapq</tt> — Heap queue algorithm</a><ul>

377

<li><a class="reference internal" href="#basic-examples">8.4.1. Basic Examples</a></li>

378

<li><a class="reference internal" href="#priority-queue-implementation-notes">8.4.2. Priority Queue Implementation Notes</a></li>

379

<li><a class="reference internal" href="#theory">8.4.3. Theory</a></li>

380

</ul>

381

</li>

382

</ul>

383

384

<h4>Previous topic</h4>

385

<a href="collections.html"

386

title="previous chapter">8.3. <tt class="docutils literal">collections</tt> — High-performance container datatypes</a>

387

<h4>Next topic</h4>

388

<a href="bisect.html"

389

title="next chapter">8.5. <tt class="docutils literal">bisect</tt> — Array bisection algorithm</a>

390

391

392

<li><a href="../bugs.html">Report a Bug</a></li>

393

<li><a href="../_sources/library/heapq.txt"

394

rel="nofollow">Show Source</a></li>

395

</ul>

396

397

398

<h3>Quick search</h3>

399

400

401

402

403

404

</form>

405

406

Enter search terms or a module, class or function name.

407

408

</div>

409

410

</div>

411

</div>

412

413

</div>

414

415

<h3>Navigation</h3>

416

<ul>

417

418

<a href="../genindex.html" title="General Index"

419

>index</a></li>

420

421

<a href="../py-modindex.html" title="Python Module Index"

422

>modules</a> |</li>

423

424

<a href="bisect.html" title="8.5. bisect — Array bisection algorithm"

425

>next</a> |</li>

426

427

<a href="collections.html" title="8.3. collections — High-performance container datatypes"

428

>previous</a> |</li>

429

<li><img src="../_static/py.png" alt=""

430

style="vertical-align: middle; margin-top: -1px"/></li>

431

<li><a href="https://www.python.org/">Python</a> »</li>

432

<li>

433

2.7.10

434

<a href="../index.html">Documentation</a> »

435

</li>

436

437

<li><a href="index.html" >The Python Standard Library</a> »</li>

438

<li><a href="datatypes.html" >8. Data Types</a> »</li>

439

</ul>

440

</div>

441

442

443

444

The Python Software Foundation is a non-profit corporation.

445

<a href="https://www.python.org/psf/donations/">Please donate.</a>

446

447

Last updated on May 23, 2015.

448

<a href="../bugs.html">Found a bug</a>?

449

450

Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.2.3.

451

</div>

452

453

</body>

454

</html>

b'\\ No newline at end of file'

Older »