~dkuhlman/python-training-materials/Materials

<h1><a class="toc-backref" href="#id1">Library and Extension FAQ</a><a class="headerlink" href="#library-and-extension-faq" title="Permalink to this headline">¶</a></h1>

Contents

<li><a class="reference internal" href="#library-and-extension-faq" id="id1">Library and Extension FAQ</a><ul>

<li><a class="reference internal" href="#general-library-questions" id="id2">General Library Questions</a><ul>

<li><a class="reference internal" href="#how-do-i-find-a-module-or-application-to-perform-task-x" id="id3">How do I find a module or application to perform task X?</a></li>

<li><a class="reference internal" href="#where-is-the-math-py-socket-py-regex-py-etc-source-file" id="id4">Where is the math.py (socket.py, regex.py, etc.) source file?</a></li>

<li><a class="reference internal" href="#how-do-i-make-a-python-script-executable-on-unix" id="id5">How do I make a Python script executable on Unix?</a></li>

<li><a class="reference internal" href="#is-there-a-curses-termcap-package-for-python" id="id6">Is there a curses/termcap package for Python?</a></li>

<li><a class="reference internal" href="#is-there-an-equivalent-to-c-s-onexit-in-python" id="id7">Is there an equivalent to C’s onexit() in Python?</a></li>

<li><a class="reference internal" href="#why-don-t-my-signal-handlers-work" id="id8">Why don’t my signal handlers work?</a></li>

</ul>

</li>

<li><a class="reference internal" href="#common-tasks" id="id9">Common tasks</a><ul>

<li><a class="reference internal" href="#how-do-i-test-a-python-program-or-component" id="id10">How do I test a Python program or component?</a></li>

<li><a class="reference internal" href="#how-do-i-create-documentation-from-doc-strings" id="id11">How do I create documentation from doc strings?</a></li>

<li><a class="reference internal" href="#how-do-i-get-a-single-keypress-at-a-time" id="id12">How do I get a single keypress at a time?</a></li>

</ul>

</li>

<li><a class="reference internal" href="#threads" id="id13">Threads</a><ul>

<li><a class="reference internal" href="#how-do-i-program-using-threads" id="id14">How do I program using threads?</a></li>

<li><a class="reference internal" href="#none-of-my-threads-seem-to-run-why" id="id15">None of my threads seem to run: why?</a></li>

100

<li><a class="reference internal" href="#how-do-i-parcel-out-work-among-a-bunch-of-worker-threads" id="id16">How do I parcel out work among a bunch of worker threads?</a></li>

101

<li><a class="reference internal" href="#what-kinds-of-global-value-mutation-are-thread-safe" id="id17">What kinds of global value mutation are thread-safe?</a></li>

102

<li><a class="reference internal" href="#can-t-we-get-rid-of-the-global-interpreter-lock" id="id18">Can’t we get rid of the Global Interpreter Lock?</a></li>

103

</ul>

104

</li>

105

<li><a class="reference internal" href="#input-and-output" id="id19">Input and Output</a><ul>

106

<li><a class="reference internal" href="#how-do-i-delete-a-file-and-other-file-questions" id="id20">How do I delete a file? (And other file questions...)</a></li>

107

108

<li><a class="reference internal" href="#how-do-i-read-or-write-binary-data" id="id22">How do I read (or write) binary data?</a></li>

109

<li><a class="reference internal" href="#i-can-t-seem-to-use-os-read-on-a-pipe-created-with-os-popen-why" id="id23">I can’t seem to use os.read() on a pipe created with os.popen(); why?</a></li>

110

<li><a class="reference internal" href="#how-do-i-run-a-subprocess-with-pipes-connected-to-both-input-and-output" id="id24">How do I run a subprocess with pipes connected to both input and output?</a></li>

111

<li><a class="reference internal" href="#how-do-i-access-the-serial-rs232-port" id="id25">How do I access the serial (RS232) port?</a></li>

112

<li><a class="reference internal" href="#why-doesn-t-closing-sys-stdout-stdin-stderr-really-close-it" id="id26">Why doesn’t closing sys.stdout (stdin, stderr) really close it?</a></li>

113

</ul>

114

</li>

115

<li><a class="reference internal" href="#network-internet-programming" id="id27">Network/Internet Programming</a><ul>

116

<li><a class="reference internal" href="#what-www-tools-are-there-for-python" id="id28">What WWW tools are there for Python?</a></li>

117

<li><a class="reference internal" href="#how-can-i-mimic-cgi-form-submission-method-post" id="id29">How can I mimic CGI form submission (METHOD=POST)?</a></li>

118

<li><a class="reference internal" href="#what-module-should-i-use-to-help-with-generating-html" id="id30">What module should I use to help with generating HTML?</a></li>

119

<li><a class="reference internal" href="#how-do-i-send-mail-from-a-python-script" id="id31">How do I send mail from a Python script?</a></li>

120

<li><a class="reference internal" href="#how-do-i-avoid-blocking-in-the-connect-method-of-a-socket" id="id32">How do I avoid blocking in the connect() method of a socket?</a></li>

121

</ul>

122

</li>

123

<li><a class="reference internal" href="#databases" id="id33">Databases</a><ul>

124

<li><a class="reference internal" href="#are-there-any-interfaces-to-database-packages-in-python" id="id34">Are there any interfaces to database packages in Python?</a></li>

125

<li><a class="reference internal" href="#how-do-you-implement-persistent-objects-in-python" id="id35">How do you implement persistent objects in Python?</a></li>

126

<li><a class="reference internal" href="#why-is-cpickle-so-slow" id="id36">Why is cPickle so slow?</a></li>

127

<li><a class="reference internal" href="#if-my-program-crashes-with-a-bsddb-or-anydbm-database-open-it-gets-corrupted-how-come" id="id37">If my program crashes with a bsddb (or anydbm) database open, it gets corrupted. How come?</a></li>

128

<li><a class="reference internal" href="#i-tried-to-open-berkeley-db-file-but-bsddb-produces-bsddb-error-22-invalid-argument-help-how-can-i-restore-my-data" id="id38">I tried to open Berkeley DB file, but bsddb produces bsddb.error: (22, ‘Invalid argument’). Help! How can I restore my data?</a></li>

129

</ul>

130

</li>

131

<li><a class="reference internal" href="#mathematics-and-numerics" id="id39">Mathematics and Numerics</a><ul>

132

<li><a class="reference internal" href="#how-do-i-generate-random-numbers-in-python" id="id40">How do I generate random numbers in Python?</a></li>

133

</ul>

134

</li>

135

</ul>

136

</li>

137

</ul>

138

</div>

139

140

<h2><a class="toc-backref" href="#id2">General Library Questions</a><a class="headerlink" href="#general-library-questions" title="Permalink to this headline">¶</a></h2>

141

142

<h3><a class="toc-backref" href="#id3">How do I find a module or application to perform task X?</a><a class="headerlink" href="#how-do-i-find-a-module-or-application-to-perform-task-x" title="Permalink to this headline">¶</a></h3>

143

Check <a class="reference internal" href="../library/index.html#library-index">the Library Reference</a> to see if there’s a relevant

144

standard library module. (Eventually you’ll learn what’s in the standard

145

library and will be able to skip this step.)

146

For third-party packages, search the <a class="reference external" href="https://pypi.python.org/pypi">Python Package Index</a> or try <a class="reference external" href="https://www.google.com">Google</a> or

147

another Web search engine. Searching for “Python” plus a keyword or two for

148

your topic of interest will usually find something helpful.

149

</div>

150

151

<h3><a class="toc-backref" href="#id4">Where is the math.py (socket.py, regex.py, etc.) source file?</a><a class="headerlink" href="#where-is-the-math-py-socket-py-regex-py-etc-source-file" title="Permalink to this headline">¶</a></h3>

152

If you can’t find a source file for a module it may be a built-in or

153

dynamically loaded module implemented in C, C++ or other compiled language.

154

In this case you may not have the source file or it may be something like

155

<code class="file docutils literal">mathmodule.c</code>, somewhere in a C source directory (not on the Python Path).

156

There are (at least) three kinds of modules in Python:

157

158

<li>modules written in Python (.py);

159

</li>

160

<li>modules written in C and dynamically loaded (.dll, .pyd, .so, .sl, etc);

161

</li>

162

<li>modules written in C and linked with the interpreter; to get a list of these,

163

type:

164

<div class="highlight-python"><div class="highlight"><pre>import sys

165

print sys.builtin_module_names

166

</pre></div>

167

</div>

168

</li>

169

</ol>

170

</div>

171

172

<h3><a class="toc-backref" href="#id5">How do I make a Python script executable on Unix?</a><a class="headerlink" href="#how-do-i-make-a-python-script-executable-on-unix" title="Permalink to this headline">¶</a></h3>

173

You need to do two things: the script file’s mode must be executable and the

174

first line must begin with <code class="docutils literal">#!</code> followed by the path of the Python

175

interpreter.

176

The first is done by executing <code class="docutils literal">chmod +x scriptfile</code> or perhaps <code class="docutils literal">chmod 755

177

scriptfile</code>.

178

The second can be done in a number of ways. The most straightforward way is to

179

write

180

<div class="highlight-python"><div class="highlight"><pre>#!/usr/local/bin/python

181

</pre></div>

182

</div>

183

as the very first line of your file, using the pathname for where the Python

184

interpreter is installed on your platform.

185

If you would like the script to be independent of where the Python interpreter

186

lives, you can use the env program. Almost all Unix variants support

187

the following, assuming the Python interpreter is in a directory on the user’s

188

<code class="xref std std-envvar docutils literal">PATH</code>:

189

<div class="highlight-python"><div class="highlight"><pre>#!/usr/bin/env python

190

</pre></div>

191

</div>

192

Don’t do this for CGI scripts. The <code class="xref std std-envvar docutils literal">PATH</code> variable for CGI scripts is

193

often very minimal, so you need to use the actual absolute pathname of the

194

interpreter.

195

Occasionally, a user’s environment is so full that the /usr/bin/env

196

program fails; or there’s no env program at all. In that case, you can try the

197

following hack (due to Alex Rezinsky):

198

<div class="highlight-python"><div class="highlight"><pre>#! /bin/sh

199

""":"

200

exec python $0 ${1+"$@"}

201

"""

202

</pre></div>

203

</div>

204

The minor disadvantage is that this defines the script’s __doc__ string.

205

However, you can fix that by adding

206

<div class="highlight-python"><div class="highlight"><pre>__doc__ = """...Whatever..."""

207

</pre></div>

208

</div>

209

</div>

210

211

<h3><a class="toc-backref" href="#id6">Is there a curses/termcap package for Python?</a><a class="headerlink" href="#is-there-a-curses-termcap-package-for-python" title="Permalink to this headline">¶</a></h3>

212

For Unix variants the standard Python source distribution comes with a curses

213

module in the <a class="reference external" href="https://hg.python.org/cpython/file/2.7/Modules">Modules</a> subdirectory, though it’s not compiled by default.

214

(Note that this is not available in the Windows distribution – there is no

215

curses module for Windows.)

216

The <a class="reference internal" href="../library/curses.html#module-curses" title="curses: An interface to the curses library, providing portable terminal handling. (Unix)"><code class="xref py py-mod docutils literal">curses</code></a> module supports basic curses features as well as many additional

217

functions from ncurses and SYSV curses such as colour, alternative character set

218

support, pads, and mouse support. This means the module isn’t compatible with

219

operating systems that only have BSD curses, but there don’t seem to be any

220

currently maintained OSes that fall into this category.

221

For Windows: use <a class="reference external" href="http://effbot.org/zone/console-index.htm">the consolelib module</a>.

222

</div>

223

224

<h3><a class="toc-backref" href="#id7">Is there an equivalent to C’s onexit() in Python?</a><a class="headerlink" href="#is-there-an-equivalent-to-c-s-onexit-in-python" title="Permalink to this headline">¶</a></h3>

225

The <a class="reference internal" href="../library/atexit.html#module-atexit" title="atexit: Register and execute cleanup functions."><code class="xref py py-mod docutils literal">atexit</code></a> module provides a register function that is similar to C’s

226

<code class="xref c c-func docutils literal">onexit()</code>.

227

</div>

228

229

<h3><a class="toc-backref" href="#id8">Why don’t my signal handlers work?</a><a class="headerlink" href="#why-don-t-my-signal-handlers-work" title="Permalink to this headline">¶</a></h3>

230

The most common problem is that the signal handler is declared with the wrong

231

argument list. It is called as

232

<div class="highlight-python"><div class="highlight"><pre>handler(signum, frame)

233

</pre></div>

234

</div>

235

so it should be declared with two arguments:

236

<div class="highlight-python"><div class="highlight"><pre>def handler(signum, frame):

237

...

238

</pre></div>

239

</div>

240

</div>

241

</div>

242

243

<h2><a class="toc-backref" href="#id9">Common tasks</a><a class="headerlink" href="#common-tasks" title="Permalink to this headline">¶</a></h2>

244

245

<h3><a class="toc-backref" href="#id10">How do I test a Python program or component?</a><a class="headerlink" href="#how-do-i-test-a-python-program-or-component" title="Permalink to this headline">¶</a></h3>

246

Python comes with two testing frameworks. The <a class="reference internal" href="../library/doctest.html#module-doctest" title="doctest: Test pieces of code within docstrings."><code class="xref py py-mod docutils literal">doctest</code></a> module finds

247

examples in the docstrings for a module and runs them, comparing the output with

248

the expected output given in the docstring.

249

The <a class="reference internal" href="../library/unittest.html#module-unittest" title="unittest: Unit testing framework for Python."><code class="xref py py-mod docutils literal">unittest</code></a> module is a fancier testing framework modelled on Java and

250

Smalltalk testing frameworks.

251

To make testing easier, you should use good modular design in your program.

252

Your program should have almost all functionality

253

encapsulated in either functions or class methods – and this sometimes has the

254

surprising and delightful effect of making the program run faster (because local

255

variable accesses are faster than global accesses). Furthermore the program

256

should avoid depending on mutating global variables, since this makes testing

257

much more difficult to do.

258

The “global main logic” of your program may be as simple as

259

<div class="highlight-python"><div class="highlight"><pre>if __name__ == "__main__":

260

main_logic()

261

</pre></div>

262

</div>

263

at the bottom of the main module of your program.

264

Once your program is organized as a tractable collection of functions and class

265

behaviours you should write test functions that exercise the behaviours. A test

266

suite that automates a sequence of tests can be associated with each module.

267

This sounds like a lot of work, but since Python is so terse and flexible it’s

268

surprisingly easy. You can make coding much more pleasant and fun by writing

269

your test functions in parallel with the “production code”, since this makes it

270

easy to find bugs and even design flaws earlier.

271

“Support modules” that are not intended to be the main module of a program may

272

include a self-test of the module.

273

274

self_test()

275

</pre></div>

276

</div>

277

Even programs that interact with complex external interfaces may be tested when

278

the external interfaces are unavailable by using “fake” interfaces implemented

279

in Python.

280

</div>

281

282

<h3><a class="toc-backref" href="#id11">How do I create documentation from doc strings?</a><a class="headerlink" href="#how-do-i-create-documentation-from-doc-strings" title="Permalink to this headline">¶</a></h3>

283

The <a class="reference internal" href="../library/pydoc.html#module-pydoc" title="pydoc: Documentation generator and online help system."><code class="xref py py-mod docutils literal">pydoc</code></a> module can create HTML from the doc strings in your Python

284

source code. An alternative for creating API documentation purely from

285

docstrings is <a class="reference external" href="http://epydoc.sourceforge.net/">epydoc</a>. <a class="reference external" href="http://sphinx-doc.org">Sphinx</a> can also include docstring content.

286

</div>

287

288

<h3><a class="toc-backref" href="#id12">How do I get a single keypress at a time?</a><a class="headerlink" href="#how-do-i-get-a-single-keypress-at-a-time" title="Permalink to this headline">¶</a></h3>

289

For Unix variants there are several solutions. It’s straightforward to do this

290

using curses, but curses is a fairly large module to learn. Here’s a solution

291

without curses:

292

<div class="highlight-python"><div class="highlight"><pre>import termios, fcntl, sys, os

293

fd = sys.stdin.fileno()

294

295

oldterm = termios.tcgetattr(fd)

296

newattr = termios.tcgetattr(fd)

297

newattr[3] = newattr[3] & ~termios.ICANON & ~termios.ECHO

298

termios.tcsetattr(fd, termios.TCSANOW, newattr)

299

300

oldflags = fcntl.fcntl(fd, fcntl.F_GETFL)

301

fcntl.fcntl(fd, fcntl.F_SETFL, oldflags | os.O_NONBLOCK)

302

303

try:

304

while 1:

305

try:

306

c = sys.stdin.read(1)

307

print "Got character", repr(c)

308

except IOError: pass

309

finally:

310

termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm)

311

312

</pre></div>

313

</div>

314

You need the <a class="reference internal" href="../library/termios.html#module-termios" title="termios: POSIX style tty control. (Unix)"><code class="xref py py-mod docutils literal">termios</code></a> and the <a class="reference internal" href="../library/fcntl.html#module-fcntl" title="fcntl: The fcntl() and ioctl() system calls. (Unix)"><code class="xref py py-mod docutils literal">fcntl</code></a> module for any of this to work,

315

and I’ve only tried it on Linux, though it should work elsewhere. In this code,

316

characters are read and printed one at a time.

317

<a class="reference internal" href="../library/termios.html#termios.tcsetattr" title="termios.tcsetattr"><code class="xref py py-func docutils literal">termios.tcsetattr()</code></a> turns off stdin’s echoing and disables canonical mode.

318

<code class="xref py py-func docutils literal">fcntl.fnctl()</code> is used to obtain stdin’s file descriptor flags and modify

319

them for non-blocking mode. Since reading stdin when it is empty results in an

320

<a class="reference internal" href="../library/exceptions.html#exceptions.IOError" title="exceptions.IOError"><code class="xref py py-exc docutils literal">IOError</code></a>, this error is caught and ignored.

321

</div>

322

</div>

323

324

<h2><a class="toc-backref" href="#id13">Threads</a><a class="headerlink" href="#threads" title="Permalink to this headline">¶</a></h2>

325

326

<h3><a class="toc-backref" href="#id14">How do I program using threads?</a><a class="headerlink" href="#how-do-i-program-using-threads" title="Permalink to this headline">¶</a></h3>

327

Be sure to use the <a class="reference internal" href="../library/threading.html#module-threading" title="threading: Higher-level threading interface."><code class="xref py py-mod docutils literal">threading</code></a> module and not the <a class="reference internal" href="../library/thread.html#module-thread" title="thread: Create multiple threads of control within one interpreter."><code class="xref py py-mod docutils literal">thread</code></a> module.

328

The <a class="reference internal" href="../library/threading.html#module-threading" title="threading: Higher-level threading interface."><code class="xref py py-mod docutils literal">threading</code></a> module builds convenient abstractions on top of the

329

low-level primitives provided by the <a class="reference internal" href="../library/thread.html#module-thread" title="thread: Create multiple threads of control within one interpreter."><code class="xref py py-mod docutils literal">thread</code></a> module.

330

Aahz has a set of slides from his threading tutorial that are helpful; see

331

<a class="reference external" href="http://www.pythoncraft.com/OSCON2001/">http://www.pythoncraft.com/OSCON2001/</a>.

332

</div>

333

334

<h3><a class="toc-backref" href="#id15">None of my threads seem to run: why?</a><a class="headerlink" href="#none-of-my-threads-seem-to-run-why" title="Permalink to this headline">¶</a></h3>

335

As soon as the main thread exits, all threads are killed. Your main thread is

336

running too quickly, giving the threads no time to do any work.

337

A simple fix is to add a sleep to the end of the program that’s long enough for

338

all the threads to finish:

339

<div class="highlight-python"><div class="highlight"><pre>import threading, time

340

341

def thread_task(name, n):

342

for i in range(n): print name, i

343

344

for i in range(10):

345

T = threading.Thread(target=thread_task, args=(str(i), i))

346

T.start()

347

348

time.sleep(10) # <----------------------------!

349

</pre></div>

350

</div>

351

But now (on many platforms) the threads don’t run in parallel, but appear to run

352

sequentially, one at a time! The reason is that the OS thread scheduler doesn’t

353

start a new thread until the previous thread is blocked.

354

A simple fix is to add a tiny sleep to the start of the run function:

355

<div class="highlight-python"><div class="highlight"><pre>def thread_task(name, n):

356

time.sleep(0.001) # <---------------------!

357

358

359

for i in range(10):

360

361

T.start()

362

363

time.sleep(10)

364

</pre></div>

365

</div>

366

Instead of trying to guess a good delay value for <a class="reference internal" href="../library/time.html#time.sleep" title="time.sleep"><code class="xref py py-func docutils literal">time.sleep()</code></a>,

367

it’s better to use some kind of semaphore mechanism. One idea is to use the

368

<a class="reference internal" href="../library/queue.html#module-Queue" title="Queue: A synchronized queue class."><code class="xref py py-mod docutils literal">Queue</code></a> module to create a queue object, let each thread append a token to

369

the queue when it finishes, and let the main thread read as many tokens from the

370

queue as there are threads.

371

</div>

372

373

<h3><a class="toc-backref" href="#id16">How do I parcel out work among a bunch of worker threads?</a><a class="headerlink" href="#how-do-i-parcel-out-work-among-a-bunch-of-worker-threads" title="Permalink to this headline">¶</a></h3>

374

Use the <a class="reference internal" href="../library/queue.html#module-Queue" title="Queue: A synchronized queue class."><code class="xref py py-mod docutils literal">Queue</code></a> module to create a queue containing a list of jobs. The

375

<a class="reference internal" href="../library/queue.html#Queue.Queue" title="Queue.Queue"><code class="xref py py-class docutils literal">Queue</code></a> class maintains a list of objects and has a <code class="docutils literal">.put(obj)</code>

376

method that adds items to the queue and a <code class="docutils literal">.get()</code> method to return them.

377

The class will take care of the locking necessary to ensure that each job is

378

handed out exactly once.

379

Here’s a trivial example:

380

<div class="highlight-python"><div class="highlight"><pre>import threading, Queue, time

381

382

# The worker thread gets jobs off the queue. When the queue is empty, it

383

# assumes there will be no more work and exits.

384

# (Realistically workers will run until terminated.)

385

def worker():

386

print 'Running worker'

387

time.sleep(0.1)

388

while True:

389

try:

390

arg = q.get(block=False)

391

except Queue.Empty:

392

print 'Worker', threading.currentThread(),

393

print 'queue empty'

394

break

395

else:

396

397

print 'running with argument', arg

398

time.sleep(0.5)

399

400

# Create queue

401

q = Queue.Queue()

402

403

# Start a pool of 5 workers

404

for i in range(5):

405

t = threading.Thread(target=worker, name='worker %i' % (i+1))

406

t.start()

407

408

# Begin adding work to the queue

409

for i in range(50):

410

q.put(i)

411

412

# Give threads time to run

413

print 'Main thread sleeping'

414

time.sleep(5)

415

</pre></div>

416

</div>

417

When run, this will produce the following output:

418

<div class="highlight-none"><div class="highlight"><pre>Running worker

419

Running worker

420

Running worker

421

Running worker

422

Running worker

423

Main thread sleeping

424

Worker <Thread(worker 1, started)> running with argument 0

425

Worker <Thread(worker 2, started)> running with argument 1

426

Worker <Thread(worker 3, started)> running with argument 2

427

Worker <Thread(worker 4, started)> running with argument 3

428

Worker <Thread(worker 5, started)> running with argument 4

429

Worker <Thread(worker 1, started)> running with argument 5

430

...

431

</pre></div>

432

</div>

433

Consult the module’s documentation for more details; the <a class="reference internal" href="../library/queue.html#Queue.Queue" title="Queue.Queue"><code class="xref py py-class docutils literal">Queue</code></a>

434

class provides a featureful interface.

435

</div>

436

437

<h3><a class="toc-backref" href="#id17">What kinds of global value mutation are thread-safe?</a><a class="headerlink" href="#what-kinds-of-global-value-mutation-are-thread-safe" title="Permalink to this headline">¶</a></h3>

438

A <a class="reference internal" href="../glossary.html#term-global-interpreter-lock">global interpreter lock</a> (GIL) is used internally to ensure that only

439

one thread runs in the Python VM at a time. In general, Python offers to switch

440

among threads only between bytecode instructions; how frequently it switches can

441

be set via <a class="reference internal" href="../library/sys.html#sys.setcheckinterval" title="sys.setcheckinterval"><code class="xref py py-func docutils literal">sys.setcheckinterval()</code></a>. Each bytecode instruction and

442

therefore all the C implementation code reached from each instruction is

443

therefore atomic from the point of view of a Python program.

444

In theory, this means an exact accounting requires an exact understanding of the

445

PVM bytecode implementation. In practice, it means that operations on shared

446

variables of built-in data types (ints, lists, dicts, etc) that “look atomic”

447

really are.

448

For example, the following operations are all atomic (L, L1, L2 are lists, D,

449

D1, D2 are dicts, x, y are objects, i, j are ints):

450

<div class="highlight-python"><div class="highlight"><pre>L.append(x)

451

L1.extend(L2)

452

x = L[i]

453

x = L.pop()

454

L1[i:j] = L2

455

L.sort()

456

x = y

457

x.field = y

458

D[x] = y

459

D1.update(D2)

460

D.keys()

461

</pre></div>

462

</div>

463

These aren’t:

464

<div class="highlight-python"><div class="highlight"><pre>i = i+1

465

L.append(L[-1])

466

L[i] = L[j]

467

D[x] = D[x] + 1

468

</pre></div>

469

</div>

470

Operations that replace other objects may invoke those other objects’

471

<a class="reference internal" href="../reference/datamodel.html#object.__del__" title="object.__del__"><code class="xref py py-meth docutils literal">__del__()</code></a> method when their reference count reaches zero, and that can

472

affect things. This is especially true for the mass updates to dictionaries and

473

lists. When in doubt, use a mutex!

474

</div>

475

476

<h3><a class="toc-backref" href="#id18">Can’t we get rid of the Global Interpreter Lock?</a><a class="headerlink" href="#can-t-we-get-rid-of-the-global-interpreter-lock" title="Permalink to this headline">¶</a></h3>

477

The <a class="reference internal" href="../glossary.html#term-global-interpreter-lock">global interpreter lock</a> (GIL) is often seen as a hindrance to Python’s

478

deployment on high-end multiprocessor server machines, because a multi-threaded

479

Python program effectively only uses one CPU, due to the insistence that

480

(almost) all Python code can only run while the GIL is held.

481

Back in the days of Python 1.5, Greg Stein actually implemented a comprehensive

482

patch set (the “free threading” patches) that removed the GIL and replaced it

483

with fine-grained locking. Unfortunately, even on Windows (where locks are very

484

efficient) this ran ordinary Python code about twice as slow as the interpreter

485

using the GIL. On Linux the performance loss was even worse because pthread

486

locks aren’t as efficient.

487

Since then, the idea of getting rid of the GIL has occasionally come up but

488

nobody has found a way to deal with the expected slowdown, and users who don’t

489

use threads would not be happy if their code ran at half the speed. Greg’s

490

free threading patch set has not been kept up-to-date for later Python versions.

491

This doesn’t mean that you can’t make good use of Python on multi-CPU machines!

492

You just have to be creative with dividing the work up between multiple

493

processes rather than multiple threads. Judicious use of C extensions will

494

also help; if you use a C extension to perform a time-consuming task, the

495

extension can release the GIL while the thread of execution is in the C code and

496

allow other threads to get some work done.

497

It has been suggested that the GIL should be a per-interpreter-state lock rather

498

than truly global; interpreters then wouldn’t be able to share objects.

499

Unfortunately, this isn’t likely to happen either. It would be a tremendous

500

amount of work, because many object implementations currently have global state.

501

For example, small integers and short strings are cached; these caches would

502

have to be moved to the interpreter state. Other object types have their own

503

free list; these free lists would have to be moved to the interpreter state.

504

And so on.

505

And I doubt that it can even be done in finite time, because the same problem

506

exists for 3rd party extensions. It is likely that 3rd party extensions are

507

being written at a faster rate than you can convert them to store all their

508

global state in the interpreter state.

509

And finally, once you have multiple interpreters not sharing any state, what

510

have you gained over running each interpreter in a separate process?

511

</div>

512

</div>

513

514

<h2><a class="toc-backref" href="#id19">Input and Output</a><a class="headerlink" href="#input-and-output" title="Permalink to this headline">¶</a></h2>

515

516

<h3><a class="toc-backref" href="#id20">How do I delete a file? (And other file questions...)</a><a class="headerlink" href="#how-do-i-delete-a-file-and-other-file-questions" title="Permalink to this headline">¶</a></h3>

517

Use <code class="docutils literal">os.remove(filename)</code> or <code class="docutils literal">os.unlink(filename)</code>; for documentation, see

518

the <a class="reference internal" href="../library/os.html#module-os" title="os: Miscellaneous operating system interfaces."><code class="xref py py-mod docutils literal">os</code></a> module. The two functions are identical; <code class="xref py py-func docutils literal">unlink()</code> is simply

519

the name of the Unix system call for this function.

520

To remove a directory, use <a class="reference internal" href="../library/os.html#os.rmdir" title="os.rmdir"><code class="xref py py-func docutils literal">os.rmdir()</code></a>; use <a class="reference internal" href="../library/os.html#os.mkdir" title="os.mkdir"><code class="xref py py-func docutils literal">os.mkdir()</code></a> to create one.

521

<code class="docutils literal">os.makedirs(path)</code> will create any intermediate directories in <code class="docutils literal">path</code> that

522

don’t exist. <code class="docutils literal">os.removedirs(path)</code> will remove intermediate directories as

523

long as they’re empty; if you want to delete an entire directory tree and its

524

contents, use <a class="reference internal" href="../library/shutil.html#shutil.rmtree" title="shutil.rmtree"><code class="xref py py-func docutils literal">shutil.rmtree()</code></a>.

525

To rename a file, use <code class="docutils literal">os.rename(old_path, new_path)</code>.

526

To truncate a file, open it using <code class="docutils literal">f = open(filename, "r+")</code>, and use

527

<code class="docutils literal">f.truncate(offset)</code>; offset defaults to the current seek position. There’s

528

also <code class="docutils literal">os.ftruncate(fd, offset)</code> for files opened with <a class="reference internal" href="../library/os.html#os.open" title="os.open"><code class="xref py py-func docutils literal">os.open()</code></a>, where

529

fd is the file descriptor (a small integer).

530

531

including <a class="reference internal" href="../library/shutil.html#shutil.copyfile" title="shutil.copyfile"><code class="xref py py-func docutils literal">copyfile()</code></a>, <a class="reference internal" href="../library/shutil.html#shutil.copytree" title="shutil.copytree"><code class="xref py py-func docutils literal">copytree()</code></a>, and

532

<a class="reference internal" href="../library/shutil.html#shutil.rmtree" title="shutil.rmtree"><code class="xref py py-func docutils literal">rmtree()</code></a>.

533

</div>

534

535

536

The <a class="reference internal" href="../library/shutil.html#module-shutil" title="shutil: High-level file operations, including copying."><code class="xref py py-mod docutils literal">shutil</code></a> module contains a <a class="reference internal" href="../library/shutil.html#shutil.copyfile" title="shutil.copyfile"><code class="xref py py-func docutils literal">copyfile()</code></a> function. Note

537

that on MacOS 9 it doesn’t copy the resource fork and Finder info.

538

</div>

539

540

<h3><a class="toc-backref" href="#id22">How do I read (or write) binary data?</a><a class="headerlink" href="#how-do-i-read-or-write-binary-data" title="Permalink to this headline">¶</a></h3>

541

To read or write complex binary data formats, it’s best to use the <a class="reference internal" href="../library/struct.html#module-struct" title="struct: Interpret strings as packed binary data."><code class="xref py py-mod docutils literal">struct</code></a>

542

module. It allows you to take a string containing binary data (usually numbers)

543

and convert it to Python objects; and vice versa.

544

For example, the following code reads two 2-byte integers and one 4-byte integer

545

in big-endian format from a file:

546

<div class="highlight-python"><div class="highlight"><pre>import struct

547

548

f = open(filename, "rb") # Open in binary mode for portability

549

s = f.read(8)

550

x, y, z = struct.unpack(">hhl", s)

551

</pre></div>

552

</div>

553

The ‘>’ in the format string forces big-endian data; the letter ‘h’ reads one

554

“short integer” (2 bytes), and ‘l’ reads one “long integer” (4 bytes) from the

555

string.

556

For data that is more regular (e.g. a homogeneous list of ints or floats),

557

you can also use the <a class="reference internal" href="../library/array.html#module-array" title="array: Space efficient arrays of uniformly typed numeric values."><code class="xref py py-mod docutils literal">array</code></a> module.

558

</div>

559

560

<h3><a class="toc-backref" href="#id23">I can’t seem to use os.read() on a pipe created with os.popen(); why?</a><a class="headerlink" href="#i-can-t-seem-to-use-os-read-on-a-pipe-created-with-os-popen-why" title="Permalink to this headline">¶</a></h3>

561

<a class="reference internal" href="../library/os.html#os.read" title="os.read"><code class="xref py py-func docutils literal">os.read()</code></a> is a low-level function which takes a file descriptor, a small

562

integer representing the opened file. <a class="reference internal" href="../library/os.html#os.popen" title="os.popen"><code class="xref py py-func docutils literal">os.popen()</code></a> creates a high-level

563

file object, the same type returned by the built-in <a class="reference internal" href="../library/functions.html#open" title="open"><code class="xref py py-func docutils literal">open()</code></a> function.

564

Thus, to read n bytes from a pipe p created with <a class="reference internal" href="../library/os.html#os.popen" title="os.popen"><code class="xref py py-func docutils literal">os.popen()</code></a>, you need to

565

use <code class="docutils literal">p.read(n)</code>.

566

</div>

567

568

<h3><a class="toc-backref" href="#id24">How do I run a subprocess with pipes connected to both input and output?</a><a class="headerlink" href="#how-do-i-run-a-subprocess-with-pipes-connected-to-both-input-and-output" title="Permalink to this headline">¶</a></h3>

569

Use the <a class="reference internal" href="../library/popen2.html#module-popen2" title="popen2: Subprocesses with accessible standard I/O streams. (deprecated)"><code class="xref py py-mod docutils literal">popen2</code></a> module. For example:

570

<div class="highlight-python"><div class="highlight"><pre>import popen2

571

fromchild, tochild = popen2.popen2("command")

572

tochild.write("input\n")

573

tochild.flush()

574

output = fromchild.readline()

575

</pre></div>

576

</div>

577

Warning: in general it is unwise to do this because you can easily cause a

578

deadlock where your process is blocked waiting for output from the child while

579

the child is blocked waiting for input from you. This can be caused by the

580

parent expecting the child to output more text than it does or by data being

581

stuck in stdio buffers due to lack of flushing. The Python parent

582

can of course explicitly flush the data it sends to the child before it reads

583

any output, but if the child is a naive C program it may have been written to

584

never explicitly flush its output, even if it is interactive, since flushing is

585

normally automatic.

586

Note that a deadlock is also possible if you use <code class="xref py py-func docutils literal">popen3()</code> to read stdout

587

and stderr. If one of the two is too large for the internal buffer (increasing

588

the buffer size does not help) and you <code class="docutils literal">read()</code> the other one first, there is

589

a deadlock, too.

590

Note on a bug in popen2: unless your program calls <code class="docutils literal">wait()</code> or <code class="docutils literal">waitpid()</code>,

591

finished child processes are never removed, and eventually calls to popen2 will

592

fail because of a limit on the number of child processes. Calling

593

<a class="reference internal" href="../library/os.html#os.waitpid" title="os.waitpid"><code class="xref py py-func docutils literal">os.waitpid()</code></a> with the <a class="reference internal" href="../library/os.html#os.WNOHANG" title="os.WNOHANG"><code class="xref py py-data docutils literal">os.WNOHANG</code></a> option can prevent this; a good

594

place to insert such a call would be before calling <code class="docutils literal">popen2</code> again.

595

In many cases, all you really need is to run some data through a command and get

596

the result back. Unless the amount of data is very large, the easiest way to do

597

this is to write it to a temporary file and run the command with that temporary

598

file as input. The standard module <a class="reference internal" href="../library/tempfile.html#module-tempfile" title="tempfile: Generate temporary files and directories."><code class="xref py py-mod docutils literal">tempfile</code></a> exports a

599

<a class="reference internal" href="../library/tempfile.html#tempfile.mktemp" title="tempfile.mktemp"><code class="xref py py-func docutils literal">mktemp()</code></a> function to generate unique temporary file names.

600

<div class="highlight-python"><div class="highlight"><pre>import tempfile

601

import os

602

603

class Popen3:

604

"""

605

This is a deadlock-safe version of popen that returns

606

an object with errorlevel, out (a string) and err (a string).

607

(capturestderr may not work under windows.)

608

Example: print Popen3('grep spam','\n\nhere spam\n\n').out

609

"""

610

def __init__(self,command,input=None,capturestderr=None):

611

outfile=tempfile.mktemp()

612

command="( %s ) > %s" % (command,outfile)

613

if input:

614

infile=tempfile.mktemp()

615

open(infile,"w").write(input)

616

command=command+" <"+infile

617

if capturestderr:

618

errfile=tempfile.mktemp()

619

command=command+" 2>"+errfile

620

self.errorlevel=os.system(command) >> 8

621

self.out=open(outfile,"r").read()

622

os.remove(outfile)

623

if input:

624

os.remove(infile)

625

if capturestderr:

626

self.err=open(errfile,"r").read()

627

os.remove(errfile)

628

</pre></div>

629

</div>

630

Note that many interactive programs (e.g. vi) don’t work well with pipes

631

substituted for standard input and output. You will have to use pseudo ttys

632

(“ptys”) instead of pipes. Or you can use a Python interface to Don Libes’

633

“expect” library. A Python extension that interfaces to expect is called “expy”

634

and available from <a class="reference external" href="http://expectpy.sourceforge.net">http://expectpy.sourceforge.net</a>. A pure Python solution that

635

works like expect is <a class="reference external" href="https://pypi.python.org/pypi/pexpect/">pexpect</a>.

636

</div>

637

638

<h3><a class="toc-backref" href="#id25">How do I access the serial (RS232) port?</a><a class="headerlink" href="#how-do-i-access-the-serial-rs232-port" title="Permalink to this headline">¶</a></h3>

639

For Win32, POSIX (Linux, BSD, etc.), Jython:

640

641

<div><a class="reference external" href="http://pyserial.sourceforge.net">http://pyserial.sourceforge.net</a></div></blockquote>

642

For Unix, see a Usenet post by Mitch Chapman:

643

644

<div><a class="reference external" href="https://groups.google.com/groups?selm=34A04430.CF9@ohioee.com">https://groups.google.com/groups?selm=34A04430.CF9@ohioee.com</a></div></blockquote>

645

</div>

646

647

<h3><a class="toc-backref" href="#id26">Why doesn’t closing sys.stdout (stdin, stderr) really close it?</a><a class="headerlink" href="#why-doesn-t-closing-sys-stdout-stdin-stderr-really-close-it" title="Permalink to this headline">¶</a></h3>

648

Python file objects are a high-level layer of abstraction on top of C streams,

649

which in turn are a medium-level layer of abstraction on top of (among other

650

things) low-level C file descriptors.

651

For most file objects you create in Python via the built-in <code class="docutils literal">file</code>

652

constructor, <code class="docutils literal">f.close()</code> marks the Python file object as being closed from

653

Python’s point of view, and also arranges to close the underlying C stream.

654

This also happens automatically in <code class="docutils literal">f</code>‘s destructor, when <code class="docutils literal">f</code> becomes

655

garbage.

656

But stdin, stdout and stderr are treated specially by Python, because of the

657

special status also given to them by C. Running <code class="docutils literal">sys.stdout.close()</code> marks

658

the Python-level file object as being closed, but does not close the

659

associated C stream.

660

To close the underlying C stream for one of these three, you should first be

661

sure that’s what you really want to do (e.g., you may confuse extension modules

662

trying to do I/O). If it is, use os.close:

663

<div class="highlight-python"><div class="highlight"><pre>os.close(0) # close C's stdin stream

664

os.close(1) # close C's stdout stream

665

os.close(2) # close C's stderr stream

666

</pre></div>

667

</div>

668

</div>

669

</div>

670

671

<h2><a class="toc-backref" href="#id27">Network/Internet Programming</a><a class="headerlink" href="#network-internet-programming" title="Permalink to this headline">¶</a></h2>

672

673

<h3><a class="toc-backref" href="#id28">What WWW tools are there for Python?</a><a class="headerlink" href="#what-www-tools-are-there-for-python" title="Permalink to this headline">¶</a></h3>

674

See the chapters titled <a class="reference internal" href="../library/internet.html#internet">Internet Protocols and Support</a> and <a class="reference internal" href="../library/netdata.html#netdata">Internet Data Handling</a> in the Library

675

Reference Manual. Python has many modules that will help you build server-side

676

and client-side web systems.

677

A summary of available frameworks is maintained by Paul Boddie at

678

<a class="reference external" href="https://wiki.python.org/moin/WebProgramming">https://wiki.python.org/moin/WebProgramming</a>.

679

Cameron Laird maintains a useful set of pages about Python web technologies at

680

<a class="reference external" href="http://phaseit.net/claird/comp.lang.python/web_python">http://phaseit.net/claird/comp.lang.python/web_python</a>.

681

</div>

682

683

<h3><a class="toc-backref" href="#id29">How can I mimic CGI form submission (METHOD=POST)?</a><a class="headerlink" href="#how-can-i-mimic-cgi-form-submission-method-post" title="Permalink to this headline">¶</a></h3>

684

I would like to retrieve web pages that are the result of POSTing a form. Is

685

there existing code that would let me do this easily?

686

Yes. Here’s a simple example that uses httplib:

687

<div class="highlight-python"><div class="highlight"><pre>#!/usr/local/bin/python

688

689

import httplib, sys, time

690

691

# build the query string

692

qs = "First=Josephine&MI=Q&Last=Public"

693

694

# connect and send the server a path

695

httpobj = httplib.HTTP('www.some-server.out-there', 80)

696

httpobj.putrequest('POST', '/cgi-bin/some-cgi-script')

697

# now generate the rest of the HTTP headers...

698

httpobj.putheader('Accept', '*/*')

699

httpobj.putheader('Connection', 'Keep-Alive')

700

httpobj.putheader('Content-type', 'application/x-www-form-urlencoded')

701

httpobj.putheader('Content-length', '%d' % len(qs))

702

httpobj.endheaders()

703

httpobj.send(qs)

704

# find out what the server said in response...

705

reply, msg, hdrs = httpobj.getreply()

706

if reply != 200:

707

sys.stdout.write(httpobj.getfile().read())

708

</pre></div>

709

</div>

710

Note that in general for percent-encoded POST operations, query strings must be

711

quoted using <a class="reference internal" href="../library/urllib.html#urllib.urlencode" title="urllib.urlencode"><code class="xref py py-func docutils literal">urllib.urlencode()</code></a>. For example, to send

712

<code class="docutils literal">name=Guy Steele, Jr.</code>:

713

<div class="highlight-python"><div class="highlight"><pre>>>> import urllib

714

>>> urllib.urlencode({'name': 'Guy Steele, Jr.'})

715

'name=Guy+Steele%2C+Jr.'

716

</pre></div>

717

</div>

718

</div>

719

720

<h3><a class="toc-backref" href="#id30">What module should I use to help with generating HTML?</a><a class="headerlink" href="#what-module-should-i-use-to-help-with-generating-html" title="Permalink to this headline">¶</a></h3>

721

You can find a collection of useful links on the <a class="reference external" href="https://wiki.python.org/moin/WebProgramming">Web Programming wiki page</a>.

722

</div>

723

724

<h3><a class="toc-backref" href="#id31">How do I send mail from a Python script?</a><a class="headerlink" href="#how-do-i-send-mail-from-a-python-script" title="Permalink to this headline">¶</a></h3>

725

Use the standard library module <a class="reference internal" href="../library/smtplib.html#module-smtplib" title="smtplib: SMTP protocol client (requires sockets)."><code class="xref py py-mod docutils literal">smtplib</code></a>.

726

Here’s a very simple interactive mail sender that uses it. This method will

727

work on any host that supports an SMTP listener.

728

<div class="highlight-python"><div class="highlight"><pre>import sys, smtplib

729

730

fromaddr = raw_input("From: ")

731

toaddrs = raw_input("To: ").split(',')

732

print "Enter message, end with ^D:"

733

msg = ''

734

while True:

735

line = sys.stdin.readline()

736

if not line:

737

break

738

msg += line

739

740

# The actual mail send

741

server = smtplib.SMTP('localhost')

742

server.sendmail(fromaddr, toaddrs, msg)

743

server.quit()

744

</pre></div>

745

</div>

746

A Unix-only alternative uses sendmail. The location of the sendmail program

747

varies between systems; sometimes it is <code class="docutils literal">/usr/lib/sendmail</code>, sometimes

748

<code class="docutils literal">/usr/sbin/sendmail</code>. The sendmail manual page will help you out. Here’s

749

some sample code:

750

<div class="highlight-python"><div class="highlight"><pre>import os

751

752

SENDMAIL = "/usr/sbin/sendmail" # sendmail location

753

p = os.popen("%s -t -i" % SENDMAIL, "w")

754

p.write("To: receiver@example.com\n")

755

p.write("Subject: test\n")

756

p.write("\n") # blank line separating headers from body

757

p.write("Some text\n")

758

p.write("some more text\n")

759

sts = p.close()

760

if sts != 0:

761

print "Sendmail exit status", sts

762

</pre></div>

763

</div>

764

</div>

765

766

<h3><a class="toc-backref" href="#id32">How do I avoid blocking in the connect() method of a socket?</a><a class="headerlink" href="#how-do-i-avoid-blocking-in-the-connect-method-of-a-socket" title="Permalink to this headline">¶</a></h3>

767

The select module is commonly used to help with asynchronous I/O on sockets.

768

To prevent the TCP connect from blocking, you can set the socket to non-blocking

769

mode. Then when you do the <code class="docutils literal">connect()</code>, you will either connect immediately

770

(unlikely) or get an exception that contains the error number as <code class="docutils literal">.errno</code>.

771

<code class="docutils literal">errno.EINPROGRESS</code> indicates that the connection is in progress, but hasn’t

772

finished yet. Different OSes will return different values, so you’re going to

773

have to check what’s returned on your system.

774

You can use the <code class="docutils literal">connect_ex()</code> method to avoid creating an exception. It will

775

just return the errno value. To poll, you can call <code class="docutils literal">connect_ex()</code> again later

776

– 0 or <code class="docutils literal">errno.EISCONN</code> indicate that you’re connected – or you can pass this

777

socket to select to check if it’s writable.

778

</div>

779

</div>

780

781

<h2><a class="toc-backref" href="#id33">Databases</a><a class="headerlink" href="#databases" title="Permalink to this headline">¶</a></h2>

782

783

<h3><a class="toc-backref" href="#id34">Are there any interfaces to database packages in Python?</a><a class="headerlink" href="#are-there-any-interfaces-to-database-packages-in-python" title="Permalink to this headline">¶</a></h3>

784

Yes.

785

Python 2.3 includes the <a class="reference internal" href="../library/bsddb.html#module-bsddb" title="bsddb: Interface to Berkeley DB database library"><code class="xref py py-mod docutils literal">bsddb</code></a> package which provides an interface to the

786

BerkeleyDB library. Interfaces to disk-based hashes such as <a class="reference internal" href="../library/dbm.html#module-dbm" title="dbm: The standard "database" interface, based on ndbm. (Unix)"><code class="xref py py-mod docutils literal">DBM</code></a>

787

and <a class="reference internal" href="../library/gdbm.html#module-gdbm" title="gdbm: GNU's reinterpretation of dbm. (Unix)"><code class="xref py py-mod docutils literal">GDBM</code></a> are also included with standard Python.

788

Support for most relational databases is available. See the

789

<a class="reference external" href="https://wiki.python.org/moin/DatabaseProgramming">DatabaseProgramming wiki page</a> for details.

790

</div>

791

792

<h3><a class="toc-backref" href="#id35">How do you implement persistent objects in Python?</a><a class="headerlink" href="#how-do-you-implement-persistent-objects-in-python" title="Permalink to this headline">¶</a></h3>

793

The <a class="reference internal" href="../library/pickle.html#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code class="xref py py-mod docutils literal">pickle</code></a> library module solves this in a very general way (though you

794

still can’t store things like open files, sockets or windows), and the

795

<a class="reference internal" href="../library/shelve.html#module-shelve" title="shelve: Python object persistence."><code class="xref py py-mod docutils literal">shelve</code></a> library module uses pickle and (g)dbm to create persistent

796

mappings containing arbitrary Python objects. For better performance, you can

797

use the <a class="reference internal" href="../library/pickle.html#module-cPickle" title="cPickle: Faster version of pickle, but not subclassable."><code class="xref py py-mod docutils literal">cPickle</code></a> module.

798

A more awkward way of doing things is to use pickle’s little sister, marshal.

799

The <a class="reference internal" href="../library/marshal.html#module-marshal" title="marshal: Convert Python objects to streams of bytes and back (with different constraints)."><code class="xref py py-mod docutils literal">marshal</code></a> module provides very fast ways to store noncircular basic

800

Python types to files and strings, and back again. Although marshal does not do

801

fancy things like store instances or handle shared references properly, it does

802

run extremely fast. For example, loading a half megabyte of data may take less

803

than a third of a second. This often beats doing something more complex and

804

general such as using gdbm with pickle/shelve.

805

</div>

806

807

<h3><a class="toc-backref" href="#id36">Why is cPickle so slow?</a><a class="headerlink" href="#why-is-cpickle-so-slow" title="Permalink to this headline">¶</a></h3>

808

By default <a class="reference internal" href="../library/pickle.html#module-pickle" title="pickle: Convert Python objects to streams of bytes and back."><code class="xref py py-mod docutils literal">pickle</code></a> uses a relatively old and slow format for backward

809

compatibility. You can however specify other protocol versions that are

810

faster:

811

<div class="highlight-python"><div class="highlight"><pre>largeString = 'z' * (100 * 1024)

812

myPickle = cPickle.dumps(largeString, protocol=1)

813

</pre></div>

814

</div>

815

</div>

816

817

<h3><a class="toc-backref" href="#id37">If my program crashes with a bsddb (or anydbm) database open, it gets corrupted. How come?</a><a class="headerlink" href="#if-my-program-crashes-with-a-bsddb-or-anydbm-database-open-it-gets-corrupted-how-come" title="Permalink to this headline">¶</a></h3>

818

Databases opened for write access with the bsddb module (and often by the anydbm

819

module, since it will preferentially use bsddb) must explicitly be closed using

820

the <code class="docutils literal">.close()</code> method of the database. The underlying library caches database

821

contents which need to be converted to on-disk form and written.

822

If you have initialized a new bsddb database but not written anything to it

823

before the program crashes, you will often wind up with a zero-length file and

824

encounter an exception the next time the file is opened.

825

</div>

826

827

<h3><a class="toc-backref" href="#id38">I tried to open Berkeley DB file, but bsddb produces bsddb.error: (22, ‘Invalid argument’). Help! How can I restore my data?</a><a class="headerlink" href="#i-tried-to-open-berkeley-db-file-but-bsddb-produces-bsddb-error-22-invalid-argument-help-how-can-i-restore-my-data" title="Permalink to this headline">¶</a></h3>

828

Don’t panic! Your data is probably intact. The most frequent cause for the error

829

is that you tried to open an earlier Berkeley DB file with a later version of

830

the Berkeley DB library.

831

Many Linux systems now have all three versions of Berkeley DB available. If you

832

are migrating from version 1 to a newer version use db_dump185 to dump a plain

833

text version of the database. If you are migrating from version 2 to version 3

834

use db2_dump to create a plain text version of the database. In either case,

835

use db_load to create a new native database for the latest version installed on

836

your computer. If you have version 3 of Berkeley DB installed, you should be

837

able to use db2_load to create a native version 2 database.

838

You should move away from Berkeley DB version 1 files because the hash file code

839

contains known bugs that can corrupt your data.

840

</div>

841

</div>

842

843

<h2><a class="toc-backref" href="#id39">Mathematics and Numerics</a><a class="headerlink" href="#mathematics-and-numerics" title="Permalink to this headline">¶</a></h2>

844

845

<h3><a class="toc-backref" href="#id40">How do I generate random numbers in Python?</a><a class="headerlink" href="#how-do-i-generate-random-numbers-in-python" title="Permalink to this headline">¶</a></h3>

846

The standard module <a class="reference internal" href="../library/random.html#module-random" title="random: Generate pseudo-random numbers with various common distributions."><code class="xref py py-mod docutils literal">random</code></a> implements a random number generator. Usage

847

is simple:

848

<div class="highlight-python"><div class="highlight"><pre>import random

849

random.random()

850

</pre></div>

851

</div>

852

This returns a random floating point number in the range [0, 1).

853

There are also many other specialized generators in this module, such as:

854

855

<li><code class="docutils literal">randrange(a, b)</code> chooses an integer in the range [a, b).</li>

856

<li><code class="docutils literal">uniform(a, b)</code> chooses a floating point number in the range [a, b).</li>

857

<li><code class="docutils literal">normalvariate(mean, sdev)</code> samples the normal (Gaussian) distribution.</li>

858

</ul>

859

Some higher-level functions operate on sequences directly, such as:

860

861

<li><code class="docutils literal">choice(S)</code> chooses random element from a given sequence</li>

862

<li><code class="docutils literal">shuffle(L)</code> shuffles a list in-place, i.e. permutes it randomly</li>

863

</ul>

864

There’s also a <code class="docutils literal">Random</code> class you can instantiate to create independent

865

multiple random number generators.

866

</div>

867

</div>

868

</div>

869

870

871

</div>

872

</div>

873

</div>

874

875

876

<h3><a href="../contents.html">Table Of Contents</a></h3>

877

<ul>

878

<li><a class="reference internal" href="#">Library and Extension FAQ</a><ul>

879

<li><a class="reference internal" href="#general-library-questions">General Library Questions</a></li>

880

<li><a class="reference internal" href="#common-tasks">Common tasks</a></li>

881

<li><a class="reference internal" href="#threads">Threads</a></li>

882

<li><a class="reference internal" href="#input-and-output">Input and Output</a></li>

883

<li><a class="reference internal" href="#network-internet-programming">Network/Internet Programming</a></li>

884

<li><a class="reference internal" href="#databases">Databases</a></li>

885

<li><a class="reference internal" href="#mathematics-and-numerics">Mathematics and Numerics</a></li>

886

</ul>

887

</li>

888

</ul>

889

890

<h4>Previous topic</h4>

891

<a href="design.html"

892

title="previous chapter">Design and History FAQ</a>

893

<h4>Next topic</h4>

894

<a href="extending.html"

895

title="next chapter">Extending/Embedding FAQ</a>

896

897

898

<li><a href="../bugs.html">Report a Bug</a></li>

899

<li><a href="../_sources/faq/library.txt"

900

rel="nofollow">Show Source</a></li>

901

</ul>

902

903

904

<h3>Quick search</h3>

905

906

907

908

909

910

</form>

911

912

Enter search terms or a module, class or function name.

913

914

</div>

915

916

</div>

917

</div>

918

919

</div>

920

921

<h3>Navigation</h3>

922

<ul>

923

924

<a href="../genindex.html" title="General Index"

925

>index</a></li>

926

927

<a href="../py-modindex.html" title="Python Module Index"

928

>modules</a> |</li>

929

930

<a href="extending.html" title="Extending/Embedding FAQ"

931

>next</a> |</li>

932

933

<a href="design.html" title="Design and History FAQ"

934

>previous</a> |</li>

935

<li><img src="../_static/py.png" alt=""

936

style="vertical-align: middle; margin-top: -1px"/></li>

937

<li><a href="https://www.python.org/">Python</a> »</li>

938

<li>

939

2.7.12

940

<a href="../index.html">Documentation</a> »

941

</li>

942

943

<li class="nav-item nav-item-1"><a href="index.html" >Python Frequently Asked Questions</a> »</li>

944

</ul>

945

</div>

946

947

948

949

The Python Software Foundation is a non-profit corporation.

950

<a href="https://www.python.org/psf/donations/">Please donate.</a>

951

952

Last updated on Sep 20, 2016.

953

<a href="../bugs.html">Found a bug</a>?

954

955

Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.3.3.

956

</div>

957

958

</body>

959

</html>

b'\\ No newline at end of file'

Older »