~dkuhlman/python-training-materials/Materials

<h1>1. Embedding Python in Another Application<a class="headerlink" href="#embedding-python-in-another-application" title="Permalink to this headline">¶</a></h1>

The previous chapters discussed how to extend Python, that is, how to extend the

functionality of Python by attaching a library of C functions to it. It is also

possible to do it the other way around: enrich your C/C++ application by

embedding Python in it. Embedding provides your application with the ability to

implement some of the functionality of your application in Python rather than C

or C++. This can be used for many purposes; one example would be to allow users

to tailor the application to their needs by writing some scripts in Python. You

can also use it yourself if some of the functionality can be written in Python

more easily.

Embedding Python is similar to extending it, but not quite. The difference is

that when you extend Python, the main program of the application is still the

Python interpreter, while if you embed Python, the main program may have nothing

to do with Python — instead, some parts of the application occasionally call

the Python interpreter to run some Python code.

So if you are embedding Python, you are providing your own main program. One of

the things this main program has to do is initialize the Python interpreter. At

the very least, you have to call the function <a class="reference internal" href="../c-api/init.html#c.Py_Initialize" title="Py_Initialize"><code class="xref c c-func docutils literal">Py_Initialize()</code></a>. There are

optional calls to pass command line arguments to Python. Then later you can

call the interpreter from any part of the application.

There are several different ways to call the interpreter: you can pass a string

containing Python statements to <a class="reference internal" href="../c-api/veryhigh.html#c.PyRun_SimpleString" title="PyRun_SimpleString"><code class="xref c c-func docutils literal">PyRun_SimpleString()</code></a>, or you can pass a

stdio file pointer and a file name (for identification in error messages only)

100

to <a class="reference internal" href="../c-api/veryhigh.html#c.PyRun_SimpleFile" title="PyRun_SimpleFile"><code class="xref c c-func docutils literal">PyRun_SimpleFile()</code></a>. You can also call the lower-level operations

101

described in the previous chapters to construct and use Python objects.

102

103

See also

104

105

<dt><a class="reference internal" href="../c-api/index.html#c-api-index">Python/C API Reference Manual</a></dt>

106

<dd>The details of Python’s C interface are given in this manual. A great deal of

107

necessary information can be found here.</dd>

108

</dl>

109

</div>

110

111

<h2>1.1. Very High Level Embedding<a class="headerlink" href="#very-high-level-embedding" title="Permalink to this headline">¶</a></h2>

112

The simplest form of embedding Python is the use of the very high level

113

interface. This interface is intended to execute a Python script without needing

114

to interact with the application directly. This can for example be used to

115

perform some operation on a file.

116

<div class="highlight-c"><div class="highlight"><pre>#include <Python.h>

117

118

int

119

main(int argc, char *argv[])

120

{

121

wchar_t *program = Py_DecodeLocale(argv[0], NULL);

122

if (program == NULL) {

123

fprintf(stderr, "Fatal error: cannot decode argv[0]\n");

124

exit(1);

125

}

126

Py_SetProgramName(program); /* optional but recommended */

127

Py_Initialize();

128

PyRun_SimpleString("from time import time,ctime\n"

129

"print('Today is', ctime(time()))\n");

130

Py_Finalize();

131

PyMem_RawFree(program);

132

return 0;

133

}

134

</pre></div>

135

</div>

136

The <a class="reference internal" href="../c-api/init.html#c.Py_SetProgramName" title="Py_SetProgramName"><code class="xref c c-func docutils literal">Py_SetProgramName()</code></a> function should be called before

137

<a class="reference internal" href="../c-api/init.html#c.Py_Initialize" title="Py_Initialize"><code class="xref c c-func docutils literal">Py_Initialize()</code></a> to inform the interpreter about paths to Python run-time

138

libraries. Next, the Python interpreter is initialized with

139

<a class="reference internal" href="../c-api/init.html#c.Py_Initialize" title="Py_Initialize"><code class="xref c c-func docutils literal">Py_Initialize()</code></a>, followed by the execution of a hard-coded Python script

140

that prints the date and time. Afterwards, the <a class="reference internal" href="../c-api/init.html#c.Py_Finalize" title="Py_Finalize"><code class="xref c c-func docutils literal">Py_Finalize()</code></a> call shuts

141

the interpreter down, followed by the end of the program. In a real program,

142

you may want to get the Python script from another source, perhaps a text-editor

143

routine, a file, or a database. Getting the Python code from a file can better

144

be done by using the <a class="reference internal" href="../c-api/veryhigh.html#c.PyRun_SimpleFile" title="PyRun_SimpleFile"><code class="xref c c-func docutils literal">PyRun_SimpleFile()</code></a> function, which saves you the

145

trouble of allocating memory space and loading the file contents.

146

</div>

147

148

<h2>1.2. Beyond Very High Level Embedding: An overview<a class="headerlink" href="#beyond-very-high-level-embedding-an-overview" title="Permalink to this headline">¶</a></h2>

149

The high level interface gives you the ability to execute arbitrary pieces of

150

Python code from your application, but exchanging data values is quite

151

cumbersome to say the least. If you want that, you should use lower level calls.

152

At the cost of having to write more C code, you can achieve almost anything.

153

It should be noted that extending Python and embedding Python is quite the same

154

activity, despite the different intent. Most topics discussed in the previous

155

chapters are still valid. To show this, consider what the extension code from

156

Python to C really does:

157

158

<li>Convert data values from Python to C,</li>

159

<li>Perform a function call to a C routine using the converted values, and</li>

160

<li>Convert the data values from the call from C to Python.</li>

161

</ol>

162

When embedding Python, the interface code does:

163

164

<li>Convert data values from C to Python,</li>

165

<li>Perform a function call to a Python interface routine using the converted

166

values, and</li>

167

<li>Convert the data values from the call from Python to C.</li>

168

</ol>

169

As you can see, the data conversion steps are simply swapped to accommodate the

170

different direction of the cross-language transfer. The only difference is the

171

routine that you call between both data conversions. When extending, you call a

172

C routine, when embedding, you call a Python routine.

173

This chapter will not discuss how to convert data from Python to C and vice

174

versa. Also, proper use of references and dealing with errors is assumed to be

175

understood. Since these aspects do not differ from extending the interpreter,

176

you can refer to earlier chapters for the required information.

177

</div>

178

179

<h2>1.3. Pure Embedding<a class="headerlink" href="#pure-embedding" title="Permalink to this headline">¶</a></h2>

180

The first program aims to execute a function in a Python script. Like in the

181

section about the very high level interface, the Python interpreter does not

182

directly interact with the application (but that will change in the next

183

section).

184

The code to run a function defined in a Python script is:

185

<div class="highlight-c"><div class="highlight"><pre>#include <Python.h>

186

187

int

188

189

{

190

PyObject *pName, *pModule, *pDict, *pFunc;

191

PyObject *pArgs, *pValue;

192

int i;

193

194

if (argc < 3) {

195

fprintf(stderr,"Usage: call pythonfile funcname [args]\n");

196

return 1;

197

}

198

199

Py_Initialize();

200

pName = PyUnicode_DecodeFSDefault(argv[1]);

201

/* Error checking of pName left out */

202

203

pModule = PyImport_Import(pName);

204

Py_DECREF(pName);

205

206

if (pModule != NULL) {

207

pFunc = PyObject_GetAttrString(pModule, argv[2]);

208

/* pFunc is a new reference */

209

210

if (pFunc && PyCallable_Check(pFunc)) {

211

pArgs = PyTuple_New(argc - 3);

212

for (i = 0; i < argc - 3; ++i) {

213

pValue = PyLong_FromLong(atoi(argv[i + 3]));

214

if (!pValue) {

215

Py_DECREF(pArgs);

216

Py_DECREF(pModule);

217

fprintf(stderr, "Cannot convert argument\n");

218

return 1;

219

}

220

/* pValue reference stolen here: */

221

PyTuple_SetItem(pArgs, i, pValue);

222

}

223

pValue = PyObject_CallObject(pFunc, pArgs);

224

Py_DECREF(pArgs);

225

if (pValue != NULL) {

226

printf("Result of call: %ld\n", PyLong_AsLong(pValue));

227

Py_DECREF(pValue);

228

}

229

else {

230

Py_DECREF(pFunc);

231

Py_DECREF(pModule);

232

PyErr_Print();

233

fprintf(stderr,"Call failed\n");

234

return 1;

235

}

236

}

237

else {

238

if (PyErr_Occurred())

239

PyErr_Print();

240

fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);

241

}

242

Py_XDECREF(pFunc);

243

Py_DECREF(pModule);

244

}

245

else {

246

PyErr_Print();

247

fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);

248

return 1;

249

}

250

Py_Finalize();

251

return 0;

252

}

253

</pre></div>

254

</div>

255

This code loads a Python script using <code class="docutils literal">argv[1]</code>, and calls the function named

256

in <code class="docutils literal">argv[2]</code>. Its integer arguments are the other values of the <code class="docutils literal">argv</code>

257

array. If you <a class="reference internal" href="#compiling">compile and link</a> this program (let’s call

258

the finished executable call), and use it to execute a Python

259

script, such as:

260

<div class="highlight-python"><div class="highlight"><pre>def multiply(a,b):

261

print("Will compute", a, "times", b)

262

c = 0

263

for i in range(0, a):

264

c = c + b

265

return c

266

</pre></div>

267

</div>

268

then the result should be:

269

<div class="highlight-c"><div class="highlight"><pre>$ call multiply multiply 3 2

270

Will compute 3 times 2

271

Result of call: 6

272

</pre></div>

273

</div>

274

Although the program is quite large for its functionality, most of the code is

275

for data conversion between Python and C, and for error reporting. The

276

interesting part with respect to embedding Python starts with

277

<div class="highlight-c"><div class="highlight"><pre>Py_Initialize();

278

279

/* Error checking of pName left out */

280

pModule = PyImport_Import(pName);

281

</pre></div>

282

</div>

283

After initializing the interpreter, the script is loaded using

284

<a class="reference internal" href="../c-api/import.html#c.PyImport_Import" title="PyImport_Import"><code class="xref c c-func docutils literal">PyImport_Import()</code></a>. This routine needs a Python string as its argument,

285

which is constructed using the <a class="reference internal" href="../c-api/unicode.html#c.PyUnicode_FromString" title="PyUnicode_FromString"><code class="xref c c-func docutils literal">PyUnicode_FromString()</code></a> data conversion

286

routine.

287

<div class="highlight-c"><div class="highlight"><pre>pFunc = PyObject_GetAttrString(pModule, argv[2]);

288

/* pFunc is a new reference */

289

290

291

...

292

}

293

Py_XDECREF(pFunc);

294

</pre></div>

295

</div>

296

Once the script is loaded, the name we’re looking for is retrieved using

297

<a class="reference internal" href="../c-api/object.html#c.PyObject_GetAttrString" title="PyObject_GetAttrString"><code class="xref c c-func docutils literal">PyObject_GetAttrString()</code></a>. If the name exists, and the object returned is

298

callable, you can safely assume that it is a function. The program then

299

proceeds by constructing a tuple of arguments as normal. The call to the Python

300

function is then made with:

301

<div class="highlight-c"><div class="highlight"><pre>pValue = PyObject_CallObject(pFunc, pArgs);

302

</pre></div>

303

</div>

304

Upon return of the function, <code class="docutils literal">pValue</code> is either NULL or it contains a

305

reference to the return value of the function. Be sure to release the reference

306

after examining the value.

307

</div>

308

309

<h2>1.4. Extending Embedded Python<a class="headerlink" href="#extending-embedded-python" title="Permalink to this headline">¶</a></h2>

310

Until now, the embedded Python interpreter had no access to functionality from

311

the application itself. The Python API allows this by extending the embedded

312

interpreter. That is, the embedded interpreter gets extended with routines

313

provided by the application. While it sounds complex, it is not so bad. Simply

314

forget for a while that the application starts the Python interpreter. Instead,

315

consider the application to be a set of subroutines, and write some glue code

316

that gives Python access to those routines, just like you would write a normal

317

Python extension. For example:

318

<div class="highlight-c"><div class="highlight"><pre>static int numargs=0;

319

320

/* Return the number of arguments of the application command line */

321

static PyObject*

322

emb_numargs(PyObject *self, PyObject *args)

323

{

324

if(!PyArg_ParseTuple(args, ":numargs"))

325

return NULL;

326

return PyLong_FromLong(numargs);

327

}

328

329

static PyMethodDef EmbMethods[] = {

330

{"numargs", emb_numargs, METH_VARARGS,

331

"Return the number of arguments received by the process."},

332

{NULL, NULL, 0, NULL}

333

};

334

335

static PyModuleDef EmbModule = {

336

PyModuleDef_HEAD_INIT, "emb", NULL, -1, EmbMethods,

337

NULL, NULL, NULL, NULL

338

};

339

340

static PyObject*

341

PyInit_emb(void)

342

{

343

return PyModule_Create(&EmbModule);

344

}

345

</pre></div>

346

</div>

347

Insert the above code just above the <code class="xref c c-func docutils literal">main()</code> function. Also, insert the

348

following two statements before the call to <a class="reference internal" href="../c-api/init.html#c.Py_Initialize" title="Py_Initialize"><code class="xref c c-func docutils literal">Py_Initialize()</code></a>:

349

<div class="highlight-c"><div class="highlight"><pre>numargs = argc;

350

PyImport_AppendInittab("emb", &PyInit_emb);

351

</pre></div>

352

</div>

353

These two lines initialize the <code class="docutils literal">numargs</code> variable, and make the

354

<code class="xref py py-func docutils literal">emb.numargs()</code> function accessible to the embedded Python interpreter.

355

With these extensions, the Python script can do things like

356

<div class="highlight-python"><div class="highlight"><pre>import emb

357

print("Number of arguments", emb.numargs())

358

</pre></div>

359

</div>

360

In a real application, the methods will expose an API of the application to

361

Python.

362

</div>

363

364

<h2>1.5. Embedding Python in C++<a class="headerlink" href="#embedding-python-in-c" title="Permalink to this headline">¶</a></h2>

365

It is also possible to embed Python in a C++ program; precisely how this is done

366

will depend on the details of the C++ system used; in general you will need to

367

write the main program in C++, and use the C++ compiler to compile and link your

368

program. There is no need to recompile Python itself using C++.

369

</div>

370

371

<h2>1.6. Compiling and Linking under Unix-like systems<a class="headerlink" href="#compiling-and-linking-under-unix-like-systems" title="Permalink to this headline">¶</a></h2>

372

It is not necessarily trivial to find the right flags to pass to your

373

compiler (and linker) in order to embed the Python interpreter into your

374

application, particularly because Python needs to load library modules

375

implemented as C dynamic extensions (<code class="file docutils literal">.so</code> files) linked against

376

it.

377

To find out the required compiler and linker flags, you can execute the

378

<code class="file docutils literal">pythonX.Y-config</code> script which is generated as part of the

379

installation process (a <code class="file docutils literal">python3-config</code> script may also be

380

available). This script has several options, of which the following will

381

be directly useful to you:

382

<ul>

383

<li><code class="docutils literal">pythonX.Y-config --cflags</code> will give you the recommended flags when

384

compiling:

385

<div class="highlight-c"><div class="highlight"><pre>$ /opt/bin/python3.4-config --cflags

386

-I/opt/include/python3.4m -I/opt/include/python3.4m -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

387

</pre></div>

388

</div>

389

</li>

390

<li><code class="docutils literal">pythonX.Y-config --ldflags</code> will give you the recommended flags when

391

linking:

392

<div class="highlight-c"><div class="highlight"><pre>$ /opt/bin/python3.4-config --ldflags

393

-L/opt/lib/python3.4/config-3.4m -lpthread -ldl -lutil -lm -lpython3.4m -Xlinker -export-dynamic

394

</pre></div>

395

</div>

396

</li>

397

</ul>

398

399

Note

400

To avoid confusion between several Python installations (and especially

401

between the system Python and your own compiled Python), it is recommended

402

that you use the absolute path to <code class="file docutils literal">pythonX.Y-config</code>, as in the above

403

example.

404

</div>

405

If this procedure doesn’t work for you (it is not guaranteed to work for

406

all Unix-like platforms; however, we welcome <a class="reference internal" href="../bugs.html#reporting-bugs">bug reports</a>)

407

you will have to read your system’s documentation about dynamic linking and/or

408

examine Python’s <code class="file docutils literal">Makefile</code> (use <a class="reference internal" href="../library/sysconfig.html#sysconfig.get_makefile_filename" title="sysconfig.get_makefile_filename"><code class="xref py py-func docutils literal">sysconfig.get_makefile_filename()</code></a>

409

to find its location) and compilation

410

options. In this case, the <a class="reference internal" href="../library/sysconfig.html#module-sysconfig" title="sysconfig: Python's configuration information"><code class="xref py py-mod docutils literal">sysconfig</code></a> module is a useful tool to

411

programmatically extract the configuration values that you will want to

412

combine together. For example:

413

<div class="highlight-python"><div class="highlight"><pre>>>> import sysconfig

414

>>> sysconfig.get_config_var('LIBS')

415

'-lpthread -ldl -lutil'

416

>>> sysconfig.get_config_var('LINKFORSHARED')

417

'-Xlinker -export-dynamic'

418

</pre></div>

419

</div>

420

</div>

421

</div>

422

423

424

</div>

425

</div>

426

</div>

427

428

429

<h3><a href="../contents.html">Table Of Contents</a></h3>

430

<ul>

431

<li><a class="reference internal" href="#">1. Embedding Python in Another Application</a><ul>

432

<li><a class="reference internal" href="#very-high-level-embedding">1.1. Very High Level Embedding</a></li>

433

<li><a class="reference internal" href="#beyond-very-high-level-embedding-an-overview">1.2. Beyond Very High Level Embedding: An overview</a></li>

434

<li><a class="reference internal" href="#pure-embedding">1.3. Pure Embedding</a></li>

435

<li><a class="reference internal" href="#extending-embedded-python">1.4. Extending Embedded Python</a></li>

436

<li><a class="reference internal" href="#embedding-python-in-c">1.5. Embedding Python in C++</a></li>

437

<li><a class="reference internal" href="#compiling-and-linking-under-unix-like-systems">1.6. Compiling and Linking under Unix-like systems</a></li>

438

</ul>

439

</li>

440

</ul>

441

442

<h4>Previous topic</h4>

443

<a href="windows.html"

444

title="previous chapter">4. Building C and C++ Extensions on Windows</a>

445

<h4>Next topic</h4>

446

<a href="../c-api/index.html"

447

title="next chapter">Python/C API Reference Manual</a>

448

449

450

<li><a href="../bugs.html">Report a Bug</a></li>

451

<li><a href="../_sources/extending/embedding.txt"

452

rel="nofollow">Show Source</a></li>

453

</ul>

454

455

456

<h3>Quick search</h3>

457

458

459

460

461

462

</form>

463

464

Enter search terms or a module, class or function name.

465

466

</div>

467

468

</div>

469

</div>

470

471

</div>

472

473

<h3>Navigation</h3>

474

<ul>

475

476

<a href="../genindex.html" title="General Index"

477

>index</a></li>

478

479

<a href="../py-modindex.html" title="Python Module Index"

480

>modules</a> |</li>

481

482

<a href="../c-api/index.html" title="Python/C API Reference Manual"

483

>next</a> |</li>

484

485

<a href="windows.html" title="4. Building C and C++ Extensions on Windows"

486

>previous</a> |</li>

487

<li><img src="../_static/py.png" alt=""

488

style="vertical-align: middle; margin-top: -1px"/></li>

489

<li><a href="https://www.python.org/">Python</a> »</li>

490

<li>

491

3.5.1

492

<a href="../index.html">Documentation </a> »

493

</li>

494

495

<li class="nav-item nav-item-1"><a href="index.html" >Extending and Embedding the Python Interpreter</a> »</li>

496

</ul>

497

</div>

498

499

500

501

The Python Software Foundation is a non-profit corporation.

502

<a href="https://www.python.org/psf/donations/">Please donate.</a>

503

504

Last updated on Jan 22, 2016.

505

<a href="../bugs.html">Found a bug</a>?

506

507

Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.3.3.

508

</div>

509

510

</body>

511

</html>

b'\\ No newline at end of file'

Older »