~ubuntu-branches/ubuntu/trusty/python-boto/trusty

<h1>An Introduction to boto’s DynamoDB v2 interface<a class="headerlink" href="#an-introduction-to-boto-s-dynamodb-v2-interface" title="Permalink to this headline">¶</a></h1>

This tutorial focuses on the boto interface to AWS’ <a class="reference external" href="http://aws.amazon.com/dynamodb/">DynamoDB</a> v2. This tutorial

assumes that you have boto already downloaded and installed.

Warning

This tutorial covers the SECOND major release of DynamoDB (including

local secondary index support). The documentation for the original

version of DynamoDB (& boto’s support for it) is at

<a class="reference internal" href="dynamodb_tut.html">DynamoDB v1</a>.

</div>

The v2 DynamoDB API has both a high-level & low-level component. The low-level

API (contained primarily within <tt class="docutils literal">boto.dynamodb2.layer1</tt>) provides an

interface that rough matches exactly what is provided by the API. It supports

all options available to the service.

The high-level API attempts to make interacting with the service more natural

from Python. It supports most of the featureset.

<h2>The High-Level API<a class="headerlink" href="#the-high-level-api" title="Permalink to this headline">¶</a></h2>

Most of the interaction centers around a single object, the <tt class="docutils literal">Table</tt>. Tables

act as a way to effectively namespace your records. If you’re familiar with

database tables from an RDBMS, tables will feel somewhat familiar.

<h3>Creating a New Table<a class="headerlink" href="#creating-a-new-table" title="Permalink to this headline">¶</a></h3>

To create a new table, you need to call <tt class="docutils literal">Table.create</tt> & specify (at a

minimum) both the table’s name as well as the key schema for the table.

Since both the key schema and local secondary indexes can not be

modified after the table is created, you’ll need to plan ahead of time how you

think the table will be used. Both the keys & indexes are also used for

querying, so you’ll want to represent the data you’ll need when querying

there as well.

For the schema, you can either have a single <tt class="docutils literal">HashKey</tt> or a combined

<tt class="docutils literal">HashKey+RangeKey</tt>. The <tt class="docutils literal">HashKey</tt> by itself should be thought of as a

unique identifier (for instance, like a username or UUID). It is typically

looked up as an exact value.

A <tt class="docutils literal">HashKey+RangeKey</tt> combination is slightly different, in that the

<tt class="docutils literal">HashKey</tt> acts like a namespace/prefix & the <tt class="docutils literal">RangeKey</tt> acts as a value

that can be referred to by a sorted range of values.

For the local secondary indexes, you can choose from an <tt class="docutils literal">AllIndex</tt>, a

<tt class="docutils literal">KeysOnlyIndex</tt> or a <tt class="docutils literal">IncludeIndex</tt> field. Each builds an index of values

that can be queried on. The <tt class="docutils literal">AllIndex</tt> duplicates all values onto the index

(to prevent additional reads to fetch the data). The <tt class="docutils literal">KeysOnlyIndex</tt>

duplicates only the keys from the schema onto the index. The <tt class="docutils literal">IncludeIndex</tt>

lets you specify a list of fieldnames to duplicate over.

100

Simple example:

101

102

>>> from boto.dynamodb2.table import Table

103

104

# Uses your ``aws_access_key_id`` & ``aws_secret_access_key`` from either a

105

# config file or environment variable & the default region.

106

>>> users = Table.create('users', schema=[

107

... HashKey('username'),

108

... ])

109

</pre></div>

110

</div>

111

A full example:

112

<div class="highlight-python"><div class="highlight"><pre>>>> import boto.dynamodb2

113

>>> from boto.dynamodb2.fields import HashKey, RangeKey, KeysOnlyIndex, AllIndex

114

>>> from boto.dynamodb2.table import Table

115

>>> from boto.dynamodb2.types import NUMBER

116

117

118

... HashKey('account_type', data_type=NUMBER),

119

... RangeKey('last_name'),

120

... ], throughput={

121

... 'read': 5,

122

... 'write': 15,

123

... }, indexes=[

124

... AllIndex('EverythingIndex', parts=[

125

126

... ])

127

... ],

128

... # If you need to specify custom parameters like keys or region info...

129

... connection= boto.dynamodb2.connect_to_region('us-east-1'))

130

</pre></div>

131

</div>

132

</div>

133

134

<h3>Using an Existing Table<a class="headerlink" href="#using-an-existing-table" title="Permalink to this headline">¶</a></h3>

135

Once a table has been created, using it is relatively simple. You can either

136

specify just the <tt class="docutils literal">table_name</tt> (allowing the object to lazily do an additional

137

call to get details about itself if needed) or provide the <tt class="docutils literal">schema/indexes</tt>

138

again (same as what was used with <tt class="docutils literal">Table.create</tt>) to avoid extra overhead.

139

Lazy example:

140

<div class="highlight-python"><div class="highlight"><pre>>>> from boto.dynamodb2.table import Table

141

>>> users = Table('users')

142

</pre></div>

143

</div>

144

Efficient example:

145

<div class="highlight-python"><div class="highlight"><pre>>>> from boto.dynamodb2.fields import HashKey, RangeKey, AllIndex

146

>>> from boto.dynamodb2.table import Table

147

>>> from boto.dynamodb2.types import NUMBER

148

>>> users = Table('users', schema=[

149

150

... RangeKey('last_name'),

151

... ], indexes=[

152

153

154

... ])

155

... ])

156

</pre></div>

157

</div>

158

</div>

159

160

<h3>Creating a New Item<a class="headerlink" href="#creating-a-new-item" title="Permalink to this headline">¶</a></h3>

161

Once you have a <tt class="docutils literal">Table</tt> instance, you can add new items to the table. There

162

are two ways to do this.

163

The first is to use the <tt class="docutils literal">Table.put_item</tt> method. Simply hand it a dictionary

164

of data & it will create the item on the server side. This dictionary should

165

be relatively flat (as you can nest in other dictionaries) & must contain

166

the keys used in the <tt class="docutils literal">schema</tt>.

167

Example:

168

169

>>> users = Table('users')

170

171

# Create the new user.

172

>>> users.put_item(data={

173

... 'username': 'johndoe',

174

... 'first_name': 'John',

175

... 'last_name': 'Doe',

176

... })

177

True

178

</pre></div>

179

</div>

180

The alternative is to manually construct an <tt class="docutils literal">Item</tt> instance & tell it to

181

<tt class="docutils literal">save</tt> itself. This is useful if the object will be around for awhile & you

182

don’t want to re-fetch it.

183

Example:

184

<div class="highlight-python"><div class="highlight"><pre>>>> from boto.dynamodb2.items import Item

185

>>> from boto.dynamodb2.table import Table

186

>>> users = Table('users')

187

188

# WARNING - This doens't save it yet!

189

>>> johndoe = Item(users, data={

190

... 'username': 'johndoe',

191

... 'first_name': 'John',

192

... 'last_name': 'Doe',

193

... })

194

# The data now gets persisted to the server.

195

>>> johndoe.save()

196

True

197

</pre></div>

198

</div>

199

</div>

200

201

<h3>Getting an Item & Accessing Data<a class="headerlink" href="#getting-an-item-accessing-data" title="Permalink to this headline">¶</a></h3>

202

With data now in DynamoDB, if you know the key of the item, you can fetch it

203

back out. Specify the key value(s) as kwargs to <tt class="docutils literal">Table.get_item</tt>.

204

Example:

205

206

>>> users = Table('users')

207

208

>>> johndoe = users.get_item(username='johndoe')

209

</pre></div>

210

</div>

211

Once you have an <tt class="docutils literal">Item</tt> instance, it presents a dictionary-like interface to

212

the data.:

213

<div class="highlight-python"><div class="highlight"><pre>>>> johndoe = users.get_item(username='johndoe')

214

215

# Read a field out.

216

>>> johndoe['first_name']

217

'John'

218

219

# Change a field (DOESN'T SAVE YET!).

220

>>> johndoe['first_name'] = 'Johann'

221

222

# Delete data from it (DOESN'T SAVE YET!).

223

>>> del johndoe['last_name']

224

</pre></div>

225

</div>

226

</div>

227

228

<h3>Updating an Item<a class="headerlink" href="#updating-an-item" title="Permalink to this headline">¶</a></h3>

229

Just creating new items or changing only the in-memory version of the <tt class="docutils literal">Item</tt>

230

isn’t particularly effective. To persist the changes to DynamoDB, you have

231

three choices.

232

The first is sending all the data with the expectation nothing has changed

233

since you read the data. DynamoDB will verify the data is in the original state

234

and, if so, will all of the item’s data. If that expectation fails, the call

235

will fail:

236

237

>>> johndoe['first_name'] = 'Johann'

238

>>> johndoe['whatever'] = "man, that's just like your opinion"

239

>>> del johndoe['last_name']

240

241

# Affects all fields, even the ones not changed locally.

242

>>> johndoe.save()

243

True

244

</pre></div>

245

</div>

246

The second is a full overwrite. If you can be confident your version of the

247

data is the most correct, you can force an overwrite of the data.:

248

249

>>> johndoe['first_name'] = 'Johann'

250

251

>>> del johndoe['last_name']

252

253

# Specify ``overwrite=True`` to fully replace the data.

254

>>> johndoe.save(overwrite=True)

255

True

256

</pre></div>

257

</div>

258

The last is a partial update. If you’ve only modified certain fields, you

259

can send a partial update that only writes those fields, allowing other

260

(potentially changed) fields to go untouched.:

261

262

>>> johndoe['first_name'] = 'Johann'

263

264

>>> del johndoe['last_name']

265

266

# Partial update, only sending/affecting the

267

# ``first_name/whatever/last_name`` fields.

268

>>> johndoe.partial_save()

269

True

270

</pre></div>

271

</div>

272

</div>

273

274

<h3>Deleting an Item<a class="headerlink" href="#deleting-an-item" title="Permalink to this headline">¶</a></h3>

275

You can also delete items from the table. You have two choices, depending on

276

what data you have present.

277

If you already have an <tt class="docutils literal">Item</tt> instance, the easiest approach is just to call

278

<tt class="docutils literal">Item.delete</tt>.:

279

<div class="highlight-python"><div class="highlight"><pre>>>> johndoe.delete()

280

True

281

</pre></div>

282

</div>

283

If you don’t have an <tt class="docutils literal">Item</tt> instance & you don’t want to incur the

284

<tt class="docutils literal">Table.get_item</tt> call to get it, you can call <tt class="docutils literal">Table.delete_item</tt> method.:

285

286

>>> users = Table('users')

287

288

>>> users.delete_item(username='johndoe')

289

True

290

</pre></div>

291

</div>

292

</div>

293

294

<h3>Batch Writing<a class="headerlink" href="#batch-writing" title="Permalink to this headline">¶</a></h3>

295

If you’re loading a lot of data at a time, making use of batch writing can

296

both speed up the process & reduce the number of write requests made to the

297

service.

298

Batch writing involves wrapping the calls you want batched in a context manager.

299

The context manager immitates the <tt class="docutils literal">Table.put_item</tt> & <tt class="docutils literal">Table.delete_item</tt>

300

APIs. Getting & using the context manager looks like:

301

302

>>> users = Table('users')

303

304

>>> with users.batch_write() as batch:

305

... batch.put_item(data={

306

... 'username': 'anotherdoe',

307

... 'first_name': 'Another',

308

... 'last_name': 'Doe',

309

... 'date_joined': int(time.time()),

310

... })

311

312

... 'username': 'alice',

313

... 'first_name': 'Alice',

314

315

... })

316

... batch.delete_item(username=jane')

317

</pre></div>

318

</div>

319

However, there are some limitations on what you can do within the context

320

manager.

321

322

<li>It can’t read data at all or do batch any other operations.</li>

323

<li>You can’t put & delete the same data within a batch request.</li>

324

</ul>

325

326

Note

327

Additionally, the context manager can only batch 25 items at a time for a

328

request (this is a DynamoDB limitation). It is handled for you so you can

329

keep writing additional items, but you should be aware that 100 <tt class="docutils literal">put_item</tt>

330

calls is 4 batch requests, not 1.

331

</div>

332

</div>

333

334

<h3>Querying<a class="headerlink" href="#querying" title="Permalink to this headline">¶</a></h3>

335

Manually fetching out each item by itself isn’t tenable for large datasets.

336

To cope with fetching many records, you can either perform a standard query,

337

query via a local secondary index or scan the entire table.

338

A standard query typically gets run against a hash+range key combination.

339

Filter parameters are passed as kwargs & use a <tt class="docutils literal">__</tt> to separate the fieldname

340

from the operator being used to filter the value.

341

In terms of querying, our original schema is less than optimal. For the

342

following examples, we’ll be using the following table setup:

343

<div class="highlight-python"><div class="highlight"><pre>>>> users = Table.create('users', schema=[

344

... HashKey('account_type'),

345

... RangeKey('last_name'),

346

... ], indexes=[

347

... AllIndex('DateJoinedIndex', parts=[

348

... HashKey('account_type'),

349

... RangeKey('date_joined', data_type=NUMBER),

350

... ]),

351

... ])

352

</pre></div>

353

</div>

354

When executing the query, you get an iterable back that contains your results.

355

These results may be spread over multiple requests as DynamoDB paginates them.

356

This is done transparently, but you should be aware it may take more than one

357

request.

358

To run a query for last names starting with the letter “D”:

359

<div class="highlight-python"><div class="highlight"><pre>>>> names_with_d = users.query(

360

... account_type__eq='standard_user',

361

... last_name__beginswith='D'

362

... )

363

364

>>> for user in names_with_d:

365

... print user['first_name']

366

'Bob'

367

'Jane'

368

'John'

369

</pre></div>

370

</div>

371

You can also reverse results (<tt class="docutils literal">reverse=True</tt>) as well as limiting them

372

(<tt class="docutils literal">limit=2</tt>):

373

<div class="highlight-python"><div class="highlight"><pre>>>> rev_with_d = users.query(

374

... account_type__eq='standard_user',

375

... last_name__beginswith='D',

376

... reverse=True,

377

... limit=2

378

... )

379

380

>>> for user in rev_with_d:

381

... print user['first_name']

382

'John'

383

'Jane'

384

</pre></div>

385

</div>

386

You can also run queries against the local secondary indexes. Simply provide

387

the index name (<tt class="docutils literal">index='FirstNameIndex'</tt>) & filter parameters against its

388

fields:

389

<div class="highlight-python"><pre># Users within the last hour.

390

>>> recent = users.query(

391

... account_type__eq='standard_user',

392

... date_joined__gte=time.time() - (60 * 60),

393

... index='DateJoinedIndex'

394

... )

395

396

>>> for user in recent:

397

... print user['first_name']

398

'Alice'

399

'Jane'</pre>

400

</div>

401

Finally, if you need to query on data that’s not in either a key or in an

402

index, you can run a <tt class="docutils literal">Table.scan</tt> across the whole table, which accepts a

403

similar but expanded set of filters. If you’re familiar with the Map/Reduce

404

concept, this is akin to what DynamoDB does.

405

406

Warning

407

Scans are consistent & run over the entire table, so relatively speaking,

408

they’re more expensive than plain queries or queries against an LSI.

409

</div>

410

An example scan of all records in the table looks like:

411

<div class="highlight-python"><div class="highlight"><pre>>>> all_users = users.scan()

412

</pre></div>

413

</div>

414

Filtering a scan looks like:

415

<div class="highlight-python"><div class="highlight"><pre>>>> owners_with_emails = users.scan(

416

... is_owner__eq=1,

417

... email__null=False,

418

... )

419

420

>>> for user in recent:

421

... print user['first_name']

422

'George'

423

'John'

424

</pre></div>

425

</div>

426

</div>

427

428

<h3>Parallel Scan<a class="headerlink" href="#parallel-scan" title="Permalink to this headline">¶</a></h3>

429

DynamoDB also includes a feature called “Parallel Scan”, which allows you

430

to make use of extra read capacity to divide up your result set & scan

431

an entire table faster.

432

This does require extra code on the user’s part & you should ensure that

433

you need the speed boost, have enough data to justify it and have the extra

434

capacity to read it without impacting other queries/scans.

435

To run it, you should pick the <tt class="docutils literal">total_segments</tt> to use, which is an integer

436

representing the number of temporary partitions you’d divide your table into.

437

You then need to spin up a thread/process for each one, giving each

438

thread/process a <tt class="docutils literal">segment</tt>, which is a zero-based integer of the segment

439

you’d like to scan.

440

An example of using parallel scan to send out email to all users might look

441

something like:

442

<div class="highlight-python"><div class="highlight"><pre>#!/usr/bin/env python

443

import threading

444

445

import boto.ses

446

import boto.dynamodb2

447

from boto.dynamodb2.table import Table

448

449

450

AWS_ACCESS_KEY_ID = '<YOUR_AWS_KEY_ID>'

451

AWS_SECRET_ACCESS_KEY = '<YOUR_AWS_SECRET_KEY>'

452

APPROVED_EMAIL = 'some@address.com'

453

454

455

def send_email(email):

456

# Using Amazon's Simple Email Service, send an email to a given

457

# email address. You must already have an email you've verified with

458

# AWS before this will work.

459

conn = boto.ses.connect_to_region(

460

'us-east-1',

461

aws_access_key_id=AWS_ACCESS_KEY_ID,

462

aws_secret_access_key=AWS_SECRET_ACCESS_KEY

463

)

464

conn.send_email(

465

APPROVED_EMAIL,

466

"[OurSite] New feature alert!",

467

"We've got some exciting news! We added a new feature to...",

468

[email]

469

)

470

471

472

def process_segment(segment=0, total_segments=10):

473

# This method/function is executed in each thread, each getting its

474

# own segment to process through.

475

conn = boto.dynamodb2.connect_to_region(

476

'us-east-1',

477

aws_access_key_id=AWS_ACCESS_KEY_ID,

478

aws_secret_access_key=AWS_SECRET_ACCESS_KEY

479

)

480

table = Table('users', connection=conn)

481

482

# We pass in the segment & total_segments to scan here.

483

for user in table.scan(segment=segment, total_segments=total_segments):

484

send_email(user['email'])

485

486

487

def send_all_emails():

488

pool = []

489

# We're choosing to divide the table in 3, then...

490

pool_size = 3

491

492

# ...spinning up a thread for each segment.

493

for i in range(pool_size):

494

worker = threading.Thread(

495

target=process_segment,

496

kwargs={

497

'segment': i,

498

'total_segments': pool_size,

499

}

500

)

501

pool.append(worker)

502

# We start them to let them start scanning & consuming their

503

# assigned segment.

504

worker.start()

505

506

# Finally, we wait for each to finish.

507

for thread in pool:

508

thread.join()

509

510

511

if __name__ == '__main__':

512

send_all_emails()

513

</pre></div>

514

</div>

515

</div>

516

517

<h3>Batch Reading<a class="headerlink" href="#batch-reading" title="Permalink to this headline">¶</a></h3>

518

Similar to batch writing, batch reading can also help reduce the number of

519

API requests necessary to access a large number of items. The

520

<tt class="docutils literal">Table.batch_get</tt> method takes a list (or any sliceable collection) of keys

521

& fetches all of them, presented as an iterator interface.

522

This is done lazily, so if you never iterate over the results, no requests are

523

executed. Additionally, if you only iterate over part of the set, the minumum

524

number of calls are made to fetch those results (typically max 100 per

525

response).

526

Example:

527

528

>>> users = Table('users')

529

530

# No request yet.

531

>>> many_users = users.batch_get(keys=[

532

{'username': 'alice'},

533

{'username': 'bob'},

534

{'username': 'fred'},

535

{'username': 'jane'},

536

{'username': 'johndoe'},

537

])

538

539

# Now the request is performed, requesting all five in one request.

540

>>> for user in many_users:

541

... print user['first_name']

542

'Alice'

543

'Bobby'

544

'Fred'

545

'Jane'

546

'John'

547

</pre></div>

548

</div>

549

</div>

550

551

<h3>Deleting a Table<a class="headerlink" href="#deleting-a-table" title="Permalink to this headline">¶</a></h3>

552

Deleting a table is a simple exercise. When you no longer need a table, simply

553

run:

554

<div class="highlight-python"><div class="highlight"><pre>>>> users.delete()

555

</pre></div>

556

</div>

557

</div>

558

559

<h3>DynamoDB Local<a class="headerlink" href="#dynamodb-local" title="Permalink to this headline">¶</a></h3>

560

<a class="reference external" href="http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.html">Amazon DynamoDB Local</a> is a utility which can be used to mock DynamoDB

561

during development. Connecting to a running DynamoDB Local server is easy:

562

<div class="highlight-python"><div class="highlight"><pre>#!/usr/bin/env python

563

from boto.dynamodb2.layer1 import DynamoDBConnection

564

565

566

# Connect to DynamoDB Local

567

conn = DynamoDBConnection(

568

host='localhost',

569

port=8000,

570

aws_secret_access_key='anything',

571

is_secure=False)

572

573

# List all local tables

574

tables = conn.list_tables()

575

</pre></div>

576

</div>

577

</div>

578

579

<h3>Next Steps<a class="headerlink" href="#next-steps" title="Permalink to this headline">¶</a></h3>

580

You can find additional information about other calls & parameter options

581

in the <a class="reference internal" href="ref/dynamodb2.html">API docs</a>.

582

</div>

583

</div>

584

</div>

585

586

587

</div>

588

</div>

589

</div>

590

591

592

<h3><a href="index.html">Table Of Contents</a></h3>

593

<ul>

594

<li><a class="reference internal" href="#">An Introduction to boto’s DynamoDB v2 interface</a><ul>

595

<li><a class="reference internal" href="#the-high-level-api">The High-Level API</a><ul>

596

<li><a class="reference internal" href="#creating-a-new-table">Creating a New Table</a></li>

597

<li><a class="reference internal" href="#using-an-existing-table">Using an Existing Table</a></li>

598

<li><a class="reference internal" href="#creating-a-new-item">Creating a New Item</a></li>

599

<li><a class="reference internal" href="#getting-an-item-accessing-data">Getting an Item & Accessing Data</a></li>

600

<li><a class="reference internal" href="#updating-an-item">Updating an Item</a></li>

601

<li><a class="reference internal" href="#deleting-an-item">Deleting an Item</a></li>

602

<li><a class="reference internal" href="#batch-writing">Batch Writing</a></li>

603

<li><a class="reference internal" href="#querying">Querying</a></li>

604

<li><a class="reference internal" href="#parallel-scan">Parallel Scan</a></li>

605

<li><a class="reference internal" href="#batch-reading">Batch Reading</a></li>

606

<li><a class="reference internal" href="#deleting-a-table">Deleting a Table</a></li>

607

<li><a class="reference internal" href="#dynamodb-local">DynamoDB Local</a></li>

608

<li><a class="reference internal" href="#next-steps">Next Steps</a></li>

609

</ul>

610

</li>

611

</ul>

612

</li>

613

</ul>

614

615

<h4>Previous topic</h4>

616

<a href="support_tut.html"

617

title="previous chapter">An Introduction to boto’s Support interface</a>

618

<h4>Next topic</h4>

619

<a href="migrations/dynamodb_v1_to_v2.html"

620

title="next chapter">Migrating from DynamoDB v1 to DynamoDB v2</a>

621

622

623

<li><a href="_sources/dynamodb2_tut.txt"

624

rel="nofollow">Show Source</a></li>

625

</ul>

626

627

<h3>Quick search</h3>

628

629

630

631

632

633

</form>

634

635

Enter search terms or a module, class or function name.

636

637

</div>

638

639

</div>

640

</div>

641

642

</div>

643

644

<h3>Navigation</h3>

645

<ul>

646

647

<a href="genindex.html" title="General Index"

648

>index</a></li>

649

650

<a href="py-modindex.html" title="Python Module Index"

651

>modules</a> |</li>

652

653

<a href="migrations/dynamodb_v1_to_v2.html" title="Migrating from DynamoDB v1 to DynamoDB v2"

654

>next</a> |</li>

655

656

<a href="support_tut.html" title="An Introduction to boto’s Support interface"

657

>previous</a> |</li>

658

659

</ul>

660

</div>

661

662

663

Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3.

664

</div>

665

</body>

666

</html>

b'\\ No newline at end of file'

Older »