6
PostgreSQL's transaction system is a three-layer system. The bottom layer
7
implements low-level transactions and subtransactions, on top of which rests
8
the mainloop's control code, which in turn implements user-visible
9
transactions and savepoints.
11
The middle layer of code is called by postgres.c before and after the
12
processing of each query, or after detecting an error:
14
StartTransactionCommand
15
CommitTransactionCommand
16
AbortCurrentTransaction
18
Meanwhile, the user can alter the system's state by issuing the SQL commands
19
BEGIN, COMMIT, ROLLBACK, SAVEPOINT, ROLLBACK TO or RELEASE. The traffic cop
20
redirects these calls to the toplevel routines
24
UserAbortTransactionBlock
29
respectively. Depending on the current state of the system, these functions
30
call low level functions to activate the real transaction system:
41
Additionally, within a transaction, CommandCounterIncrement is called to
42
increment the command counter, which allows future commands to "see" the
43
effects of previous commands within the same transaction. Note that this is
44
done automatically by CommitTransactionCommand after each query inside a
45
transaction block, but some utility functions also do it internally to allow
46
some operations (usually in the system catalogs) to be seen by future
47
operations in the same utility command. (For example, in DefineRelation it is
48
done after creating the heap so the pg_class row is visible, to be able to
52
For example, consider the following sequence of user commands:
56
3) INSERT INTO foo VALUES (...)
59
In the main processing loop, this results in the following function call
62
/ StartTransactionCommand;
64
1) < ProcessUtility; << BEGIN
65
\ BeginTransactionBlock;
66
\ CommitTransactionCommand;
68
/ StartTransactionCommand;
69
2) / ProcessQuery; << SELECT ...
70
\ CommitTransactionCommand;
71
\ CommandCounterIncrement;
73
/ StartTransactionCommand;
74
3) / ProcessQuery; << INSERT ...
75
\ CommitTransactionCommand;
76
\ CommandCounterIncrement;
78
/ StartTransactionCommand;
79
/ ProcessUtility; << COMMIT
80
4) < EndTransactionBlock;
81
\ CommitTransactionCommand;
84
The point of this example is to demonstrate the need for
85
StartTransactionCommand and CommitTransactionCommand to be state smart -- they
86
should call CommandCounterIncrement between the calls to BeginTransactionBlock
87
and EndTransactionBlock and outside these calls they need to do normal start,
88
commit or abort processing.
90
Furthermore, suppose the "SELECT * FROM foo" caused an abort condition. In
91
this case AbortCurrentTransaction is called, and the transaction is put in
92
aborted state. In this state, any user input is ignored except for
93
transaction-termination statements, or ROLLBACK TO <savepoint> commands.
95
Transaction aborts can occur in two ways:
97
1) system dies from some internal cause (syntax error, etc)
98
2) user types ROLLBACK
100
The reason we have to distinguish them is illustrated by the following two
105
1) user types BEGIN 1) user types BEGIN
106
2) user does something 2) user does something
107
3) user does not like what 3) system aborts for some reason
108
she sees and types ABORT (syntax error, etc)
110
In case 1, we want to abort the transaction and return to the default state.
111
In case 2, there may be more commands coming our way which are part of the
112
same transaction block; we have to ignore these commands until we see a COMMIT
115
Internal aborts are handled by AbortCurrentTransaction, while user aborts are
116
handled by UserAbortTransactionBlock. Both of them rely on AbortTransaction
117
to do all the real work. The only difference is what state we enter after
118
AbortTransaction does its work:
120
* AbortCurrentTransaction leaves us in TBLOCK_ABORT,
121
* UserAbortTransactionBlock leaves us in TBLOCK_ABORT_END
123
Low-level transaction abort handling is divided in two phases:
124
* AbortTransaction executes as soon as we realize the transaction has
125
failed. It should release all shared resources (locks etc) so that we do
126
not delay other backends unnecessarily.
127
* CleanupTransaction executes when we finally see a user COMMIT
128
or ROLLBACK command; it cleans things up and gets us out of the transaction
129
completely. In particular, we mustn't destroy TopTransactionContext until
132
Also, note that when a transaction is committed, we don't close it right away.
133
Rather it's put in TBLOCK_END state, which means that when
134
CommitTransactionCommand is called after the query has finished processing,
135
the transaction has to be closed. The distinction is subtle but important,
136
because it means that control will leave the xact.c code with the transaction
137
open, and the main loop will be able to keep processing inside the same
138
transaction. So, in a sense, transaction commit is also handled in two
139
phases, the first at EndTransactionBlock and the second at
140
CommitTransactionCommand (which is where CommitTransaction is actually
143
The rest of the code in xact.c are routines to support the creation and
144
finishing of transactions and subtransactions. For example, AtStart_Memory
145
takes care of initializing the memory subsystem at main transaction start.
148
Subtransaction Handling
149
-----------------------
151
Subtransactions are implemented using a stack of TransactionState structures,
152
each of which has a pointer to its parent transaction's struct. When a new
153
subtransaction is to be opened, PushTransaction is called, which creates a new
154
TransactionState, with its parent link pointing to the current transaction.
155
StartSubTransaction is in charge of initializing the new TransactionState to
156
sane values, and properly initializing other subsystems (AtSubStart routines).
158
When closing a subtransaction, either CommitSubTransaction has to be called
159
(if the subtransaction is committing), or AbortSubTransaction and
160
CleanupSubTransaction (if it's aborting). In either case, PopTransaction is
161
called so the system returns to the parent transaction.
163
One important point regarding subtransaction handling is that several may need
164
to be closed in response to a single user command. That's because savepoints
165
have names, and we allow to commit or rollback a savepoint by name, which is
166
not necessarily the one that was last opened. Also a COMMIT or ROLLBACK
167
command must be able to close out the entire stack. We handle this by having
168
the utility command subroutine mark all the state stack entries as commit-
169
pending or abort-pending, and then when the main loop reaches
170
CommitTransactionCommand, the real work is done. The main point of doing
171
things this way is that if we get an error while popping state stack entries,
172
the remaining stack entries still show what we need to do to finish up.
174
In the case of ROLLBACK TO <savepoint>, we abort all the subtransactions up
175
through the one identified by the savepoint name, and then re-create that
176
subtransaction level with the same name. So it's a completely new
177
subtransaction as far as the internals are concerned.
179
Other subsystems are allowed to start "internal" subtransactions, which are
180
handled by BeginInternalSubtransaction. This is to allow implementing
181
exception handling, e.g. in PL/pgSQL. ReleaseCurrentSubTransaction and
182
RollbackAndReleaseCurrentSubTransaction allows the subsystem to close said
183
subtransactions. The main difference between this and the savepoint/release
184
path is that we execute the complete state transition immediately in each
185
subroutine, rather than deferring some work until CommitTransactionCommand.
186
Another difference is that BeginInternalSubtransaction is allowed when no
187
explicit transaction block has been established, while DefineSavepoint is not.
190
Transaction and Subtransaction Numbering
191
----------------------------------------
193
Transactions and subtransactions are assigned permanent XIDs only when/if
194
they first do something that requires one --- typically, insert/update/delete
195
a tuple, though there are a few other places that need an XID assigned.
196
If a subtransaction requires an XID, we always first assign one to its
197
parent. This maintains the invariant that child transactions have XIDs later
198
than their parents, which is assumed in a number of places.
200
The subsidiary actions of obtaining a lock on the XID and and entering it into
201
pg_subtrans and PG_PROC are done at the time it is assigned.
203
A transaction that has no XID still needs to be identified for various
204
purposes, notably holding locks. For this purpose we assign a "virtual
205
transaction ID" or VXID to each top-level transaction. VXIDs are formed from
206
two fields, the backendID and a backend-local counter; this arrangement allows
207
assignment of a new VXID at transaction start without any contention for
208
shared memory. To ensure that a VXID isn't re-used too soon after backend
209
exit, we store the last local counter value into shared memory at backend
210
exit, and initialize it from the previous value for the same backendID slot
211
at backend start. All these counters go back to zero at shared memory
212
re-initialization, but that's OK because VXIDs never appear anywhere on-disk.
214
Internally, a backend needs a way to identify subtransactions whether or not
215
they have XIDs; but this need only lasts as long as the parent top transaction
216
endures. Therefore, we have SubTransactionId, which is somewhat like
217
CommandId in that it's generated from a counter that we reset at the start of
218
each top transaction. The top-level transaction itself has SubTransactionId 1,
219
and subtransactions have IDs 2 and up. (Zero is reserved for
220
InvalidSubTransactionId.) Note that subtransactions do not have their
221
own VXIDs; they use the parent top transaction's VXID.
224
Interlocking Transaction Begin, Transaction End, and Snapshots
225
--------------------------------------------------------------
227
We try hard to minimize the amount of overhead and lock contention involved
228
in the frequent activities of beginning/ending a transaction and taking a
229
snapshot. Unfortunately, we must have some interlocking for this, because
230
we must ensure consistency about the commit order of transactions.
231
For example, suppose an UPDATE in xact A is blocked by xact B's prior
232
update of the same row, and xact B is doing commit while xact C gets a
233
snapshot. Xact A can complete and commit as soon as B releases its locks.
234
If xact C's GetSnapshotData sees xact B as still running, then it had
235
better see xact A as still running as well, or it will be able to see two
236
tuple versions - one deleted by xact B and one inserted by xact A. Another
237
reason why this would be bad is that C would see (in the row inserted by A)
238
earlier changes by B, and it would be inconsistent for C not to see any
239
of B's changes elsewhere in the database.
241
Formally, the correctness requirement is "if a snapshot A considers
242
transaction X as committed, and any of transaction X's snapshots considered
243
transaction Y as committed, then snapshot A must consider transaction Y as
246
What we actually enforce is strict serialization of commits and rollbacks
247
with snapshot-taking: we do not allow any transaction to exit the set of
248
running transactions while a snapshot is being taken. (This rule is
249
stronger than necessary for consistency, but is relatively simple to
250
enforce, and it assists with some other issues as explained below.) The
251
implementation of this is that GetSnapshotData takes the ProcArrayLock in
252
shared mode (so that multiple backends can take snapshots in parallel),
253
but ProcArrayEndTransaction must take the ProcArrayLock in exclusive mode
254
while clearing MyProc->xid at transaction end (either commit or abort).
256
ProcArrayEndTransaction also holds the lock while advancing the shared
257
latestCompletedXid variable. This allows GetSnapshotData to use
258
latestCompletedXid + 1 as xmax for its snapshot: there can be no
259
transaction >= this xid value that the snapshot needs to consider as
262
In short, then, the rule is that no transaction may exit the set of
263
currently-running transactions between the time we fetch latestCompletedXid
264
and the time we finish building our snapshot. However, this restriction
265
only applies to transactions that have an XID --- read-only transactions
266
can end without acquiring ProcArrayLock, since they don't affect anyone
267
else's snapshot nor latestCompletedXid.
269
Transaction start, per se, doesn't have any interlocking with these
270
considerations, since we no longer assign an XID immediately at transaction
271
start. But when we do decide to allocate an XID, GetNewTransactionId must
272
store the new XID into the shared ProcArray before releasing XidGenLock.
273
This ensures that all top-level XIDs <= latestCompletedXid are either
274
present in the ProcArray, or not running anymore. (This guarantee doesn't
275
apply to subtransaction XIDs, because of the possibility that there's not
276
room for them in the subxid array; instead we guarantee that they are
277
present or the overflow flag is set.) If a backend released XidGenLock
278
before storing its XID into MyProc, then it would be possible for another
279
backend to allocate and commit a later XID, causing latestCompletedXid to
280
pass the first backend's XID, before that value became visible in the
281
ProcArray. That would break GetOldestXmin, as discussed below.
283
We allow GetNewTransactionId to store the XID into MyProc->xid (or the
284
subxid array) without taking ProcArrayLock. This was once necessary to
285
avoid deadlock; while that is no longer the case, it's still beneficial for
286
performance. We are thereby relying on fetch/store of an XID to be atomic,
287
else other backends might see a partially-set XID. This also means that
288
readers of the ProcArray xid fields must be careful to fetch a value only
289
once, rather than assume they can read it multiple times and get the same
290
answer each time. (Use volatile-qualified pointers when doing this, to
291
ensure that the C compiler does exactly what you tell it to.)
293
Another important activity that uses the shared ProcArray is GetOldestXmin,
294
which must determine a lower bound for the oldest xmin of any active MVCC
295
snapshot, system-wide. Each individual backend advertises the smallest
296
xmin of its own snapshots in MyProc->xmin, or zero if it currently has no
297
live snapshots (eg, if it's between transactions or hasn't yet set a
298
snapshot for a new transaction). GetOldestXmin takes the MIN() of the
299
valid xmin fields. It does this with only shared lock on ProcArrayLock,
300
which means there is a potential race condition against other backends
301
doing GetSnapshotData concurrently: we must be certain that a concurrent
302
backend that is about to set its xmin does not compute an xmin less than
303
what GetOldestXmin returns. We ensure that by including all the active
304
XIDs into the MIN() calculation, along with the valid xmins. The rule that
305
transactions can't exit without taking exclusive ProcArrayLock ensures that
306
concurrent holders of shared ProcArrayLock will compute the same minimum of
307
currently-active XIDs: no xact, in particular not the oldest, can exit
308
while we hold shared ProcArrayLock. So GetOldestXmin's view of the minimum
309
active XID will be the same as that of any concurrent GetSnapshotData, and
310
so it can't produce an overestimate. If there is no active transaction at
311
all, GetOldestXmin returns latestCompletedXid + 1, which is a lower bound
312
for the xmin that might be computed by concurrent or later GetSnapshotData
313
calls. (We know that no XID less than this could be about to appear in
314
the ProcArray, because of the XidGenLock interlock discussed above.)
316
GetSnapshotData also performs an oldest-xmin calculation (which had better
317
match GetOldestXmin's) and stores that into RecentGlobalXmin, which is used
318
for some tuple age cutoff checks where a fresh call of GetOldestXmin seems
319
too expensive. Note that while it is certain that two concurrent
320
executions of GetSnapshotData will compute the same xmin for their own
321
snapshots, as argued above, it is not certain that they will arrive at the
322
same estimate of RecentGlobalXmin. This is because we allow XID-less
323
transactions to clear their MyProc->xmin asynchronously (without taking
324
ProcArrayLock), so one execution might see what had been the oldest xmin,
325
and another not. This is OK since RecentGlobalXmin need only be a valid
326
lower bound. As noted above, we are already assuming that fetch/store
327
of the xid fields is atomic, so assuming it for xmin as well is no extra
331
pg_clog and pg_subtrans
332
-----------------------
334
pg_clog and pg_subtrans are permanent (on-disk) storage of transaction related
335
information. There is a limited number of pages of each kept in memory, so
336
in many cases there is no need to actually read from disk. However, if
337
there's a long running transaction or a backend sitting idle with an open
338
transaction, it may be necessary to be able to read and write this information
339
from disk. They also allow information to be permanent across server restarts.
341
pg_clog records the commit status for each transaction that has been assigned
342
an XID. A transaction can be in progress, committed, aborted, or
343
"sub-committed". This last state means that it's a subtransaction that's no
344
longer running, but its parent has not updated its state yet. It is not
345
necessary to update a subtransaction's transaction status to subcommit, so we
346
can just defer it until main transaction commit. The main role of marking
347
transactions as sub-committed is to provide an atomic commit protocol when
348
transaction status is spread across multiple clog pages. As a result, whenever
349
transaction status spreads across multiple pages we must use a two-phase commit
350
protocol: the first phase is to mark the subtransactions as sub-committed, then
351
we mark the top level transaction and all its subtransactions committed (in
352
that order). Thus, subtransactions that have not aborted appear as in-progress
353
even when they have already finished, and the subcommit status appears as a
354
very short transitory state during main transaction commit. Subtransaction
355
abort is always marked in clog as soon as it occurs. When the transaction
356
status all fit in a single CLOG page, we atomically mark them all as committed
357
without bothering with the intermediate sub-commit state.
359
Savepoints are implemented using subtransactions. A subtransaction is a
360
transaction inside a transaction; its commit or abort status is not only
361
dependent on whether it committed itself, but also whether its parent
362
transaction committed. To implement multiple savepoints in a transaction we
363
allow unlimited transaction nesting depth, so any particular subtransaction's
364
commit state is dependent on the commit status of each and every ancestor
367
The "subtransaction parent" (pg_subtrans) mechanism records, for each
368
transaction with an XID, the TransactionId of its parent transaction. This
369
information is stored as soon as the subtransaction is assigned an XID.
370
Top-level transactions do not have a parent, so they leave their pg_subtrans
371
entries set to the default value of zero (InvalidTransactionId).
373
pg_subtrans is used to check whether the transaction in question is still
374
running --- the main Xid of a transaction is recorded in the PGPROC struct,
375
but since we allow arbitrary nesting of subtransactions, we can't fit all Xids
376
in shared memory, so we have to store them on disk. Note, however, that for
377
each transaction we keep a "cache" of Xids that are known to be part of the
378
transaction tree, so we can skip looking at pg_subtrans unless we know the
379
cache has been overflowed. See storage/ipc/procarray.c for the gory details.
381
slru.c is the supporting mechanism for both pg_clog and pg_subtrans. It
382
implements the LRU policy for in-memory buffer pages. The high-level routines
383
for pg_clog are implemented in transam.c, while the low-level functions are in
384
clog.c. pg_subtrans is contained completely in subtrans.c.
387
Write-Ahead Log Coding
388
----------------------
390
The WAL subsystem (also called XLOG in the code) exists to guarantee crash
391
recovery. It can also be used to provide point-in-time recovery, as well as
392
hot-standby replication via log shipping. Here are some notes about
393
non-obvious aspects of its design.
395
A basic assumption of a write AHEAD log is that log entries must reach stable
396
storage before the data-page changes they describe. This ensures that
397
replaying the log to its end will bring us to a consistent state where there
398
are no partially-performed transactions. To guarantee this, each data page
399
(either heap or index) is marked with the LSN (log sequence number --- in
400
practice, a WAL file location) of the latest XLOG record affecting the page.
401
Before the bufmgr can write out a dirty page, it must ensure that xlog has
402
been flushed to disk at least up to the page's LSN. This low-level
403
interaction improves performance by not waiting for XLOG I/O until necessary.
404
The LSN check exists only in the shared-buffer manager, not in the local
405
buffer manager used for temp tables; hence operations on temp tables must not
408
During WAL replay, we can check the LSN of a page to detect whether the change
409
recorded by the current log entry is already applied (it has been, if the page
410
LSN is >= the log entry's WAL location).
412
Usually, log entries contain just enough information to redo a single
413
incremental update on a page (or small group of pages). This will work only
414
if the filesystem and hardware implement data page writes as atomic actions,
415
so that a page is never left in a corrupt partly-written state. Since that's
416
often an untenable assumption in practice, we log additional information to
417
allow complete reconstruction of modified pages. The first WAL record
418
affecting a given page after a checkpoint is made to contain a copy of the
419
entire page, and we implement replay by restoring that page copy instead of
420
redoing the update. (This is more reliable than the data storage itself would
421
be because we can check the validity of the WAL record's CRC.) We can detect
422
the "first change after checkpoint" by noting whether the page's old LSN
423
precedes the end of WAL as of the last checkpoint (the RedoRecPtr).
425
The general schema for executing a WAL-logged action is
427
1. Pin and exclusive-lock the shared buffer(s) containing the data page(s)
430
2. START_CRIT_SECTION() (Any error during the next three steps must cause a
431
PANIC because the shared buffers will contain unlogged changes, which we
432
have to ensure don't get to disk. Obviously, you should check conditions
433
such as whether there's enough free space on the page before you start the
436
3. Apply the required changes to the shared buffer(s).
438
4. Mark the shared buffer(s) as dirty with MarkBufferDirty(). (This must
439
happen before the WAL record is inserted; see notes in SyncOneBuffer().)
441
5. Build a WAL log record and pass it to XLogInsert(); then update the page's
442
LSN and TLI using the returned XLOG location. For instance,
444
recptr = XLogInsert(rmgr_id, info, rdata);
446
PageSetLSN(dp, recptr);
447
PageSetTLI(dp, ThisTimeLineID);
449
6. END_CRIT_SECTION()
451
7. Unlock and unpin the buffer(s).
453
XLogInsert's "rdata" argument is an array of pointer/size items identifying
454
chunks of data to be written in the XLOG record, plus optional shared-buffer
455
IDs for chunks that are in shared buffers rather than temporary variables.
456
The "rdata" array must mention (at least once) each of the shared buffers
457
being modified, unless the action is such that the WAL replay routine can
458
reconstruct the entire page contents. XLogInsert includes the logic that
459
tests to see whether a shared buffer has been modified since the last
460
checkpoint. If not, the entire page contents are logged rather than just the
461
portion(s) pointed to by "rdata".
463
Because XLogInsert drops the rdata components associated with buffers it
464
chooses to log in full, the WAL replay routines normally need to test to see
465
which buffers were handled that way --- otherwise they may be misled about
466
what the XLOG record actually contains. XLOG records that describe multi-page
467
changes therefore require some care to design: you must be certain that you
468
know what data is indicated by each "BKP" bit. An example of the trickiness
469
is that in a HEAP_UPDATE record, BKP(1) normally is associated with the source
470
page and BKP(2) is associated with the destination page --- but if these are
471
the same page, only BKP(1) would have been set.
473
For this reason as well as the risk of deadlocking on buffer locks, it's best
474
to design WAL records so that they reflect small atomic actions involving just
475
one or a few pages. The current XLOG infrastructure cannot handle WAL records
476
involving references to more than three shared buffers, anyway.
478
In the case where the WAL record contains enough information to re-generate
479
the entire contents of a page, do *not* show that page's buffer ID in the
480
rdata array, even if some of the rdata items point into the buffer. This is
481
because you don't want XLogInsert to log the whole page contents. The
482
standard replay-routine pattern for this case is
484
reln = XLogOpenRelation(rnode);
485
buffer = XLogReadBuffer(reln, blkno, true);
486
Assert(BufferIsValid(buffer));
487
page = (Page) BufferGetPage(buffer);
489
... initialize the page ...
491
PageSetLSN(page, lsn);
492
PageSetTLI(page, ThisTimeLineID);
493
MarkBufferDirty(buffer);
494
UnlockReleaseBuffer(buffer);
496
In the case where the WAL record provides only enough information to
497
incrementally update the page, the rdata array *must* mention the buffer
498
ID at least once; otherwise there is no defense against torn-page problems.
499
The standard replay-routine pattern for this case is
501
if (record->xl_info & XLR_BKP_BLOCK_n)
502
<< do nothing, page was rewritten from logged copy >>;
504
reln = XLogOpenRelation(rnode);
505
buffer = XLogReadBuffer(reln, blkno, false);
506
if (!BufferIsValid(buffer))
507
<< do nothing, page has been deleted >>;
508
page = (Page) BufferGetPage(buffer);
510
if (XLByteLE(lsn, PageGetLSN(page)))
512
/* changes are already applied */
513
UnlockReleaseBuffer(buffer);
517
... apply the change ...
519
PageSetLSN(page, lsn);
520
PageSetTLI(page, ThisTimeLineID);
521
MarkBufferDirty(buffer);
522
UnlockReleaseBuffer(buffer);
524
As noted above, for a multi-page update you need to be able to determine
525
which XLR_BKP_BLOCK_n flag applies to each page. If a WAL record reflects
526
a combination of fully-rewritable and incremental updates, then the rewritable
527
pages don't count for the XLR_BKP_BLOCK_n numbering. (XLR_BKP_BLOCK_n is
528
associated with the n'th distinct buffer ID seen in the "rdata" array, and
529
per the above discussion, fully-rewritable buffers shouldn't be mentioned in
532
Due to all these constraints, complex changes (such as a multilevel index
533
insertion) normally need to be described by a series of atomic-action WAL
534
records. What do you do if the intermediate states are not self-consistent?
535
The answer is that the WAL replay logic has to be able to fix things up.
536
In btree indexes, for example, a page split requires insertion of a new key in
537
the parent btree level, but for locking reasons this has to be reflected by
538
two separate WAL records. The replay code has to remember "unfinished" split
539
operations, and match them up to subsequent insertions in the parent level.
540
If no matching insert has been found by the time the WAL replay ends, the
541
replay code has to do the insertion on its own to restore the index to
542
consistency. Such insertions occur after WAL is operational, so they can
543
and should write WAL records for the additional generated actions.
549
As of PostgreSQL 8.3 it is possible to perform asynchronous commits - i.e.,
550
we don't wait while the WAL record for the commit is fsync'ed.
551
We perform an asynchronous commit when synchronous_commit = off. Instead
552
of performing an XLogFlush() up to the LSN of the commit, we merely note
553
the LSN in shared memory. The backend then continues with other work.
554
We record the LSN only for an asynchronous commit, not an abort; there's
555
never any need to flush an abort record, since the presumption after a
556
crash would be that the transaction aborted anyway.
558
We always force synchronous commit when the transaction is deleting
559
relations, to ensure the commit record is down to disk before the relations
560
are removed from the filesystem. Also, certain utility commands that have
561
non-roll-backable side effects (such as filesystem changes) force sync
562
commit to minimize the window in which the filesystem change has been made
563
but the transaction isn't guaranteed committed.
565
Every wal_writer_delay milliseconds, the walwriter process performs an
566
XLogBackgroundFlush(). This checks the location of the last completely
567
filled WAL page. If that has moved forwards, then we write all the changed
568
buffers up to that point, so that under full load we write only whole
569
buffers. If there has been a break in activity and the current WAL page is
570
the same as before, then we find out the LSN of the most recent
571
asynchronous commit, and flush up to that point, if required (i.e.,
572
if it's in the current WAL page). This arrangement in itself would
573
guarantee that an async commit record reaches disk during at worst the
574
second walwriter cycle after the transaction completes. However, we also
575
allow XLogFlush to flush full buffers "flexibly" (ie, not wrapping around
576
at the end of the circular WAL buffer area), so as to minimize the number
577
of writes issued under high load when multiple WAL pages are filled per
578
walwriter cycle. This makes the worst-case delay three walwriter cycles.
580
There are some other subtle points to consider with asynchronous commits.
581
First, for each page of CLOG we must remember the LSN of the latest commit
582
affecting the page, so that we can enforce the same flush-WAL-before-write
583
rule that we do for ordinary relation pages. Otherwise the record of the
584
commit might reach disk before the WAL record does. Again, abort records
585
need not factor into this consideration.
587
In fact, we store more than one LSN for each clog page. This relates to
588
the way we set transaction status hint bits during visibility tests.
589
We must not set a transaction-committed hint bit on a relation page and
590
have that record make it to disk prior to the WAL record of the commit.
591
Since visibility tests are normally made while holding buffer share locks,
592
we do not have the option of changing the page's LSN to guarantee WAL
593
synchronization. Instead, we defer the setting of the hint bit if we have
594
not yet flushed WAL as far as the LSN associated with the transaction.
595
This requires tracking the LSN of each unflushed async commit. It is
596
convenient to associate this data with clog buffers: because we will flush
597
WAL before writing a clog page, we know that we do not need to remember a
598
transaction's LSN longer than the clog page holding its commit status
599
remains in memory. However, the naive approach of storing an LSN for each
600
clog position is unattractive: the LSNs are 32x bigger than the two-bit
601
commit status fields, and so we'd need 256K of additional shared memory for
602
each 8K clog buffer page. We choose instead to store a smaller number of
603
LSNs per page, where each LSN is the highest LSN associated with any
604
transaction commit in a contiguous range of transaction IDs on that page.
605
This saves storage at the price of some possibly-unnecessary delay in
606
setting transaction hint bits.
608
How many transactions should share the same cached LSN (N)? If the
609
system's workload consists only of small async-commit transactions, then
610
it's reasonable to have N similar to the number of transactions per
611
walwriter cycle, since that is the granularity with which transactions will
612
become truly committed (and thus hintable) anyway. The worst case is where
613
a sync-commit xact shares a cached LSN with an async-commit xact that
614
commits a bit later; even though we paid to sync the first xact to disk,
615
we won't be able to hint its outputs until the second xact is sync'd, up to
616
three walwriter cycles later. This argues for keeping N (the group size)
617
as small as possible. For the moment we are setting the group size to 32,
618
which makes the LSN cache space the same size as the actual clog buffer
619
space (independently of BLCKSZ).
621
It is useful that we can run both synchronous and asynchronous commit
622
transactions concurrently, but the safety of this is perhaps not
623
immediately obvious. Assume we have two transactions, T1 and T2. The Log
624
Sequence Number (LSN) is the point in the WAL sequence where a transaction
625
commit is recorded, so LSN1 and LSN2 are the commit records of those
626
transactions. If T2 can see changes made by T1 then when T2 commits it
627
must be true that LSN2 follows LSN1. Thus when T2 commits it is certain
628
that all of the changes made by T1 are also now recorded in the WAL. This
629
is true whether T1 was asynchronous or synchronous. As a result, it is
630
safe for asynchronous commits and synchronous commits to work concurrently
631
without endangering data written by synchronous commits. Sub-transactions
632
are not important here since the final write to disk only occurs at the
633
commit of the top level transaction.
635
Changes to data blocks cannot reach disk unless WAL is flushed up to the
636
point of the LSN of the data blocks. Any attempt to write unsafe data to
637
disk will trigger a write which ensures the safety of all data written by
638
that and prior transactions. Data blocks and clog pages are both protected
641
Changes to a temp table are not WAL-logged, hence could reach disk in
642
advance of T1's commit, but we don't care since temp table contents don't
643
survive crashes anyway.
645
Database writes made via any of the paths we have introduced to avoid WAL
646
overhead for bulk updates are also safe. In these cases it's entirely
647
possible for the data to reach disk before T1's commit, because T1 will
648
fsync it down to disk without any sort of interlock, as soon as it finishes
649
the bulk update. However, all these paths are designed to write data that
650
no other transaction can see until after T1 commits. The situation is thus
651
not different from ordinary WAL-logged updates.
added
removed
src/backend/access/transam/README
