~ubuntu-branches/ubuntu/wily/sqlite3/wily

onfocus="entersearch()" onblur="leavesearch()" style="width:24ex;padding:1px 1ex; border:solid white 1px; font-size:0.9em ; font-style:italic;color:#044a64;" value="Search SQLite Docs...">

112

113

</form>

114

</div>

115

</table>

116

</div></div></div></div>

117

</td></tr></table>

118

119

120

121

122

123

124

The SQLite OS Interface or "VFS"

125

</h1>

126

127

<p>

128

This article describes the SQLite OS portability layer or "VFS" - the

129

module at the bottom of the SQLite implementation stack

130

that provides portability across operating systems.

131

</p>

132

133

134

<h2>1.0 The VFS In Relation To The Rest Of SQLite</h2>

135

136

<p>

137

The internal organization of the SQLite library can be viewed as the

138

stack of modules shown to the right.

139

The Tokenizer, Parser, and Code Generator components are used to

140

process SQL statements and convert them into executable programs

141

in a virtual machine language or byte code.

142

Roughly speaking, these top three layers implement

143

<a href="c3ref/prepare.html">sqlite3_prepare_v2()</a>. The byte code generated by the top three

144

layers is a <a href="c3ref/stmt.html">prepared statement</a>.

145

The Virtual Machine module is responsible for running the SQL statement

146

byte code. The B-Tree module organizes a database file into multiple

147

key/value stores with ordered keys and logarithmic performance.

148

The Pager module is responsible for loading pages of the database

149

file into memory, for implementing and controlling transactions, and

150

for creating and maintaining the journal files that prevent database

151

corruption following a crash or power failure.

152

The OS Interface is a thin abstraction that provides a common set of

153

routines for adapting SQLite to run on different operating systems.

154

Roughly speaking, the bottom four layers implement

155

<a href="c3ref/step.html">sqlite3_step()</a>.

156

</p>

157

158

<p>

159

This article is about the bottom layer.

160

</p>

161

162

<p>The OS Interface - also called the "VFS" - is what makes SQLite

163

portable across operating systems. Whenever any of the other modules

164

in SQLite needs to communicate with the operating

165

system, they invoke methods in the VFS. The VFS then invokes the

166

operating-specific code needed to satisfy the request.

167

Hence, porting SQLite to a new

168

operating system is simply a matter of writing a new OS interface layer

169

or "VFS".</p>

170

171

<h2>2.0 Multiple VFSes</h2>

172

173

<p>

174

The standard SQLite source tree contains built-in VFSes for os/2, unix,

175

and windows. Alternative VFSes can be

176

added at start-time or run-time using the

177

<a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a> interface.

178

</p>

179

180

<p>

181

Multiple VFSes can be registered at the same time.

182

Each VFS has a unique names.

183

Separate <a href="c3ref/sqlite3.html">database connections</a> within the same process can be using

184

different VFSes at the same time. For that matter, if a single

185

database connection has multiple database files open using

186

the <a href="lang_attach.html">ATTACH</a> command, then each attached database might be using a

187

different VFS.

188

</p>

189

190

<p>

191

Unix builds come with multiple VFSes built-in. The default VFS

192

for unix is called "unix" and is the VFS used in an overwhelming

193

majority of applications. Other VFSes that can be found in unix

194

may include:

195

</p>

196

197

<ol>

198

<li><p><b>unix-dotfile</b> - uses dot-file locking rather than

199

POSIX advisory locks.

200

<li><p><b>unix-excl</b> - obtains and holds an exclusive lock on

201

database files, preventing other processes from accessing the

202

database. Also keeps the <a href="fileformat2.html#walindexformat">wal-index</a> in heap rather than in

203

shared memory.

204

<li><p><b>unix-none</b> - all file locking operations are no-ops.

205

<li><p><b>unix-namedsem</b> - uses named semaphores for file locking.

206

VXWorks only.

207

</ol>

208

209

<p>

210

The various unix VFSes differ only in the way they handle file locking -

211

they share most of their implementation in common with one another and

212

are all located in the same SQLite source file:

213

<a href="http://www.sqlite.org/src/doc/trunk/src/os_unix.c">os_unix.c</a>.

214

Note that except for "unix" and "unix-excl", the various unix VFSes all

215

use incompatible locking implementations. If two processes are accessing

216

the same SQLite database using different unix VFSes, they may

217

not see each others locks and may end up interfering with one another,

218

resulting in database corruption. The "unix-none" VFS in particular

219

does no locking at all and will easily result in database corruption if

220

used by two or more database connections at the same time.

221

Programmers are encouraged to use only "unix" or "unix-excl" unless

222

there is a compelling reason to do otherwise.

223

</p>

224

225

<h2>2.1 Specifying Which VFS To Use</h2>

226

227

<p>

228

There is always one VFS which is the default VFS. On unix systems,

229

the "unix" VFS comes up as the default and on windows it is "win32".

230

If no other actions are taken, new database connections will make use

231

of the default VFS.

232

</p>

233

234

<p>

235

The default VFS can be changed by registering or re-registering the

236

VFS using the <a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a> interface with a second parameter

237

of 1. Hence, if a (unix) process to always use the "unix-nolock" VFS

238

in place of "unix", the following code would work:

239

</p>

240

241

242

sqlite3_vfs_register(sqlite3_vfs_find("unix-nolock"), 1);

243

</pre></blockquote>

244

245

<p>

246

An alternate VFS can also be specified as the 4th parameter to the

247

<a href="c3ref/open.html">sqlite3_open_v2()</a> function. For example:

248

</p>

249

250

251

int rc = sqlite3_open_v2("demo.db", &db, SQLITE_OPEN_READWRITE, "unix-nolock");

252

</pre></blockquote>

253

254

<p>

255

Finally, if <a href="uri.html">URI filenames</a> have been enabled, then the alternative

256

VFS can be specified using the "vfs=" parameter on the URI. This technique

257

works with <a href="c3ref/open.html">sqlite3_open()</a>, <a href="c3ref/open.html">sqlite3_open16()</a>, <a href="c3ref/open.html">sqlite3_open_v2()</a>, and

258

when a new database is <a href="lang_attach.html">ATTACH</a>-ed to an existing database connection.

259

For example:

260

</p>

261

262

263

ATTACH 'file:demo2.db?vfs=unix-none' AS demo2;

264

</pre></blockquote>

265

266

<p>

267

The VFS specified by a URI has the highest priority. After that comes

268

a VFS specified as the fourth argument to <a href="c3ref/open.html">sqlite3_open_v2()</a>. The

269

default VFS is used if no VFS is specified otherwise.

270

</p>

271

272

<h2>2.2 VFS Shims</h2>

273

274

<p>

275

From the point of view of the uppers layers of the SQLite stack, each

276

open database file uses exactly one VFS.

277

But in practice, a particular VFS might

278

just be a thin wrapper around another VFS that does the real work.

279

We call a wrapper VFS a "shim".

280

</p>

281

282

<p>

283

A simple example of a shim is the "vfstrace" VFS. This is a VFS

284

(implemented in the

285

<a href="http://www.sqlite.org/src/doc/trunk/src/test_vfstrace.c">test_vfstrace.c</a>

286

source file) that writes a message associated with each VFS method call

287

into a log file, then passes control off to another VFS to do the actual

288

work.

289

</p>

290

291

<h2>2.3 Other Example VFSes</h2>

292

293

<p>

294

The following are other VFS implementations available in the public

295

SQLite source tree:

296

</p>

297

298

<ul>

299

<li><p>

300

<a href="http://www.sqlite.org/src/doc/trunk/src/test_demovfs.c">test_demovfs.c</a> -

301

This file implements a very simple VFS named "demo" that uss POSIX

302

functions such as

303

open(), read(), write(), fsync(), close(), fsync(), sleep(), time(),

304

and so forth. This VFS only works on unix systems. But it is not

305

intended as a replacement for the standard "unix" VFS used by default

306

on unix platforms. The "demo" VFS is deliberately kept very simple

307

so that it can be used as a learning aid or as template for building

308

other VFSes or for porting SQLite to new operating systems.

309

310

<li><p>

311

<a href="http://www.sqlite.org/src/doc/trunk/src/test_quota.c">test_quota.c</a> -

312

This file implements a shim called "quota" that enforces cumulative

313

file size limits on a collection of database files. An auxiliary

314

interface is used to define "quote groups". A quota group is a

315

set of files (database files, journals, and temporary files) whose

316

names all match a <a href="lang_expr.html#glob">GLOB</a> pattern. The sum of the sizes of all files

317

in each quota group is tracked, and if that sum exceeds a threshold

318

defined for the quota group, a callback function is invoked. That

319

callback can either increase the threshold or cause the operation

320

that would have exceeded the quota to fail with an

321

<a href="c3ref/c_abort.html">SQLITE_FULL</a> error. One of the uses of this shim is used to enforce

322

resource limits on application databases in Firefox.

323

324

<li><p>

325

<a href="http://www.sqlite.org/src/doc/trunk/src/test_multiplex.c">test_multiplex.c</a> -

326

This file implements a shim that allows database files to exceed the

327

maximum file size of the underlying filesystem. This shim presents

328

an interface to the upper six layers of SQLite that makes it look like

329

very large files are being used, when in reality each such large file

330

is split up into many smaller files on the underlying system.

331

This shim has been used, for example, to allow databases to grow

332

larger than 2 gibibytes on FAT16 filesystems.

333

334

<li><p>

335

<a href="http://www.sqlite.org/src/doc/trunk/src/test_onefile.c">test_onefile.c</a> -

336

This file implements a demonstration VFS named "fs" that shows how SQLite

337

can be used on an embedded device that lacks a filesystem. Content is

338

written directly to the underlying media. A VFS derived from this

339

demonstration code could be used by a gadget with a limited amount of

340

flash memory to make SQLite behave as the filesystem for the flash memory

341

on the device.

342

343

<li><p>

344

<a href="http://www.sqlite.org/src/doc/trunk/src/test_journal.c">test_journal.c</a> -

345

This file implements a shim used during SQLite testing that verifies that

346

the database and rollback journal are written in the correct order and

347

are "synced" at appropriate times in order to guarantee that the database

348

can recover from a power lose are hard reset at any time. The shim

349

checks several invariants on the operation of databases and rollback

350

journals and raises exceptions if any of those invariants are violated.

351

These invariants, in turn, assure that the database is always recoverable.

352

Running a large suite of test cases using this shim provides added

353

assurance that SQLite databases will not be damaged by unexpected

354

power failures or device resets.

355

356

<li><p>

357

<a href="http://www.sqlite.org/src/doc/trunk/src/test_vfs.c">test_vfs.c</a> -

358

This file implements a shim that can be used to simulate filesystem faults.

359

This shim is used during testing to verify that SQLite responses sanely

360

to hardware malfunctions or to other error conditions such as running out

361

of filesystem space that are difficult to test on a real system.

362

</ul>

363

364

<p>

365

There are other VFS implementations both in the core SQLite source code

366

library and in available extensions. The list above is not meant to be

367

exhaustive but merely representative of the kinds of features that can

368

be realized using the VFS interface.

369

</p>

370

371

<h2>3.0 VFS Implementations</h2>

372

373

<p>

374

A new VFS is implemented by subclassing three objects:

375

</p>

376

377

<ul>

378

<li><a href="c3ref/vfs.html">sqlite3_vfs</a>

379

<li><a href="c3ref/io_methods.html">sqlite3_io_methods</a>

380

<li><a href="c3ref/file.html">sqlite3_file</a>

381

</ul>

382

383

<p>

384

An <a href="c3ref/vfs.html">sqlite3_vfs</a> object defines the name of the VFS and the core

385

methods that implement the interface to the operating system, such

386

as checking for existence of files, deleting files, creating files

387

and opening and for reading and/or writing, converting filenames

388

into their canonical form. The <a href="c3ref/vfs.html">sqlite3_vfs</a> object also contains

389

methods for obtaining randomness from the operating system, for

390

suspending a process (sleeping) and for finding the current date and

391

time.

392

</p>

393

394

<p>

395

The <a href="c3ref/file.html">sqlite3_file</a> object represents an open file.

396

The xOpen method of <a href="c3ref/vfs.html">sqlite3_vfs</a> constructs an <a href="c3ref/file.html">sqlite3_file</a>

397

object when the file is opened. The <a href="c3ref/file.html">sqlite3_file</a> keeps track

398

of the state of the file while it is opened.

399

</p>

400

401

<p>

402

The <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object holds the methods used to interact

403

with an open file. Each <a href="c3ref/file.html">sqlite3_file</a> contains a pointer to

404

an <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object that is appropriate for the file it

405

represents. The <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object contains methods to do

406

things such as read and write from the file, to truncate the file,

407

to flush any changes to persistent storage, to find the size of the

408

file, to lock and unlock the file, and to close file and destroy

409

the <a href="c3ref/file.html">sqlite3_file</a> object.

410

</p>

411

412

<p>

413

Writing the code for a new VFS involves constructing a subclass for

414

the <a href="c3ref/vfs.html">sqlite3_vfs</a> object and then registering that VFS object using

415

a call to <a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a>. The VFS implementation also

416

provides subclasses for <a href="c3ref/file.html">sqlite3_file</a> and <a href="c3ref/io_methods.html">sqlite3_io_methods</a> but

417

those objects are not registered directly with SQLite. Instead, the

418

<a href="c3ref/file.html">sqlite3_file</a> object is returned from the xOpen method of

419

<a href="c3ref/vfs.html">sqlite3_vfs</a> and the <a href="c3ref/file.html">sqlite3_file</a> object points to an instance

420

of the <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object.

421

</p>

422

423

<h2>To Be Continued...</h2>

424