# page_content

c2: Result and Error Codes
c5: rescode.html
c4:
Overview 


Many of the routines in the SQLite C-language Interface return
numeric result codes indicating either success or failure, and 
in the event of a failure, providing some idea of the cause of
the failure.  This document strives to explain what each
of those numeric result codes means.

1. Result Codes versus Error Codes


"Error codes" are a subset of "result codes" that indicate that
something has gone wrong.  There are only a few non-error result
codes:  SQLITE_OK, SQLITE_ROW, and SQLITE_DONE.  The term
"error code" means any result code other than these three.



2. Primary Result Codes versus Extended Result Codes


Result codes are signed 32-bit integers.
The least significant 8 bits of the result code define a broad category
and are called the "primary result code".  More significant bits provide
more detailed information about the error and are called the
"extended result code"


Note that the primary result code is always a part of the extended
result code.  Given a full 32-bit extended result code, the application
can always find the corresponding primary result code merely by extracting
the least significant 8 bits of the extended result code.


All extended result codes are also error codes.  Hence the terms
"extended result code" and "extended error code" are interchangeable.


For historic compatibility, the C-language interfaces return
primary result codes by default.  
The extended result code for the most recent error can be
retrieved using the sqlite3_extended_errcode() interface.
The sqlite3_extended_result_codes() interface can be used to put
a database connection into a mode where it returns the
extended result codes instead of the primary result codes.

3. Definitions


All result codes are integers.
Symbolic names for all result codes are created using
"#define" macros in the sqlite3.h header file.
There are separate sections in the sqlite3.h header file for
the result code definitions and the extended result code definitions.


Primary result code symbolic names are of the form "SQLITE_XXXXXX" where
XXXXXX is a sequence of uppercase alphabetic characters.  Extended
result code names are of the form "SQLITE_XXXXXX_YYYYYYY" where
the XXXXXX part is the corresponding primary result code and the
YYYYYYY is an extension that further classifies the result code.


The names and numeric values for existing result codes are fixed
and unchanging.  However, new result codes, and especially new extended
result codes, might appear in future releases of SQLite.


4. Primary Result Code List
The 31 result codes are
   defined in sqlite3.h and are listed in
   alphabetical order below:


SQLITE_ABORT&nbsp;(4)
SQLITE_AUTH&nbsp;(23)
SQLITE_BUSY&nbsp;(5)
SQLITE_CANTOPEN&nbsp;(14)
SQLITE_CONSTRAINT&nbsp;(19)
SQLITE_CORRUPT&nbsp;(11)
SQLITE_DONE&nbsp;(101)
SQLITE_EMPTY&nbsp;(16)
SQLITE_ERROR&nbsp;(1)
SQLITE_FORMAT&nbsp;(24)
SQLITE_FULL&nbsp;(13)
SQLITE_INTERNAL&nbsp;(2)
SQLITE_INTERRUPT&nbsp;(9)
SQLITE_IOERR&nbsp;(10)
SQLITE_LOCKED&nbsp;(6)
SQLITE_MISMATCH&nbsp;(20)
SQLITE_MISUSE&nbsp;(21)
SQLITE_NOLFS&nbsp;(22)
SQLITE_NOMEM&nbsp;(7)
SQLITE_NOTADB&nbsp;(26)
SQLITE_NOTFOUND&nbsp;(12)
SQLITE_NOTICE&nbsp;(27)
SQLITE_OK&nbsp;(0)
SQLITE_PERM&nbsp;(3)
SQLITE_PROTOCOL&nbsp;(15)
SQLITE_RANGE&nbsp;(25)
SQLITE_READONLY&nbsp;(8)
SQLITE_ROW&nbsp;(100)
SQLITE_SCHEMA&nbsp;(17)
SQLITE_TOOBIG&nbsp;(18)
SQLITE_WARNING&nbsp;(28)




5. Extended Result Code List
The 74 extended result codes
  are defined in sqlite3.h and are
  listed in alphabetical order below:


SQLITE_ABORT_ROLLBACK&nbsp;(516)
SQLITE_AUTH_USER&nbsp;(279)
SQLITE_BUSY_RECOVERY&nbsp;(261)
SQLITE_BUSY_SNAPSHOT&nbsp;(517)
SQLITE_BUSY_TIMEOUT&nbsp;(773)
SQLITE_CANTOPEN_CONVPATH&nbsp;(1038)
SQLITE_CANTOPEN_DIRTYWAL&nbsp;(1294)
SQLITE_CANTOPEN_FULLPATH&nbsp;(782)
SQLITE_CANTOPEN_ISDIR&nbsp;(526)
SQLITE_CANTOPEN_NOTEMPDIR&nbsp;(270)
SQLITE_CANTOPEN_SYMLINK&nbsp;(1550)
SQLITE_CONSTRAINT_CHECK&nbsp;(275)
SQLITE_CONSTRAINT_COMMITHOOK&nbsp;(531)
SQLITE_CONSTRAINT_DATATYPE&nbsp;(3091)
SQLITE_CONSTRAINT_FOREIGNKEY&nbsp;(787)
SQLITE_CONSTRAINT_FUNCTION&nbsp;(1043)
SQLITE_CONSTRAINT_NOTNULL&nbsp;(1299)
SQLITE_CONSTRAINT_PINNED&nbsp;(2835)
SQLITE_CONSTRAINT_PRIMARYKEY&nbsp;(1555)
SQLITE_CONSTRAINT_ROWID&nbsp;(2579)
SQLITE_CONSTRAINT_TRIGGER&nbsp;(1811)
SQLITE_CONSTRAINT_UNIQUE&nbsp;(2067)
SQLITE_CONSTRAINT_VTAB&nbsp;(2323)
SQLITE_CORRUPT_INDEX&nbsp;(779)
SQLITE_CORRUPT_SEQUENCE&nbsp;(523)
SQLITE_CORRUPT_VTAB&nbsp;(267)
SQLITE_ERROR_MISSING_COLLSEQ&nbsp;(257)
SQLITE_ERROR_RETRY&nbsp;(513)
SQLITE_ERROR_SNAPSHOT&nbsp;(769)
SQLITE_IOERR_ACCESS&nbsp;(3338)
SQLITE_IOERR_AUTH&nbsp;(7178)
SQLITE_IOERR_BEGIN_ATOMIC&nbsp;(7434)
SQLITE_IOERR_BLOCKED&nbsp;(2826)
SQLITE_IOERR_CHECKRESERVEDLOCK&nbsp;(3594)
SQLITE_IOERR_CLOSE&nbsp;(4106)
SQLITE_IOERR_COMMIT_ATOMIC&nbsp;(7690)
SQLITE_IOERR_CONVPATH&nbsp;(6666)
SQLITE_IOERR_CORRUPTFS&nbsp;(8458)
SQLITE_IOERR_DATA&nbsp;(8202)
SQLITE_IOERR_DELETE&nbsp;(2570)
SQLITE_IOERR_DELETE_NOENT&nbsp;(5898)
SQLITE_IOERR_DIR_CLOSE&nbsp;(4362)
SQLITE_IOERR_DIR_FSYNC&nbsp;(1290)
SQLITE_IOERR_FSTAT&nbsp;(1802)
SQLITE_IOERR_FSYNC&nbsp;(1034)
SQLITE_IOERR_GETTEMPPATH&nbsp;(6410)
SQLITE_IOERR_LOCK&nbsp;(3850)
SQLITE_IOERR_MMAP&nbsp;(6154)
SQLITE_IOERR_NOMEM&nbsp;(3082)
SQLITE_IOERR_RDLOCK&nbsp;(2314)
SQLITE_IOERR_READ&nbsp;(266)
SQLITE_IOERR_ROLLBACK_ATOMIC&nbsp;(7946)
SQLITE_IOERR_SEEK&nbsp;(5642)
SQLITE_IOERR_SHMLOCK&nbsp;(5130)
SQLITE_IOERR_SHMMAP&nbsp;(5386)
SQLITE_IOERR_SHMOPEN&nbsp;(4618)
SQLITE_IOERR_SHMSIZE&nbsp;(4874)
SQLITE_IOERR_SHORT_READ&nbsp;(522)
SQLITE_IOERR_TRUNCATE&nbsp;(1546)
SQLITE_IOERR_UNLOCK&nbsp;(2058)
SQLITE_IOERR_VNODE&nbsp;(6922)
SQLITE_IOERR_WRITE&nbsp;(778)
SQLITE_LOCKED_SHAREDCACHE&nbsp;(262)
SQLITE_LOCKED_VTAB&nbsp;(518)
SQLITE_NOTICE_RECOVER_ROLLBACK&nbsp;(539)
SQLITE_NOTICE_RECOVER_WAL&nbsp;(283)
SQLITE_OK_LOAD_PERMANENTLY&nbsp;(256)
SQLITE_READONLY_CANTINIT&nbsp;(1288)
SQLITE_READONLY_CANTLOCK&nbsp;(520)
SQLITE_READONLY_DBMOVED&nbsp;(1032)
SQLITE_READONLY_DIRECTORY&nbsp;(1544)
SQLITE_READONLY_RECOVERY&nbsp;(264)
SQLITE_READONLY_ROLLBACK&nbsp;(776)
SQLITE_WARNING_AUTOINDEX&nbsp;(284)



6. Result Code Meanings

The meanings for all 105
result code values are shown below,
in numeric order.



(0) SQLITE_OK

  The SQLITE_OK result code means that the operation was successful and
  that there were no errors.  Most other result codes indicate an error.




(1) SQLITE_ERROR

  The SQLITE_ERROR result code is a generic error code that is used when
  no other more specific error code is available.




(2) SQLITE_INTERNAL

  The SQLITE_INTERNAL result code indicates an internal malfunction.
  In a working version of SQLite, an application should never see this
  result code.  If application does encounter this result code, it shows
  that there is a bug in the database engine.
  
  SQLite does not currently generate this result code.
  However, application-defined SQL functions or
  virtual tables, or VFSes, or other extensions might cause this 
  result code to be returned.




(3) SQLITE_PERM

  The SQLITE_PERM result code indicates that the requested access mode
  for a newly created database could not be provided.




(4) SQLITE_ABORT

  The SQLITE_ABORT result code indicates that an operation was aborted
  prior to completion, usually be application request.
  See also: SQLITE_INTERRUPT.
  
  If the callback function to sqlite3_exec() returns non-zero, then
  sqlite3_exec() will return SQLITE_ABORT.
  
  If a ROLLBACK operation occurs on the same database connection as
  a pending read or write, then the pending read or write may fail with
  an SQLITE_ABORT or SQLITE_ABORT_ROLLBACK error.
  
  In addition to being a result code,
  the SQLITE_ABORT value is also used as a conflict resolution mode
  returned from the sqlite3_vtab_on_conflict() interface.




(5) SQLITE_BUSY

  The SQLITE_BUSY result code indicates that the database file could not
  be written (or in some cases read) because of concurrent activity by 
  some other database connection, usually a database connection in a
  separate process.
  
  For example, if process A is in the middle of a large write transaction
  and at the same time process B attempts to start a new write transaction,
  process B will get back an SQLITE_BUSY result because SQLite only supports
  one writer at a time.  Process B will need to wait for process A to finish
  its transaction before starting a new transaction.  The
  sqlite3_busy_timeout() and sqlite3_busy_handler() interfaces and
  the busy_timeout pragma are available to process B to help it deal
  with SQLITE_BUSY errors.
  
  An SQLITE_BUSY error can occur at any point in a transaction: when the
  transaction is first started, during any write or update operations, or
  when the transaction commits.
  To avoid encountering SQLITE_BUSY errors in the middle of a transaction,
  the application can use BEGIN IMMEDIATE instead of just BEGIN to
  start a transaction.  The BEGIN IMMEDIATE command might itself return
  SQLITE_BUSY, but if it succeeds, then SQLite guarantees that no 
  subsequent operations on the same database through the next COMMIT 
  will return SQLITE_BUSY.
  
  See also:  SQLITE_BUSY_RECOVERY and SQLITE_BUSY_SNAPSHOT.
  
  The SQLITE_BUSY result code differs from SQLITE_LOCKED in that
  SQLITE_BUSY indicates a conflict with a
  separate database connection, probably in a separate process,
  whereas SQLITE_LOCKED 
  indicates a conflict within the same database connection (or sometimes
  a database connection with a shared cache).




(6) SQLITE_LOCKED

  The SQLITE_LOCKED result code indicates that a write operation could not
  continue because of a conflict within the same database connection or
  a conflict with a different database connection that uses a shared cache.
  
  For example, a DROP TABLE statement cannot be run while another thread
  is reading from that table on the same database connection because 
  dropping the table would delete the table out from under the concurrent
  reader.
  
  The SQLITE_LOCKED result code differs from SQLITE_BUSY in that
  SQLITE_LOCKED indicates a conflict on the same database connection
  (or on a connection with a shared cache) whereas SQLITE_BUSY indicates
  a conflict with a different database connection, probably in a different
  process.




(7) SQLITE_NOMEM

  The SQLITE_NOMEM result code indicates that SQLite was unable to allocate
  all the memory it needed to complete the operation.  In other words, an
  internal call to sqlite3_malloc() or sqlite3_realloc() has failed in
  a case where the memory being allocated was required in order to continue
  the operation.




(8) SQLITE_READONLY

  The SQLITE_READONLY result code is returned when an attempt is made to 
  alter some data for which the current database connection does not have
  write permission.




(9) SQLITE_INTERRUPT

  The SQLITE_INTERRUPT result code indicates that an operation was
  interrupted by the sqlite3_interrupt() interface.
  See also: SQLITE_ABORT




(10) SQLITE_IOERR

  The SQLITE_IOERR result code says that the operation could not finish
  because the operating system reported an I/O error.
  
  A full disk drive will normally give an SQLITE_FULL error rather than
  an SQLITE_IOERR error.
  
  There are many different extended result codes for I/O errors that
  identify the specific I/O operation that failed.




(11) SQLITE_CORRUPT

  The SQLITE_CORRUPT result code indicates that the database file has
  been corrupted.  See the How To Corrupt Your Database Files for
  further discussion on how corruption can occur.




(12) SQLITE_NOTFOUND

  The SQLITE_NOTFOUND result code is exposed in three ways:
  
  
  SQLITE_NOTFOUND can be returned by the sqlite3_file_control() interface
  to indicate that the file control opcode passed as the third argument
  was not recognized by the underlying VFS.
  
  SQLITE_NOTFOUND can also be returned by the xSetSystemCall() method of
  an sqlite3_vfs object.
  
  SQLITE_NOTFOUND can be returned by sqlite3_vtab_rhs_value() to indicate
  that the right-hand operand of a constraint is not available to the
  xBestIndex method that made the call.
  
  
  The SQLITE_NOTFOUND result code is also used
  internally by the SQLite implementation, but those internal uses are
  not exposed to the application.




(13) SQLITE_FULL

  The SQLITE_FULL result code indicates that a write could not complete
  because the disk is full.  Note that this error can occur when trying
  to write information into the main database file, or it can also
  occur when writing into temporary disk files.
  
  Sometimes applications encounter this error even though there is an
  abundance of primary disk space because the error occurs when writing
  into temporary disk files on a system where temporary files are stored
  on a separate partition with much less space that the primary disk.




(14) SQLITE_CANTOPEN

  The SQLITE_CANTOPEN result code indicates that SQLite was unable to
  open a file.  The file in question might be a primary database file
  or one of several temporary disk files.




(15) SQLITE_PROTOCOL

  The SQLITE_PROTOCOL result code indicates a problem with the file locking
  protocol used by SQLite.  The SQLITE_PROTOCOL error is currently only
  returned when using WAL mode and attempting to start a new transaction.
  There is a race condition that can occur when two separate 
  database connections both try to start a transaction at the same time
  in WAL mode.  The loser of the race backs off and tries again, after
  a brief delay.  If the same connection loses the locking race dozens
  of times over a span of multiple seconds, it will eventually give up and
  return SQLITE_PROTOCOL.  The SQLITE_PROTOCOL error should appear in practice
  very, very rarely, and only when there are many separate processes all
  competing intensely to write to the same database.




(16) SQLITE_EMPTY

  The SQLITE_EMPTY result code is not currently used.




(17) SQLITE_SCHEMA

  The SQLITE_SCHEMA result code indicates that the database schema
  has changed.  This result code can be returned from sqlite3_step() for
  a prepared statement that was generated using sqlite3_prepare() or
  sqlite3_prepare16().  If the database schema was changed by some other
  process in between the time that the statement was prepared and the time
  the statement was run, this error can result.
  
  If a prepared statement is generated from sqlite3_prepare_v2() then
  the statement is automatically re-prepared if the schema changes, up to
  SQLITE_MAX_SCHEMA_RETRY times (default: 50).  The sqlite3_step()
  interface will only return SQLITE_SCHEMA back to the application if 
  the failure persists after these many retries.




(18) SQLITE_TOOBIG

  The SQLITE_TOOBIG error code indicates that a string or BLOB was
  too large.  The default maximum length of a string or BLOB in SQLite is
  1,000,000,000 bytes.  This maximum length can be changed at compile-time
  using the SQLITE_MAX_LENGTH compile-time option, or at run-time using
  the sqlite3_limit(db,SQLITE_LIMIT_LENGTH,...) interface.  The
  SQLITE_TOOBIG error results when SQLite encounters a string or BLOB
  that exceeds the compile-time or run-time limit.
  
  The SQLITE_TOOBIG error code can also result when an oversized SQL
  statement is passed into one of the sqlite3_prepare_v2() interfaces.
  The maximum length of an SQL statement defaults to a much smaller
  value of 1,000,000,000 bytes.  The maximum SQL statement length can be
  set at compile-time using SQLITE_MAX_SQL_LENGTH or at run-time
  using sqlite3_limit(db,SQLITE_LIMIT_SQL_LENGTH,...).




(19) SQLITE_CONSTRAINT

  The SQLITE_CONSTRAINT error code means that an SQL constraint violation
  occurred while trying to process an SQL statement.  Additional information
  about the failed constraint can be found by consulting the
  accompanying error message (returned via sqlite3_errmsg() or
  sqlite3_errmsg16()) or by looking at the extended error code.
  
  The SQLITE_CONSTRAINT code can also be used as the return value from
  the xBestIndex() method of a virtual table implementation.  When
  xBestIndex() returns SQLITE_CONSTRAINT, that indicates that the particular
  combination of inputs submitted to xBestIndex() cannot result in a
  usable query plan and should not be given further consideration.




(20) SQLITE_MISMATCH

  The SQLITE_MISMATCH error code indicates a datatype mismatch.
  
  SQLite is normally very forgiving about mismatches between the type of
  a value and the declared type of the container in which that value is
  to be stored.  For example, SQLite allows the application to store
  a large BLOB in a column with a declared type of BOOLEAN.  But in a few
  cases, SQLite is strict about types.  The SQLITE_MISMATCH error is
  returned in those few cases when the types do not match.
  
  The rowid of a table must be an integer.  Attempt to set the rowid
  to anything other than an integer (or a NULL which will be automatically
  converted into the next available integer rowid) results in an
  SQLITE_MISMATCH error.




(21) SQLITE_MISUSE

  The SQLITE_MISUSE return code might be returned if the application uses
  any SQLite interface in a way that is undefined or unsupported.  For
  example, using a prepared statement after that prepared statement has
  been finalized might result in an SQLITE_MISUSE error.
  
  SQLite tries to detect misuse and report the misuse using this result code.
  However, there is no guarantee that the detection of misuse will be
  successful.  Misuse detection is probabilistic.  Applications should
  never depend on an SQLITE_MISUSE return value.
  
  If SQLite ever returns SQLITE_MISUSE from any interface, that means that
  the application is incorrectly coded and needs to be fixed.  Do not ship
  an application that sometimes returns SQLITE_MISUSE from a standard
  SQLite interface because that application contains potentially serious bugs.




(22) SQLITE_NOLFS

  The SQLITE_NOLFS error can be returned on systems that do not support
  large files when the database grows to be larger than what the filesystem
  can handle.  "NOLFS" stands for "NO Large File Support".




(23) SQLITE_AUTH

  The SQLITE_AUTH error is returned when the
  authorizer callback indicates that an
  SQL statement being prepared is not authorized.




(24) SQLITE_FORMAT

  The SQLITE_FORMAT error code is not currently used by SQLite.




(25) SQLITE_RANGE

  The SQLITE_RANGE error indices that the parameter number argument
  to one of the sqlite3_bind routines or the
  column number in one of the sqlite3_column
  routines is out of range.




(26) SQLITE_NOTADB

  When attempting to open a file, the SQLITE_NOTADB error indicates that
  the file being opened does not appear to be an SQLite database file.




(27) SQLITE_NOTICE

  The SQLITE_NOTICE result code is not returned by any C/C++ interface.
  However, SQLITE_NOTICE (or rather one of its extended error codes)
  is sometimes used as the first argument in an sqlite3_log() callback
  to indicate that an unusual operation is taking place.




(28) SQLITE_WARNING

  The SQLITE_WARNING result code is not returned by any C/C++ interface.
  However, SQLITE_WARNING (or rather one of its extended error codes)
  is sometimes used as the first argument in an sqlite3_log() callback
  to indicate that an unusual and possibly ill-advised operation is
  taking place.




(100) SQLITE_ROW

  The SQLITE_ROW result code returned by
  sqlite3_step() indicates that another row of output is available.




(101) SQLITE_DONE

  The SQLITE_DONE result code indicates that an operation has completed.
  The SQLITE_DONE result code is most commonly seen as a return value
  from sqlite3_step() indicating that the SQL statement has run to
  completion.  But SQLITE_DONE can also be returned by other multi-step
  interfaces such as sqlite3_backup_step().




(256) SQLITE_OK_LOAD_PERMANENTLY

  The sqlite3_load_extension() interface loads an
  extension into a single
  database connection.  The default behavior is for that extension to be
  automatically unloaded when the database connection closes.  However,
  if the extension entry point returns SQLITE_OK_LOAD_PERMANENTLY instead
  of SQLITE_OK, then the extension remains loaded into the process address
  space after the database connection closes.  In other words, the
  xDlClose methods of the sqlite3_vfs object is not called for the
  extension when the database connection closes.
  
  The SQLITE_OK_LOAD_PERMANENTLY return code is useful to
  loadable extensions that register new VFSes, for example.




(257) SQLITE_ERROR_MISSING_COLLSEQ

  The SQLITE_ERROR_MISSING_COLLSEQ result code means that an SQL
  statement could not be prepared because a collating sequence named
  in that SQL statement could not be located.
  
  Sometimes when this error code is encountered, the
  sqlite3_prepare_v2() routine will convert the error into
  SQLITE_ERROR_RETRY and try again to prepare the SQL statement
  using a different query plan that does not require the use of
  the unknown collating sequence.




(261) SQLITE_BUSY_RECOVERY

  The SQLITE_BUSY_RECOVERY error code is an extended error code
  for SQLITE_BUSY that indicates that an operation could not continue
  because another process is busy recovering a WAL mode database file
  following a crash.  The SQLITE_BUSY_RECOVERY error code only occurs
  on WAL mode databases.




(262) SQLITE_LOCKED_SHAREDCACHE

  The SQLITE_LOCKED_SHAREDCACHE result code indicates that access to
  an SQLite data record is blocked by another database connection that
  is using the same record in shared cache mode.  When two or more
  database connections share the same cache and one of the connections is
  in the middle of modifying a record in that cache, then other connections
  are blocked from accessing that data while the modifications are on-going
  in order to prevent the readers from seeing a corrupt or partially
  completed change.




(264) SQLITE_READONLY_RECOVERY

  The SQLITE_READONLY_RECOVERY error code is an extended error code
  for SQLITE_READONLY.  The SQLITE_READONLY_RECOVERY error code indicates
  that a WAL mode database cannot be opened because the database file
  needs to be recovered and recovery requires write access but only
  read access is available.




(266) SQLITE_IOERR_READ

  The SQLITE_IOERR_READ error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to read from a file on disk.  This error might result
  from a hardware malfunction or because a filesystem came unmounted
  while the file was open.




(267) SQLITE_CORRUPT_VTAB

  The SQLITE_CORRUPT_VTAB error code is an extended error code
  for SQLITE_CORRUPT used by virtual tables.  A virtual table might
  return SQLITE_CORRUPT_VTAB to indicate that content in the virtual table
  is corrupt.




(270) SQLITE_CANTOPEN_NOTEMPDIR

  The SQLITE_CANTOPEN_NOTEMPDIR error code is no longer used.




(275) SQLITE_CONSTRAINT_CHECK

  The SQLITE_CONSTRAINT_CHECK error code is an extended error code
  for SQLITE_CONSTRAINT indicating that a CHECK constraint failed.




(279) SQLITE_AUTH_USER

  The SQLITE_AUTH_USER error code is an extended error code
  for SQLITE_AUTH indicating that an operation was attempted on a
  database for which the logged in user lacks sufficient authorization.




(283) SQLITE_NOTICE_RECOVER_WAL

  The SQLITE_NOTICE_RECOVER_WAL result code is
  passed to the callback of
  sqlite3_log() when a WAL mode database file is recovered.




(284) SQLITE_WARNING_AUTOINDEX

  The SQLITE_WARNING_AUTOINDEX result code is
  passed to the callback of
  sqlite3_log() whenever automatic indexing is used.
  This can serve as a warning to application designers that the
  database might benefit from additional indexes.




(513) SQLITE_ERROR_RETRY

  The SQLITE_ERROR_RETRY is used internally to provoke sqlite3_prepare_v2()
  (or one of its sibling routines for creating prepared statements) to
  try again to prepare a statement that failed with an error on the
  previous attempt.




(516) SQLITE_ABORT_ROLLBACK

  The SQLITE_ABORT_ROLLBACK error code is an extended error code
  for SQLITE_ABORT indicating that an SQL statement aborted because
  the transaction that was active when the SQL statement first started
  was rolled back.  Pending write operations always fail with this error
  when a rollback occurs.  A ROLLBACK will cause a pending read operation
  to fail only if the schema was changed within the transaction being rolled
  back.




(517) SQLITE_BUSY_SNAPSHOT

  The SQLITE_BUSY_SNAPSHOT error code is an extended error code
  for SQLITE_BUSY that occurs on WAL mode databases when a database
  connection tries to promote a read transaction into a write transaction
  but finds that another database connection has already written to the
  database and thus invalidated prior reads.
  
  The following scenario illustrates how an SQLITE_BUSY_SNAPSHOT error
  might arise:
  
   Process A starts a read transaction on the database and does one
       or more SELECT statement.  Process A keeps the transaction open.
   Process B updates the database, changing values previous read by
       process A.
   Process A now tries to write to the database.  But process A's view
       of the database content is now obsolete because process B has
       modified the database file after process A read from it.  Hence
       process A gets an SQLITE_BUSY_SNAPSHOT error.
  




(518) SQLITE_LOCKED_VTAB

  The SQLITE_LOCKED_VTAB result code is not used by the SQLite core, but
  it is available for use by extensions.  Virtual table implementations
  can return this result code to indicate that they cannot complete the
  current operation because of locks held by other threads or processes.
  
  The R-Tree extension returns this result code when an attempt is made
  to update the R-Tree while another prepared statement is actively reading
  the R-Tree.  The update cannot proceed because any change to an R-Tree
  might involve reshuffling and rebalancing of nodes, which would disrupt
  read cursors, causing some rows to be repeated and other rows to be
  omitted.




(520) SQLITE_READONLY_CANTLOCK

  The SQLITE_READONLY_CANTLOCK error code is an extended error code
  for SQLITE_READONLY.  The SQLITE_READONLY_CANTLOCK error code indicates
  that SQLite is unable to obtain a read lock on a WAL mode database
  because the shared-memory file associated with that database is read-only.




(522) SQLITE_IOERR_SHORT_READ

  The SQLITE_IOERR_SHORT_READ error code is an extended error code
  for SQLITE_IOERR indicating that a read attempt in the VFS layer
  was unable to obtain as many bytes as was requested.  This might be
  due to a truncated file.




(523) SQLITE_CORRUPT_SEQUENCE

  The SQLITE_CORRUPT_SEQUENCE result code means that the schema of
  the sqlite_sequence table is corrupt.  The sqlite_sequence table
  is used to help implement the AUTOINCREMENT feature.  The
  sqlite_sequence table should have the following format:
  
  CREATE TABLE sqlite_sequence(name,seq);
  
  If SQLite discovers that the sqlite_sequence table has any other
  format, it returns the SQLITE_CORRUPT_SEQUENCE error.




(526) SQLITE_CANTOPEN_ISDIR

  The SQLITE_CANTOPEN_ISDIR error code is an extended error code
  for SQLITE_CANTOPEN indicating that a file open operation failed because
  the file is really a directory.




(531) SQLITE_CONSTRAINT_COMMITHOOK

  The SQLITE_CONSTRAINT_COMMITHOOK error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a
  commit hook callback returned non-zero that thus
  caused the SQL statement to be rolled back.




(539) SQLITE_NOTICE_RECOVER_ROLLBACK

  The SQLITE_NOTICE_RECOVER_ROLLBACK result code is
  passed to the callback of
  sqlite3_log() when a hot journal is rolled back.




(769) SQLITE_ERROR_SNAPSHOT

  The SQLITE_ERROR_SNAPSHOT result code might be returned when attempting
  to start a read transaction on an historical version of the database
  by using the sqlite3_snapshot_open() interface.  If the historical
  snapshot is no longer available, then the read transaction will fail
  with the SQLITE_ERROR_SNAPSHOT.  This error code is only possible if
  SQLite is compiled with -DSQLITE_ENABLE_SNAPSHOT.




(773) SQLITE_BUSY_TIMEOUT

  The SQLITE_BUSY_TIMEOUT error code indicates that a blocking Posix
  advisory file lock request in the VFS layer failed due to a timeout.
  Blocking Posix advisory locks are only
  available as a proprietary SQLite extension and even then are only
  supported if SQLite is compiled with the SQLITE_EANBLE_SETLK_TIMEOUT
  compile-time option.




(776) SQLITE_READONLY_ROLLBACK

  The SQLITE_READONLY_ROLLBACK error code is an extended error code
  for SQLITE_READONLY.  The SQLITE_READONLY_ROLLBACK error code indicates
  that a database cannot be opened because it has a hot journal that
  needs to be rolled back but cannot because the database is readonly.




(778) SQLITE_IOERR_WRITE

  The SQLITE_IOERR_WRITE error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to write into a file on disk.  This error might result
  from a hardware malfunction or because a filesystem came unmounted
  while the file was open.  This error should not occur if the filesystem
  is full as there is a separate error code (SQLITE_FULL) for that purpose.




(779) SQLITE_CORRUPT_INDEX

  The SQLITE_CORRUPT_INDEX result code means that SQLite detected
  an entry is or was missing from an index.  This is a special case of
  the SQLITE_CORRUPT error code that suggests that the problem might
  be resolved by running the REINDEX command, assuming no other
  problems exist elsewhere in the database file.




(782) SQLITE_CANTOPEN_FULLPATH

  The SQLITE_CANTOPEN_FULLPATH error code is an extended error code
  for SQLITE_CANTOPEN indicating that a file open operation failed because
  the operating system was unable to convert the filename into a full pathname.




(787) SQLITE_CONSTRAINT_FOREIGNKEY

  The SQLITE_CONSTRAINT_FOREIGNKEY error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a foreign key constraint failed.




(1032) SQLITE_READONLY_DBMOVED

  The SQLITE_READONLY_DBMOVED error code is an extended error code
  for SQLITE_READONLY.  The SQLITE_READONLY_DBMOVED error code indicates
  that a database cannot be modified because the database file has been
  moved since it was opened, and so any attempt to modify the database
  might result in database corruption if the processes crashes because the
  rollback journal would not be correctly named.




(1034) SQLITE_IOERR_FSYNC

  The SQLITE_IOERR_FSYNC error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to flush previously written content out of OS and/or
  disk-control buffers and into persistent storage.  In other words,
  this code indicates a problem with the fsync() system call in unix
  or the FlushFileBuffers() system call in windows.




(1038) SQLITE_CANTOPEN_CONVPATH

  The SQLITE_CANTOPEN_CONVPATH error code is an extended error code
  for SQLITE_CANTOPEN used only by Cygwin VFS and indicating that
  the cygwin_conv_path() system call failed while trying to open a file.
  See also: SQLITE_IOERR_CONVPATH




(1043) SQLITE_CONSTRAINT_FUNCTION

  The SQLITE_CONSTRAINT_FUNCTION error code is not currently used
  by the SQLite core.  However, this error code is available for use
  by extension functions.




(1288) SQLITE_READONLY_CANTINIT

  The SQLITE_READONLY_CANTINIT result code originates in the xShmMap method
  of a VFS to indicate that the shared memory region used by WAL mode
  exists buts its content is unreliable and unusable by the current process
  since the current process does not have write permission on the shared
  memory region.  (The shared memory region for WAL mode is normally a
  file with a "-wal" suffix that is mmapped into the process space.  If
  the current process does not have write permission on that file, then it
  cannot write into shared memory.)
  
  Higher level logic within SQLite will normally intercept the error code
  and create a temporary in-memory shared memory region so that the current
  process can at least read the content of the database.  This result code
  should not reach the application interface layer.




(1290) SQLITE_IOERR_DIR_FSYNC

  The SQLITE_IOERR_DIR_FSYNC error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to invoke fsync() on a directory.  The unix VFS attempts
  to fsync() directories after creating or deleting certain files to
  ensure that those files will still appear in the filesystem following
  a power loss or system crash.  This error code indicates a problem
  attempting to perform that fsync().




(1294) SQLITE_CANTOPEN_DIRTYWAL

  The SQLITE_CANTOPEN_DIRTYWAL result code is not used at this time.




(1299) SQLITE_CONSTRAINT_NOTNULL

  The SQLITE_CONSTRAINT_NOTNULL error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a NOT NULL constraint failed.




(1544) SQLITE_READONLY_DIRECTORY

  The SQLITE_READONLY_DIRECTORY result code indicates that the database
  is read-only because process does not have permission to create
  a journal file in the same directory as the database and the creation of
  a journal file is a prerequisite for writing.




(1546) SQLITE_IOERR_TRUNCATE

  The SQLITE_IOERR_TRUNCATE error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to truncate a file to a smaller size.




(1550) SQLITE_CANTOPEN_SYMLINK

  The SQLITE_CANTOPEN_SYMLINK result code is returned by the
  sqlite3_open() interface and its siblings when the
  SQLITE_OPEN_NOFOLLOW flag is used and the database file is
  a symbolic link.




(1555) SQLITE_CONSTRAINT_PRIMARYKEY

  The SQLITE_CONSTRAINT_PRIMARYKEY error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a PRIMARY KEY constraint failed.




(1802) SQLITE_IOERR_FSTAT

  The SQLITE_IOERR_FSTAT error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to invoke fstat() (or the equivalent) on a file in order
  to determine information such as the file size or access permissions.




(1811) SQLITE_CONSTRAINT_TRIGGER

  The SQLITE_CONSTRAINT_TRIGGER error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a RAISE function within
  a trigger fired, causing the SQL statement to abort.




(2058) SQLITE_IOERR_UNLOCK

  The SQLITE_IOERR_UNLOCK error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within xUnlock method on the sqlite3_io_methods object.




(2067) SQLITE_CONSTRAINT_UNIQUE

  The SQLITE_CONSTRAINT_UNIQUE error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a UNIQUE constraint failed.




(2314) SQLITE_IOERR_RDLOCK

  The SQLITE_IOERR_RDLOCK error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within xLock method on the sqlite3_io_methods object while trying
  to obtain a read lock.




(2323) SQLITE_CONSTRAINT_VTAB

  The SQLITE_CONSTRAINT_VTAB error code is not currently used
  by the SQLite core.  However, this error code is available for use
  by application-defined virtual tables.




(2570) SQLITE_IOERR_DELETE

  The SQLITE_IOERR_DELETE error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within xDelete method on the sqlite3_vfs object.




(2579) SQLITE_CONSTRAINT_ROWID

  The SQLITE_CONSTRAINT_ROWID error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a rowid is not unique.




(2826) SQLITE_IOERR_BLOCKED

  The SQLITE_IOERR_BLOCKED error code is no longer used.




(2835) SQLITE_CONSTRAINT_PINNED

  The SQLITE_CONSTRAINT_PINNED error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that an UPDATE trigger attempted
  do delete the row that was being updated in the middle of the update.




(3082) SQLITE_IOERR_NOMEM

  The SQLITE_IOERR_NOMEM error code is sometimes returned by the VFS
  layer to indicate that an operation could not be completed due to the
  inability to allocate sufficient memory.  This error code is normally
  converted into SQLITE_NOMEM by the higher layers of SQLite before
  being returned to the application.




(3091) SQLITE_CONSTRAINT_DATATYPE

  The SQLITE_CONSTRAINT_DATATYPE error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that an insert or update attempted
  to store a value inconsistent with the column's declared type
  in a table defined as STRICT.




(3338) SQLITE_IOERR_ACCESS

  The SQLITE_IOERR_ACCESS error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xAccess method on the sqlite3_vfs object.




(3594) SQLITE_IOERR_CHECKRESERVEDLOCK

  The SQLITE_IOERR_CHECKRESERVEDLOCK error code is
  an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xCheckReservedLock method on the sqlite3_io_methods object.




(3850) SQLITE_IOERR_LOCK

  The SQLITE_IOERR_LOCK error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the
  advisory file locking logic.
  Usually an SQLITE_IOERR_LOCK error indicates a problem obtaining
  a PENDING lock.  However it can also indicate miscellaneous
  locking errors on some of the specialized VFSes used on Macs.




(4106) SQLITE_IOERR_CLOSE

  The SQLITE_IOERR_CLOSE error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xClose method on the sqlite3_io_methods object.




(4362) SQLITE_IOERR_DIR_CLOSE

  The SQLITE_IOERR_DIR_CLOSE error code is no longer used.




(4618) SQLITE_IOERR_SHMOPEN

  The SQLITE_IOERR_SHMOPEN error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xShmMap method on the sqlite3_io_methods object
  while trying to open a new shared memory segment.




(4874) SQLITE_IOERR_SHMSIZE

  The SQLITE_IOERR_SHMSIZE error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xShmMap method on the sqlite3_io_methods object
  while trying to enlarge a "shm" file as part of
  WAL mode transaction processing.  This error may indicate that
  the underlying filesystem volume is out of space.




(5130) SQLITE_IOERR_SHMLOCK

  The SQLITE_IOERR_SHMLOCK error code is no longer used.




(5386) SQLITE_IOERR_SHMMAP

  The SQLITE_IOERR_SHMMAP error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xShmMap method on the sqlite3_io_methods object
  while trying to map a shared memory segment into the process address space.




(5642) SQLITE_IOERR_SEEK

  The SQLITE_IOERR_SEEK error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xRead or xWrite methods on the sqlite3_io_methods object
  while trying to seek a file descriptor to the beginning point of the
  file where the read or write is to occur.




(5898) SQLITE_IOERR_DELETE_NOENT

  The SQLITE_IOERR_DELETE_NOENT error code
  is an extended error code
  for SQLITE_IOERR indicating that the
  xDelete method on the sqlite3_vfs object failed because the
  file being deleted does not exist.




(6154) SQLITE_IOERR_MMAP

  The SQLITE_IOERR_MMAP error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xFetch or xUnfetch methods on the sqlite3_io_methods object
  while trying to map or unmap part of the database file into the
  process address space.




(6410) SQLITE_IOERR_GETTEMPPATH

  The SQLITE_IOERR_GETTEMPPATH error code is an extended error code
  for SQLITE_IOERR indicating that the VFS is unable to determine
  a suitable directory in which to place temporary files.




(6666) SQLITE_IOERR_CONVPATH

  The SQLITE_IOERR_CONVPATH error code is an extended error code
  for SQLITE_IOERR used only by Cygwin VFS and indicating that
  the cygwin_conv_path() system call failed.
  See also: SQLITE_CANTOPEN_CONVPATH




(6922) SQLITE_IOERR_VNODE

  The SQLITE_IOERR_VNODE error code is a code reserved for use
  by extensions.  It is not used by the SQLite core.




(7178) SQLITE_IOERR_AUTH

  The SQLITE_IOERR_AUTH error code is a code reserved for use
  by extensions.  It is not used by the SQLite core.




(7434) SQLITE_IOERR_BEGIN_ATOMIC

  The SQLITE_IOERR_BEGIN_ATOMIC error code indicates that the
  underlying operating system reported and error on the
  SQLITE_FCNTL_BEGIN_ATOMIC_WRITE file-control.  This only comes
  up when SQLITE_ENABLE_ATOMIC_WRITE is enabled and the database
  is hosted on a filesystem that supports atomic writes.




(7690) SQLITE_IOERR_COMMIT_ATOMIC

  The SQLITE_IOERR_COMMIT_ATOMIC error code indicates that the
  underlying operating system reported and error on the
  SQLITE_FCNTL_COMMIT_ATOMIC_WRITE file-control.  This only comes
  up when SQLITE_ENABLE_ATOMIC_WRITE is enabled and the database
  is hosted on a filesystem that supports atomic writes.




(7946) SQLITE_IOERR_ROLLBACK_ATOMIC

  The SQLITE_IOERR_ROLLBACK_ATOMIC error code indicates that the
  underlying operating system reported and error on the
  SQLITE_FCNTL_ROLLBACK_ATOMIC_WRITE file-control.  This only comes
  up when SQLITE_ENABLE_ATOMIC_WRITE is enabled and the database
  is hosted on a filesystem that supports atomic writes.




(8202) SQLITE_IOERR_DATA

  The SQLITE_IOERR_DATA error code is an extended error code
  for SQLITE_IOERR used only by checksum VFS shim to indicate that
  the checksum on a page of the database file is incorrect.




(8458) SQLITE_IOERR_CORRUPTFS

  The SQLITE_IOERR_CORRUPTFS error code is an extended error code
  for SQLITE_IOERR used only by a VFS to indicate that a seek or read
  failure was due to the request not falling within the file's boundary
  rather than an ordinary device failure. This often indicates a
  corrupt filesystem.



This page last modified on  2023-09-14 14:42:56 UTC


c2: The Checksum VFS Shim
c5: cksumvfs.html
c4:
1. Overview

The checksum VFS extension is a VFS shim that adds an 8-byte
checksum to the end of every page in an SQLite database.  The checksum
is added as each page is written and verified as each page is read.
The checksum is intended to help detect database corruption caused by
random bit-flips in the mass storage device.

The checksum VFS extension requires 
SQLite version 3.32.0 (2020-05-22) or later.  It will not
work with earlier versions of SQLite.

2. Compiling

The checksum VFS module is a loadable extension.  It is not
included in the amalgamation.  It must be added to SQLite
either at compile-time or at run-time.  The source code to
the checksum VFS module is in the
ext/misc/cksumvfs.c
source file in the
SQLite source tree.

To build the checksum VFS module into a run-time loadable
extension, use commands similar to the following:


(linux) &rarr; gcc -fPIC -shared cksumvfs.c -o cksumvfs.so
(mac) &rarr; clang -fPIC -dynamiclib cksumvfs.c -o cksumvfs.dylib
(windows) &rarr; cl cksumvfs.c -link -dll -out:cksumvfs.dll


You may want to add additional compiler options, of course,
according to the needs of your project.

To statically link this extension with your product,
compile it like any other C-language module but add the
"-DSQLITE_CKSUMVFS_STATIC" option so that this module knows that
it is being statically linked rather than dynamically linked.

3. Loading

To load this extension as a shared library, you first have to
bring up a dummy SQLite database connection to use as the argument
to the sqlite3_load_extension() API call.  Then you invoke the
sqlite3_load_extension() API and shutdown the dummy database
connection.  All subsequent database connections that are opened
will include this extension.  For example:

sqlite3 *db;
sqlite3_open(":memory:", &db);
sqlite3_load_extension(db, "./cksumvfs");
sqlite3_close(db);


If this extension is compiled with -DSQLITE_CKSUMVFS_STATIC and
statically linked against the application, initialize it using
a single API call as follows:

sqlite3_cksumvfs_init();


Cksumvfs is a VFS shim. When loaded, "cksmvfs" becomes the new
default VFS, and it uses the prior default VFS as the next VFS
down in the stack.  This is normally what you want.  However, in
complex situations where multiple VFS shims are being loaded,
it might be important to ensure that cksumvfs is loaded in the
correct order so that it sequences itself into the default VFS
Shim stack in the right order.

4. Usage

Open database connections using the sqlite3_open() or 
sqlite3_open_v2() interfaces, as normal.  Ordinary database files
(without a checksum) will operate normally.  Databases with 
checksums will return an SQLITE_IOERR_DATA error if a page is
encountered that contains an invalid checksum.

Checksumming only works on databases that have a reserve bytes
value of exactly 8.  The default value for reserve-bytes is 0.
Hence, newly created database files will omit the checksum by
default.  To create a database that includes a checksum, change
the reserve-bytes value to 8 by running code similar to this:

int n = 8;
sqlite3_file_control(db, 0, SQLITE_FCNTL_RESERVE_BYTES, &n);


If you do this immediately after creating a new database file,
before anything else has been written into the file, then that
might be all that you need to do.  Otherwise, the API call
above should be followed by:

sqlite3_exec(db, "VACUUM", 0, 0, 0);


It never hurts to run the VACUUM, even if you don't need it.
If the database is in WAL mode, you should shutdown and
reopen all database connections before continuing.

From the CLI, use the ".filectrl reserve_bytes 8" command, 
followed by "VACUUM;".

Note that SQLite allows the number of reserve-bytes to be
increased but not decreased.  So if a database file already
has a reserve-bytes value greater than 8, there is no way to
activate checksumming on that database, other than to dump
and restore the database file.  Note also that other extensions
might also make use of the reserve-bytes.  Checksumming will
be incompatible with those other extensions.

5. Verification Of Checksums

If any checksum is incorrect, the "PRAGMA quick_check" command
will find it.  To verify that checksums are actually enabled
and running, use SQL like the following:

SELECT count(*), verify_checksum(data)
  FROM sqlite_dbpage
 GROUP BY 2;


There are three possible outputs from the verify_checksum()
function: 1, 0, and NULL.  1 is returned if the checksum is
correct.  0 is returned if the checksum is incorrect.  NULL
is returned if the page is unreadable.  If checksumming is
enabled, the read will fail if the checksum is wrong, so the
usual result from verify_checksum() on a bad checksum is NULL.

If everything is OK, the query above should return a single
row where the second column is 1.  Any other result indicates
either that there is a checksum error, or checksum validation
is disabled.

6. Controlling Checksum Verification

The cksumvfs extension implements a new PRAGMA statement that can
be used to disable, re-enable, or query the status of checksum
verification:

PRAGMA checksum_verification;          -- query status
PRAGMA checksum_verification=OFF;      -- disable verification
PRAGMA checksum_verification=ON;       -- re-enable verification


The "checksum_verification" pragma will return "1" (true) or "0"
(false) if checksum verification is enabled or disabled, respectively.
"Verification" in this context means the feature that causes
SQLITE_IOERR_DATA errors if a checksum mismatch is detected while
reading.  Checksums are always kept up-to-date as long as the
reserve bytes value of the database is 8, regardless of the setting
of this pragma.  Checksum verification can be disabled (for example)
to do forensic analysis of a database that has previously reported
a checksum error.

The "checksum_verification" pragma will always respond with "0" if
the database file does not have a reserve bytes value of 8.  The
pragma will return no rows at all if the cksumvfs extension is
not loaded.
This page last modified on  2022-01-08 05:02:57 UTC


c2: SQLite's Built-in printf()
c5: printf.html
c4:
1. Overview

SQLite contains its own implementation of the string formatting routine "printf()",
accessible via the following interfaces:


 format() &rarr; an SQL function returning the formatted string
 sqlite3_mprintf() &rarr; Store the formatted string in memory obtained
     sqlite3_malloc64().
 sqlite3_snprintf() &rarr; Store the formatted string in a static buffer
 sqlite3_str_appendf() &rarr; Append formatted text to a dynamic string
 sqlite3_vmprintf() &rarr; Varargs version of sqlite3_mprintf()
 sqlite3_vsnprintf() &rarr; Varargs version of sqlite3_snprintf()
 sqlite3_str_vappendf() &rarr; Varargs version of sqlite3_str_appendf()


The same core string formatter is also used internally by SQLite.

1.1. Advantages

Why does SQLite have its own private built-in printf() implementation?
Why not use the printf() implementation from the standard C library?
Several reasons:




By using its own built-in implementation, SQLite guarantees that the
output will be the same on all platforms and in all LOCALEs.
This is important for consistency and for testing.  It would be problematic
if one machine gave an answer of "5.25e+08" and another gave an answer
of "5.250e+008".  Both answers are correct, but it is better when SQLite
always gives the same answer.


We know of no way to use the standard library printf() C interface to
implement the format() SQL function feature of SQLite.  The built-in
printf() implementation can be easily adapted to that task, however.


The printf() in SQLite supports new non-standard substitution
types (%q, %Q, %w, and %z), and enhanced substitution
behavior (%s and %z) that are useful both internally to SQLite
and to applications using SQLite.
Standard library printf()s cannot normally be extended in this way.


Via the sqlite3_mprintf() and sqlite3_vmprintf() interfaces,
the built-in printf() implementation supports the ability to render an
arbitrary-length string into a memory buffer obtained from sqlite3_malloc64().
This is safer and less error prone than trying to precompute an upper size
limit on the result string, allocate an appropriately sized buffer, and
then calling snprintf().


The SQLite-specific printf() supports a new flag (!) called the
"alternate-form-2" flag.  The alternate-form-2 flag changes the processing
of floating-point conversions in subtle ways so that the output is always
an SQL-compatible text representation of a floating-point number - something
that is not possible to achieve with standard-library printf().  For
string substitutions, the alternate-form-2 flag causes the width and
precision to be measured in characters instead of bytes, which simplifies
processing of strings containing multi-byte UTF8 characters.


The built-in SQLite has compile-time options such as
SQLITE_PRINTF_PRECISION_LIMIT that provide defense against 
denial-of-service attacks for application that expose the
printf() functionality to untrusted users.


Using a built-in printf() implementation means that SQLite has one
fewer dependency on the host environment, making it more portable.


1.2. Disadvantages


In fairness, having a built-in implementation of printf() also comes with
some disadvantages.  To wit:



The built-in printf() implementation uses extra code space 
(about 7800 bytes on GCC 5.4 with -Os).


The floating-point to text conversion subfunction for the built-in printf()
is limited in precision to 16 significant digits or 26 significant digits
if the "!" alternate-form-2 flag is used.
Every IEEE-754 double can be represented exactly as a decimal
value, but for many doubles the exact decimal representation requires
more than 16 or 26 significant digits.
The SQLite printf() function only renders the first 16 or 26 significant digits
because that can be done efficient and because 16 decimal digits it is sufficient
to distinguish every possible double value.  Use the decimal extension to get
the exact decimal equivalent of double value for the rare cases where that is
required.


The order of the buffer pointer and buffer size parameters in the built-in
snprintf() implementation is reversed from the order used in standard-library
implementations.


The built-in printf() implementation does not handle posix positional referencing
modifiers that allow the order of arguments to printf() to be different from the
order of the %-substitutions.  In the built-in printf(), the order of the arguments
must exactly match the order of the %-substitutions.



In spite of the disadvantages, the developers believe that having a built-in
printf() implementation inside of SQLite is a net positive.

2. Formatting Details

The format string for printf() is a template for the generated
string.  Substitutions are made whenever a "%" character appears in
the format string.  The "%" is followed by one or more additional
characters that describe the substitution.  Each substitution has
the following format:


%&#91;flags&#93;&#91;width&#93;&#91;.precision&#93;&#91;length&#93;type


All substitutions begin with a single "%" and end with a single type character.
The other elements of the substitution are optional.

To include a single "%" character in the output, put two consecutive
"%" characters in the template.

2.1. Substitution Types

The following chart shows the substitution types supported by SQLite:




Substitution TypeMeaning

%
Two "%" characters in a row are translated into a single "%" in the output,
    without substituting any values.

d, i
The argument is a signed integer which is displayed in decimal.

u
The argument is an unsigned integer which is displayed in decimal.

f
The argument is a double which is displayed in decimal.

e, E
The argument is a double which is displayed in exponential notation.
    The exponent character is 'e' or 'E' depending on the type.

g, G
The argument is a double which is displayed in either normal decimal
    notation or if the exponent is not close to zero, in exponential
    notation.

x, X
The argument is an integer which is displayed in hexadecimal.
    Lower-case hexadecimal is used for %x and upper-case is used
    for %X

o
The argument is an integer which is displayed in octal.

s, z



    The argument is either a zero-terminated string that is displayed,
    or a null pointer which is treated as an empty string.  For
    the %z type in C-language interfaces, sqlite3_free() is invoked
    on the string after it has been copied into the output. The %s and %z
    substitutions are identical for the SQL printf() function, with
    a NULL argument treated as an empty string.
    The %s substitution is universal among printf functions, but
    the %z substitution and safe treatment of null pointers
    are SQLite enhancements, not found in other
    printf() implementations.

c
For the C-language interfaces, the argument is an integer which
    is interpreted as a character.  For the format() SQL function the
    argument is a string from which the first character is extracted and
    displayed.

p
The argument is a pointer which is displayed as a hexadecimal address.
    Since the SQL language has no concept of a pointer, the %p substitution
    for the format() SQL function works like %x.

n
The argument is a pointer to an integer.  Nothing is displayed for
    this substitution type.  Instead, the integer to which the argument
    points is overwritten with the number of characters in the generated
    string that result from all format symbols to the left of the %n.

q, Q


    The argument is a zero-terminated string.  The string is printed with
    all single quote (') characters doubled so that the string can safely
    appear inside an SQL string literal.  The %Q substitution type also
    puts single-quotes on both ends of the substituted string.
    If the argument
    to %Q is a null pointer then the output is an unquoted "NULL".  In other
    words, a null pointer generates an SQL NULL, and a non-null pointer generates
    a valid SQL string literal.  If the argument to %q is a null pointer
    then no output is generated.  Thus a null-pointer to %q is the same as
    an empty string.
    For these  substitutions, the precision is the number of bytes or
    characters taken from the argument, not the number of bytes or characters that
    are written into the output.
    
    The %q and %Q substitutions are SQLite enhancements, not found in
    most other printf() implementations.

w


    This substitution works like %q except that it doubles all double-quote
    characters (") instead of single-quotes, making the result suitable for
    using with a double-quoted identifier name in an SQL statement.
    
    The %w substitution is an SQLite enhancements, not found in
    most other printf() implementations.



2.2. The Optional Length Field

The length of the argument value can be specified by one or more letters
that occur just prior to the substitution type letter.  In SQLite, the
length only matter for integer types.  The length is ignored for the
format() SQL function which always uses 64-bit values.  The following
table shows the length specifiers allowed by SQLite:




Length Specifier
Meaning

(default)
An "int" or "unsigned int".  32-bits on all modern systems.

l
A "long int" or "long unsigned int".  Also 32-bits on all modern systems.

ll
A "long long int" or "long long unsigned" or an "sqlite3_int64" or
    "sqlite3_uint64" value.  These are 64-bit integers on all modern systems.



Only the "ll" length modifier ever makes a difference for SQLite.  And
it only makes a difference when using the C-language interfaces.

2.3. The Optional Width Field

The width field specifies the minimum width of the substituted value in
the output.  If the string or number that is written into the output is shorter
than the width, then the value is padded.  Padding is on the left (the
value is right-justified) by default.  If the "-" flag is used, then the
padding is on the right and the value is left-justified.

The width is measured in bytes by default.  However, if the "!" flag is
present then the width is in characters.  This only makes a difference for
multi-byte utf-8 characters, and those only occur on string substitutions.

If the width is a single "*" character instead of a number, then the
actual width value is read as an integer from the argument list.  If the
value read is negative, then the absolute value is used for the width and
the value is left-justified as if the "-" flag were present.

If the value being substituted is larger than the width, then full value
is added to the output.  In other words, the width is the minimum width of
the value as it is rendered in the output.

2.4. The Optional Precision Field

The precision field, if it is present, must follow the width separated
by a single "." character.  If there is no width, then the "." that introduces
the precision immediately follows either the flags (if there are any) or
the initial "%".

For string substitutions (%s, %z, %q, %Q, or %w) the precision is the number
of byte or character used from the argument.  The number is bytes by default but
is characters if the "!" flag is present.  If there is no precision, then the
entire string is substituted.  Examples:  "%.3s" substitutes the first 3 bytes
of the argument string.  "%!.3s" substitutes the first three characters of the
argument string.

For integer substitutions (%d, %i, %x, %X, %o, and %p) the precision specifies
minimum number of digits to display.  Leading zeros are added if necessary, to
expand the output to the minimum number of digits.

For floating-point substitutions (%e, %E, %f, %g, %G) the precision specifies 
the number of digits to display to the right of the decimal point.

For the character substitution (%c) a precision N greater than 1 causes the
character to be repeated N times.  This is a non-standard extension found only
in SQLite.

If the precision is a single "*" character instead of a number, then the
actual precision value is read as an integer from the argument list.

2.5. The Options Flags Field

Flags consist of zero or more characters that immediately follow the
"%" that introduces the substitution.  The various flags and their meanings
are as follows:




Flag
Meaning

-
Left-justify the value in the output.  The default is to right-justify.
If the width is zero or is otherwise less than the length of the value being
substituted, then there is no padding and the "-" flag is a no-op.

+
For signed numeric substitutions, include a "+" sign before positive numbers.
A "-" sign always appears before negative numbers regardless of flag settings.

(space)
For signed numeric substitutions, prepend a single space before positive
numbers.

0
(The zero-padding option)
Prepend as many "0" characters to numeric substitutions as necessary to
expand the value out to the specified width.  If the width field is omitted,
then this flag is a no-op.
Infinity and NaN (Not-A-Number) floating point values are normally rendered
as "Inf" and "NaN", respectively, but with the zero-padding option enabled
they are rendered as "9.0e+999" and "null".  In other words, with the zero-padding
option, floating-point Infinity and NaN are rendered as valid SQL and JSON literals.

#
This is the "alternate-form-1" flag.
For %g and %G substitutions, this causes trailing zeros to be removed.
This flag forces a decimal point to appear for all floating-point substitutions.
For %o, %x, and %X substitutions, the alternate-form-1 flag cause the value
to be prepended with "0", "0x", or "0X", respectively.



,
The comma option causes comma separators to be added to the output of numeric
substitutions (%d, %f, and similar) before every 3rd digits to the left of
the decimal point.  No commas are added for digits to the right of the decimal point.
This can help humans to more easily discern the magnitude of large integer values.
For example,
the value 2147483647 would be rendered as "2147483647" using "%d" but would
appear as "2,147,483,647" with "%,d".  This flag is a non-standard extension.

!
This is the "alternate-form-2 flag.
For string substitutions, this flag causes the width and precision to be understand
in terms of characters rather than bytes.
For floating point substitutions, the alternate-form-2 flag increases the 
maximum number of significant digits displayed from 16 to 26,
forces the display of the decimal point and causes at least one digit
to appear after the decimal point.
The alternate-form-2 flag is a non-standard extension that appears in no
other printf() implementations, as far as we know.



3. Implementation And History


The core string formatting routine is the sqlite3VXPrintf() function found in the
printf.c source file.  All the
various interfaces invoke (sometimes indirectly) this one core function.
The sqlite3VXPrintf() function began as code written by the first author
of SQLite (Hipp) when he was a graduate student at Duke University in the
late 1980s.  Hipp kept this printf() implementation in his personal toolbox until
he started working on SQLite in 2000.  The code was incorporated into the
SQLite source tree on 2000-10-08
for SQLite version 1.0.9.


The Fossil Version Control System uses its own
printf() implementation that is derived from an early version of the SQLite
printf() implementation, but those two implementations have since diverged.


The sqlite3_snprintf() function has its buffer pointer and buffer size
arguments reversed from what is found in the standard C library snprintf()
routine. This is because there was no snprintf() routine in the
standard C library
when Hipp was first implementing his version, and he chose a different order
than the designers of the standard C library.
This page last modified on  2023-06-29 23:18:37 UTC


c2: Command Line Shell For SQLite
c5: cli.html
c4:
1. Getting Started

The SQLite project provides a simple command-line program named
sqlite3 (or sqlite3.exe on Windows)
that allows the user to manually enter and execute SQL
statements against an SQLite database or against a
ZIP archive.  This document provides a brief
introduction on how to use the sqlite3 program.

Start the sqlite3 program by typing "sqlite3" at the
command prompt, optionally followed
by the name of the file that holds the SQLite database
(or ZIP archive).  If the named
file does not exist, a new database file with the given name will be
created automatically.  If no database file is specified on the
command-line, a temporary database is created and automatically deleted when
the "sqlite3" program exits.

On startup, the sqlite3 program will show a brief banner
message then prompt you to enter SQL.  Type in SQL statements (terminated
by a semicolon), press "Enter" and the SQL will be executed.

For example, to create a new SQLite database named "ex1"
with a single table named "tbl1", you might do this:

$ sqlite3 ex1
SQLite version 3.36.0 2021-06-18 18:36:39
Enter ".help" for usage hints.
sqlite&gt; create table tbl1(one text, two int);
sqlite&gt; insert into tbl1 values('hello!',10);
sqlite&gt; insert into tbl1 values('goodbye', 20);
sqlite&gt; select * from tbl1;
hello!|10
goodbye|20
sqlite&gt;


Terminate the sqlite3 program by typing your system
End-Of-File character (usually a Control-D).  Use the interrupt
character (usually a Control-C) to stop a long-running SQL statement.

Make sure you type a semicolon at the end of each SQL command!
The sqlite3 program looks for a semicolon to know when your SQL command is
complete.  If you omit the semicolon, sqlite3 will give you a
continuation prompt and wait for you to enter more text to
complete the SQL command.  This feature allows you to
enter SQL commands that span multiple lines.  For example:


sqlite&gt; CREATE TABLE tbl2 (
   ...&gt;   f1 varchar(30) primary key,
   ...&gt;   f2 text,
   ...&gt;   f3 real
   ...&gt; );
sqlite&gt;




2. Double-click Startup On Windows

Windows users can double-click on the sqlite3.exe icon to cause
the command-line shell to pop-up a terminal window running SQLite.  However,
because double-clicking starts the sqlite3.exe without command-line arguments,
no database file will have been specified, so SQLite will use a temporary
database that is deleted when the session exits.
To use a persistent disk file as the database, enter the ".open" command
immediately after the terminal window starts up:

SQLite version 3.36.0 2021-06-18 18:36:39
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite&gt; .open ex1.db
sqlite&gt;


The example above causes the database file named "ex1.db" to be opened
and used.  The "ex1.db" file is created if it does not previously exist.
You might want to
use a full pathname to ensure that the file is in the directory that you
think it is in.  Use forward-slashes as the directory separator character.
In other words use "c:/work/ex1.db", not "c:\work\ex1.db".

Alternatively, you can create a new database using the default temporary
storage, then save that database into a disk file using the ".save" command:

SQLite version 3.36.0 2021-06-18 18:36:39
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite&gt; ... many SQL commands omitted ...
sqlite&gt; .save ex1.db
sqlite&gt;


Be careful when using the ".save" command as it will overwrite any
preexisting database files having the same name without prompting for
confirmation.  As with the ".open" command, you might want to use a
full pathname with forward-slash directory separators to avoid ambiguity.



3. Special commands to sqlite3 (dot-commands)


Most of the time, sqlite3 just reads lines of input and passes them
on to the SQLite library for execution.
But input lines that begin with a dot (".")
are intercepted and interpreted by the sqlite3 program itself.
These "dot commands" are typically used to change the output format
of queries, or to execute certain prepackaged query statements.
There were originally just a few dot commands, but over the years
many new features have accumulated so that today there are over 60.



For a listing of the available dot commands, you can enter ".help" with
no arguments.  Or enter ".help TOPIC" for detailed information about TOPIC.
The list of available dot-commands follows:


sqlite&gt; .help
.archive ...             Manage SQL archives
.auth ON|OFF             Show authorizer callbacks
.backup ?DB? FILE        Backup DB (default "main") to FILE
.bail on|off             Stop after hitting an error.  Default OFF
.cd DIRECTORY            Change the working directory to DIRECTORY
.changes on|off          Show number of rows changed by SQL
.check GLOB              Fail if output since .testcase does not match
.clone NEWDB             Clone data into NEWDB from the existing database
.connection &#91;close&#93; &#91;#&#93;  Open or close an auxiliary database connection
.databases               List names and files of attached databases
.dbconfig ?op? ?val?     List or change sqlite3_db_config() options
.dbinfo ?DB?             Show status information about the database
.dump ?OBJECTS?          Render database content as SQL
.echo on|off             Turn command echo on or off
.eqp on|off|full|...     Enable or disable automatic EXPLAIN QUERY PLAN
.excel                   Display the output of next command in spreadsheet
.exit ?CODE?             Exit this program with return-code CODE
.expert                  EXPERIMENTAL. Suggest indexes for queries
.explain ?on|off|auto?   Change the EXPLAIN formatting mode.  Default: auto
.filectrl CMD ...        Run various sqlite3_file_control() operations
.fullschema ?--indent?   Show schema and the content of sqlite_stat tables
.headers on|off          Turn display of headers on or off
.help ?-all? ?PATTERN?   Show help text for PATTERN
.import FILE TABLE       Import data from FILE into TABLE
.imposter INDEX TABLE    Create imposter table TABLE on index INDEX
.indexes ?TABLE?         Show names of indexes
.limit ?LIMIT? ?VAL?     Display or change the value of an SQLITE_LIMIT
.lint OPTIONS            Report potential schema issues.
.load FILE ?ENTRY?       Load an extension library
.log FILE|off            Turn logging on or off.  FILE can be stderr/stdout
.mode MODE ?TABLE?       Set output mode
.nonce STRING            Disable safe mode for one command if the nonce matches
.nullvalue STRING        Use STRING in place of NULL values
.once ?OPTIONS? ?FILE?   Output for the next SQL command only to FILE
.open ?OPTIONS? ?FILE?   Close existing database and reopen FILE
.output ?FILE?           Send output to FILE or stdout if FILE is omitted
.parameter CMD ...       Manage SQL parameter bindings
.print STRING...         Print literal STRING
.progress N              Invoke progress handler after every N opcodes
.prompt MAIN CONTINUE    Replace the standard prompts
.quit                    Stop interpreting input stream, exit if primary.
.read FILE               Read input from FILE
.recover                 Recover as much data as possible from corrupt db.
.restore ?DB? FILE       Restore content of DB (default "main") from FILE
.save FILE               Write in-memory database into FILE
.scanstats on|off        Turn sqlite3_stmt_scanstatus() metrics on or off
.schema ?PATTERN?        Show the CREATE statements matching PATTERN
.selftest ?OPTIONS?      Run tests defined in the SELFTEST table
.separator COL ?ROW?     Change the column and row separators
.session ?NAME? CMD ...  Create or control sessions
.sha3sum ...             Compute a SHA3 hash of database content
.shell CMD ARGS...       Run CMD ARGS... in a system shell
.show                    Show the current values for various settings
.stats ?ARG?             Show stats or turn stats on or off
.system CMD ARGS...      Run CMD ARGS... in a system shell
.tables ?TABLE?          List names of tables matching LIKE pattern TABLE
.testcase NAME           Begin redirecting output to 'testcase-out.txt'
.testctrl CMD ...        Run various sqlite3_test_control() operations
.timeout MS              Try opening locked tables for MS milliseconds
.timer on|off            Turn SQL timer on or off
.trace ?OPTIONS?         Output each SQL statement as it is run
.vfsinfo ?AUX?           Information about the top-level VFS
.vfslist                 List all available VFSes
.vfsname ?AUX?           Print the name of the VFS stack
.width NUM1 NUM2 ...     Set minimum column widths for columnar output
sqlite&gt;




4. Rules for "dot-commands", SQL and More

4.1. Line Structure

The CLI's input is parsed into a sequence consisting of:
  
    SQL statements;
    dot-commands; or
    CLI comments
  
SQL statements are free-form, and can be spread across multiple lines,
  with whitespace or SQL comments embedded anywhere.
  They are terminated by either a ';' character at the end of an input line,
  or a '/' character or the word "go" on a line by itself.
  When not at the end of an input line, the ';' character
  acts to separate SQL statements.
  Trailing whitespace is ignored for purposes of termination.
A dot-command has a more restrictive structure:

It must begin with its "." at the left margin
    with no preceding whitespace.
It must be entirely contained on a single input line.
It cannot occur in the middle of an ordinary SQL
    statement.  In other words, it cannot occur at a
    continuation prompt.
There is no comment syntax for dot-commands.

The CLI also accepts whole-line comments that
begin with a '#' character and extend to the end of the line.
There can be no with whitespace prior to the '#'.

4.2. Dot-command arguments
The arguments passed to dot-commands are parsed from the command tail,
  per these rules:

  The trailing newline and any other trailing whitespace is discarded;
  Whitespace immediately following the dot-command name, or any argument
    input end bound is discarded;
  An argument input begins with any non-whitespace character;
  An argument input ends with a character which
    depends upon its leading character thusly:
  
    for a leading single-quote ('), a single-quote acts
      as the end delimiter;
    for a leading double-quote ("), an unescaped double-quote
      acts as the end delimiter;
    for any other leading character, the end delimiter is
      any whitespace; and
    the command tail end acts as the end delimiter for any argument;
  
  Within a double-quoted argument input, a backslash-escaped double-quote
    is part of the argument rather than its terminating quote;
  Within a double-quoted argument, traditional C-string literal, backslash
    escape sequence translation is done; and
  Argument input delimiters (the bounding quotes or whitespace)
    are discarded to yield the passed argument.


4.3. Dot-command execution
The dot-commands
are interpreted by the sqlite3.exe command-line program, not by
SQLite itself.  So none of the dot-commands will work as an argument
to SQLite interfaces such as sqlite3_prepare() or sqlite3_exec().



5. Changing Output Formats

The sqlite3 program is able to show the results of a query
in 14 different output formats:


 ascii
 box
 csv
 column
 html
 insert
 json
 line
 list
 markdown
 quote
 table
 tabs
 tcl


You can use the ".mode" dot command to switch between these output
formats.
The default output mode is "list".  In
list mode, each row of a query result is written on one line of
output and each column within that row is separated by a specific
separator string.  The default separator is a pipe symbol ("|").
List mode is especially useful when you are going to send the output
of a query to another program (such as AWK) for additional processing.

sqlite&gt; .mode list
sqlite&gt; select * from tbl1;
hello!|10
goodbye|20
sqlite&gt;


Type ".mode" with no arguments to show the current mode:

sqlite&gt; .mode
current output mode: list
sqlite&gt;


Use the ".separator" dot command to change the separator.
For example, to change the separator to a comma and
a space, you could do this:

sqlite&gt; .separator ", "
sqlite&gt; select * from tbl1;
hello!, 10
goodbye, 20
sqlite&gt;


The next ".mode" command might reset the ".separator" back to some
default value (depending on its arguments).
So you will likely need to repeat the ".separator" command whenever you
change modes if you want to continue using a non-standard separator.



In "quote" mode, the output is formatted as SQL literals.  Strings are
enclosed in single-quotes and internal single-quotes are escaped by doubling.
Blobs are displayed in hexadecimal blob literal notation (Ex: x'abcd').
Numbers are displayed as ASCII text and NULL values are shown as "NULL".
All columns are separated from each other by a comma (or whatever alternative
character is selected using ".separator").

sqlite&gt; .mode quote
sqlite&gt; select * from tbl1;
'hello!',10
'goodbye',20
sqlite&gt;


In "line" mode, each column in a row of the database
is shown on a line by itself.  Each line consists of the column
name, an equal sign and the column data.  Successive records are
separated by a blank line.  Here is an example of line mode
output:

sqlite&gt; .mode line
sqlite&gt; select * from tbl1;
one = hello!
two = 10

one = goodbye
two = 20
sqlite&gt;





In column mode, each record is shown on a separate line with the
data aligned in columns.  For example:

sqlite&gt; .mode column
sqlite&gt; select * from tbl1;
one       two
--------  ---
hello!    10
goodbye   20
sqlite&gt;


In "column" mode (and also in "box", "table", and "markdown" modes)
the width of columns adjusts automatically.  But you can override this,
providing a speicified width for each column using the ".width" command.
The arguments to ".width" are integers which are the number of
characters to devote to each column.  Negative numbers mean right-justify.
Thus:

sqlite&gt; .width 12 -6
sqlite&gt; select * from tbl1;
one              two
------------  ------
hello!            10
goodbye           20
sqlite&gt;


A width of 0 means the column width is chosen automatically.
Unspecified column widths become zero.  Hence, the command
".width" with no arguments resets all column widths to zero and
hence causes all column widths to be determined automatically.

The "column" mode is a tabular output format.  Other
tabular output formats are "box", "markdown", and "table":

sqlite&gt; .width
sqlite&gt; .mode markdown
sqlite&gt; select * from tbl1;
|   one   | two |
|---------|-----|
| hello!  | 10  |
| goodbye | 20  |
sqlite&gt; .mode table
sqlite&gt; select * from tbl1;
+---------+-----+
|   one   | two |
+---------+-----+
| hello!  | 10  |
| goodbye | 20  |
+---------+-----+
sqlite&gt; .mode box
sqlite&gt; select * from tbl1;
┌─────────┬─────┐
│   one   │ two │
├─────────┼─────┤
│ hello!  │ 10  │
│ goodbye │ 20  │
└─────────┴─────┘
sqlite&gt;





The columnar modes accept some addition options to control formatting.
The "--wrap N" option (where N is an integer) causes columns
to wrap text that is longer than N characters.  Wrapping is disabled if
N is zero.

sqlite&gt; insert into tbl1 values('The quick fox jumps over a lazy brown dog.',90);
sqlite&gt; .mode box --wrap 30
sqlite&gt; select * from tbl1 where two&gt;50;
┌────────────────────────────────┬─────┐
│              one               │ two │
├────────────────────────────────┼─────┤
│ The quick fox jumps over a laz │ 90  │
│ y brown dog.                   │     │
└────────────────────────────────┴─────┘
sqlite&gt;


Wrapping happens after exactly N characters,
which might be in the middle of a word.
To wrap at a word boundary, add the "--wordwrap on" option
(or just "-ww" for short):

sqlite&gt; .mode box --wrap 30 -ww
sqlite&gt; select * from tbl1 where two&gt;50;
┌─────────────────────────────┬─────┐
│             one             │ two │
├─────────────────────────────┼─────┤
│ The quick fox jumps over a  │ 90  │
│ lazy brown dog.             │     │
└─────────────────────────────┴─────┘
sqlite&gt;


The "--quote" option causes the results in each column to be
quoted like an SQL literal, as in the "quote" mode.  See the on-line
help for additional options.



The command ".mode box --wrap 60 --quote" is so useful for general-purpose
database queries that it is given its own alias.  Instead of typing out
that whole 27-character command, you can just say ".mode qbox".

Another useful output mode is "insert".  In insert mode, the output
is formatted to look like SQL INSERT statements.  Use insert
mode to generate text that can later be used to input data into a
different database.

When specifying insert mode, you have to give an extra argument
which is the name of the table to be inserted into.  For example:

sqlite&gt; .mode insert new_table
sqlite&gt; select * from tbl1 where two&lt;50;
INSERT INTO "new_table" VALUES('hello',10);
INSERT INTO "new_table" VALUES('goodbye',20);
sqlite&gt;



Other output modes include "csv", "json", and "tcl".  Try these
yourself to see what they do.




6. Querying the database schema

The sqlite3 program provides several convenience commands that
are useful for looking at the schema of the database.  There is
nothing that these commands do that cannot be done by some other
means.  These commands are provided purely as a shortcut.



For example, to see a list of the tables in the database, you
can enter ".tables".


sqlite&gt; .tables
tbl1 tbl2
sqlite&gt;



The ".tables" command is similar to setting list mode then
executing the following query:

SELECT name FROM sqlite_schema
WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%'
ORDER BY 1


But the ".tables" command does more.  It queries the sqlite_schema table
for all attached databases, not just the primary database.  And it arranges
its output into neat columns.

The ".indexes" command works in a similar way to list all of
the indexes. If the ".indexes" command is given an argument which is
the name of a table, then it shows just indexes on that table.



The ".schema" command shows the complete schema for the database,
or for a single table if an optional tablename argument is provided:

sqlite&gt; .schema
create table tbl1(one varchar(10), two smallint)
CREATE TABLE tbl2 (
  f1 varchar(30) primary key,
  f2 text,
  f3 real
);
sqlite&gt; .schema tbl2
CREATE TABLE tbl2 (
  f1 varchar(30) primary key,
  f2 text,
  f3 real
);
sqlite&gt;



The ".schema" command is roughly the same as setting
list mode, then entering the following query:

SELECT sql FROM sqlite_schema
ORDER BY tbl_name, type DESC, name


As with ".tables", the ".schema" command shows the schema for
all attached databases.  If you only want to see the schema for
a single database (perhaps "main") then you can add an argument
to ".schema" to restrict its output:

sqlite&gt; .schema main.*


The ".schema" command can be augmented with the "--indent" option,
in which case it tries to reformat the various CREATE statements of
the schema so that they are more easily readable by humans.



The ".databases" command shows a list of all databases open in
the current connection.  There will always be at least 2.  The first
one is "main", the original database opened.  The second is "temp",
the database used for temporary tables. There may be additional
databases listed for databases attached using the ATTACH statement.
The first output column is the name the database is attached with,
and the second result column is the filename of the external file.
There may be a third result column which will be either "'r/o'" or
"'r/w'" depending on whether the database file is read-only or read-write.
And there might be a fourth result column showing the result of
sqlite3_txn_state() for that database file.

sqlite&gt; .databases




The ".fullschema" dot-command works like the ".schema" command in
that it displays the entire database schema.  But ".fullschema" also
includes dumps of the statistics tables "sqlite_stat1", "sqlite_stat3",
and "sqlite_stat4", if they exist.  The ".fullschema" command normally
provides all of the information needed to exactly recreate a query
plan for a specific query.  When reporting suspected problems with
the SQLite query planner to the SQLite development team, developers
are requested to provide the complete ".fullschema" output as part
of the trouble report.  Note that the sqlite_stat3 and sqlite_stat4
tables contain samples of index entries and so might contain sensitive
data, so do not send the ".fullschema" output of a proprietary database
over a public channel.



7. Opening Database Files

The ".open" command opens a new database connection, after first closing the
previously opened database command.  In its simplest form, the ".open" command merely
invokes sqlite3_open() on the file named as its argument.  Use the name ":memory:"
to open a new in-memory database that disappears when the CLI exits or when the
".open" command is run again.
Or use no name to open a private, temporary on-disk database which
will also disappear upon exit or use of ".open".

If the --new option is included with ".open", then the database is reset prior
to being opened.  Any prior data is destroyed.  This is a destructive overwrite of
prior data and no confirmation is requested, so use this option carefully.

The --readonly option opens the database in read-only mode.  Write will be
prohibited.

The --deserialize option causes the entire content of the on-disk file to be
read into memory and then opened as an in-memory database using the
sqlite3_deserialize() interface.  This will, of course, require a lot of memory
if you have a large database.  Also, any changes you make to the database will not
be saved back to disk unless you explicitly save them using the ".save" or ".backup"
commands.

The --append option causes the SQLite database to be appended to an existing
file rather than working as a stand-alone file.  See the
appendvfs extension for
more information.

The --zip option causes the specified input file to be interpreted as a ZIP archive
instead of as an SQLite database file.

The --hexdb option causes the database content to be to be read from subsequent
lines of input in a hex format, rather than from a separate file on disk.
The "dbtotxt" command-line tool can be used to generate
the appropriate text for a database.  The --hexdb option is intended for use by the
SQLite developers for testing purposes.  We do not know of any use cases for this
option outside of internal SQLite testing and development.

8. Redirecting I/O



8.1. Writing results to a file

By default, sqlite3 sends query results to standard output.  You
can change this using the ".output" and ".once" commands.  Just put
the name of an output file as an argument to .output and all subsequent
query results will be written to that file.  Or use the .once command
instead of .output and output will only be redirected for the single next
command before reverting to the console.  Use .output with no arguments to
begin writing to standard output again.  For example:

sqlite&gt; .mode list
sqlite&gt; .separator |
sqlite&gt; .output test_file_1.txt
sqlite&gt; select * from tbl1;
sqlite&gt; .exit
$ cat test_file_1.txt
hello|10
goodbye|20
$


If the first character of the ".output" or ".once" filename is a pipe
symbol ("|") then the remaining characters are treated as a command and the
output is sent to that command.  This makes it easy to pipe the results
of a query into some other process.  For example, the
"open -f" command on a Mac opens a text editor to display the content that
it reads from standard input.  So to see the results of a query
in a text editor, one could type:

sqlite&gt; .once | open -f
sqlite&gt; SELECT * FROM bigTable;


If the ".output" or ".once" commands have an argument of "-e" then
output is collected into a temporary file and the system text editor is
invoked on that text file.  Thus, the command ".once -e" achieves the
same result as ".once '|open -f'" but with the benefit of being portable
across all systems.

If the ".output" or ".once" commands have a "-x" argument, that causes
them to accumulate output as Comma-Separated-Values (CSV) in a temporary
file, then invoke the default system utility for viewing CSV files
(usually a spreadsheet program) on the result.  This is a quick way of
sending the result of a query to a spreadsheet for easy viewing:

sqlite&gt; .once -x
sqlite&gt; SELECT * FROM bigTable;




The ".excel" command is an alias for ".once -x".  It does exactly the same
thing.



8.2. Reading SQL from a file

In interactive mode, sqlite3 reads input text (either SQL statements
or dot-commands) from the keyboard.  You can also redirect input from
a file when you launch sqlite3, of course, but then you do not have the
ability to interact with the program.  Sometimes it is useful to run an
SQL script contained in a file entering other commands from the command-line.
For this, the ".read" dot-command is provided.

The ".read" command takes a single argument which is (usually) the name
of a file from which to read input text.

sqlite&gt; .read myscript.sql


The ".read" command temporarily stops reading from the keyboard and instead
takes its input from the file named.  Upon reaching the end of the file,
input reverts back to the keyboard.  The script file may contain dot-commands,
just like ordinary interactive input.

If the argument to ".read" begins with the "|" character, then instead of
opening the argument as a file, it runs the argument (without the leading "|")
as a command, then uses the output of that command as its input.  Thus, if you
have a script that generates SQL, you can execute that SQL directly using
a command similar to the following:

sqlite&gt; .read |myscript.bat





8.3. File I/O Functions

The command-line shell adds two application-defined SQL functions that
facilitate reading content from a file into a table column, and writing the
content of a column into a file, respectively.

The readfile(X) SQL function reads the entire content of the file named
X and returns that content as a BLOB.  This can be used to load content into
a table.  For example:

sqlite&gt; CREATE TABLE images(name TEXT, type TEXT, img BLOB);
sqlite&gt; INSERT INTO images(name,type,img)
   ...&gt;   VALUES('icon','jpeg',readfile('icon.jpg'));


The writefile(X,Y) SQL function write the blob Y into the file named X
and returns the number of bytes written.  Use this function to extract
the content of a single table column into a file.  For example:

sqlite&gt; SELECT writefile('icon.jpg',img) FROM images WHERE name='icon';


Note that the readfile(X) and writefile(X,Y) functions are extension
functions and are not built into the core SQLite library.  These routines
are available as a loadable extension in the
ext/misc/fileio.c
source file in the SQLite source code repositories.



8.4. The edit() SQL function

The CLI has another built-in SQL function named edit().  Edit() takes
one or two arguments.  The first argument is a value - often a large
multi-line string to be edited.  The second argument is the invocation
for a text editor. (It may include options to affect the editor's
behavior.) If the second argument is omitted, the VISUAL environment
variable is used.  The edit() function writes its first argument into a
temporary file, invokes the editor on the temporary file, rereads the file
back into memory after the editor is done, then returns the edited text.

The edit() function can be used to make changes to large text
values.  For example:

sqlite&gt; UPDATE docs SET body=edit(body) WHERE name='report-15';


In this example, the content of the docs.body field for the entry where
docs.name is "report-15" will be sent to the editor.  After the editor returns,
the result will be written back into the docs.body field.

The default operation of edit() is to invoke a text editor.  But by using
an alternative edit program in the second argument, you can also get it to edit
images or other non-text resources.  For example, if you want to modify a JPEG
image that happens to be stored in a field of a table, you could run:

sqlite&gt; UPDATE pics SET img=edit(img,'gimp') WHERE id='pic-1542';


The edit program can also be used as a viewer, by simply ignoring the
return value.  For example, to merely look at the image above, you might run:

sqlite&gt; SELECT length(edit(img,'gimp')) WHERE id='pic-1542';




8.5. Importing files as CSV or other formats

Use the ".import" command to import CSV (comma separated value)
or similarly delimited data into an SQLite table.
The ".import" command takes two arguments which are the
source from which data is to be read and the name of the
SQLite table into which the data is to be inserted. The source argument
is the name of a file to be read or, if it begins with a "|" character,
it specifies a command which will be run to produce the input data.

Note that it may be important to set the "mode" before running the
 ".import" command.  This is prudent to prevent the command-line shell
from trying to interpret the input file text as some format other than
how the file is structured. If the --csv or --ascii options are used,
they control import input delimiters. Otherwise, the delimiters are
those in effect for the current output mode.

To import into a table not in the "main" schema, the --schema option
may be used to specify that the table is in some other schema. This can
be useful for ATTACH'ed databases or to import into a TEMP table.

When .import is run, its treatment of the first input row depends
upon whether the target table already exists. If it does not exist,
the table is automatically created and the content of the first input
row is used to set the name of all the columns in the table. In this
case, the table data content is taken from the second and subsequent
input rows. If the target table already exists, every row of the
input, including the first, is taken to be actual data content.  If
the input file contains an initial row of column labels, you can make
the .import command skip that initial row using the "--skip 1" option.

Here is an example usage, loading a pre-existing temporary table
from a CSV file which has column names in its first row:

sqlite&gt; .import --csv --skip 1 --schema temp C:/work/somedata.csv tab1


While reading input data in modes other than 'ascii', ".import"
interprets input as records composed of fields according to the RFC 4180
specification with this exception: The input record and field separators
are as set by the mode or by use of the .separator command. Fields are
always subject to quote removal to reverse quoting done per RFC 4180,
except in ascii mode.

To import data with arbitrary delimiters and no quoting,
first set ascii mode (".mode&nbsp;ascii"), then set the field
and record delimiters using the ".separator" command. This
will suppress dequoting. Upon ".import", the data will be split
into fields and records according to the delimiters so specified.



8.6. Export to CSV

To export an SQLite table (or part of a table) as CSV, simply set
the "mode" to "csv" and then run a query to extract the desired rows
of the table. The output will formatted as CSV per RFC 4180.

sqlite&gt; .headers on
sqlite&gt; .mode csv
sqlite&gt; .once c:/work/dataout.csv
sqlite&gt; SELECT * FROM tab1;
sqlite&gt; .system c:/work/dataout.csv


In the example above, the ".headers on" line causes column labels to
be printed as the first row of output.  This means that the first row of
the resulting CSV file will contain column labels.  If column labels are
not desired, set ".headers off" instead. (The ".headers off" setting is
the default and can be omitted if the headers have not been previously
turned on.)

The line ".once FILENAME" causes all query output to go into
the named file instead of being printed on the console.  In the example
above, that line causes the CSV content to be written into a file named
"C:/work/dataout.csv".

The final line of the example (the ".system c:/work/dataout.csv")
has the same effect as double-clicking on the c:/work/dataout.csv file
in windows.  This will typically bring up a spreadsheet program to display
the CSV file.

That command only works as written on Windows.
The equivalent line on a Mac would be:

sqlite&gt; .system open dataout.csv


On Linux and other unix systems you will need to enter something like:


sqlite&gt; .system xdg-open dataout.csv




8.6.1.  Export to Excel 

To simplify export to a spreadsheet, the CLI provides the
".excel" command which captures the output of a single query and sends
that output to the default spreadsheet program on the host computer.
Use it like this:

sqlite&gt; .excel
sqlite&gt; SELECT * FROM tab;



The command above writes the output of the query as CSV into a temporary
file, invokes the default handler for CSV files (usually the preferred
spreadsheet program such as Excel or LibreOffice), then deletes the
temporary file.  This is essentially a short-hand method of doing
the sequence of ".csv", ".once", and ".system" commands described above.


The ".excel" command is really an alias for ".once -x".  The -x option
to .once causes it to writes results as CSV into a temporary file that
is named with a ".csv" suffix, then invoke the systems default handler
for CSV files.


There is also a ".once -e" command which works similarly, except that
it names the temporary file with a ".txt" suffix so that the default
text editor for the system will be invoked, instead of the default
spreadsheet.



8.6.2.  Export to TSV (tab separated values)


Exporting to pure TSV, without any field quoting, can be done by
entering ".mode tabs" before running a query. However, the output
will not be read correctly in tabs mode by the ".import" command
if it contains doublequote characters. To get TSV quoted per
RFC 4180 so that it can be input in tabs mode with ".import",
first enter ".mode&nbsp;csv", then enter '.separator&nbsp;"\t"'
before running a query.



9. Accessing ZIP Archives As Database Files

In addition to reading and writing SQLite database files,
the sqlite3 program will also read and write ZIP archives.
Simply specify a ZIP archive filename in place of an SQLite database
filename on the initial command line, or in the ".open" command,
and sqlite3 will automatically detect that the file is a
ZIP archive instead of an SQLite database and will open it as such.
This works regardless of file suffix.  So you can open JAR, DOCX,
and ODP files and any other file format that is really a ZIP
archive and SQLite will read it for you.

A ZIP archive appears to be a database containing a single table
with the following schema:

CREATE TABLE zip(
  name,     // Name of the file
  mode,     // Unix-style file permissions
  mtime,    // Timestamp, seconds since 1970
  sz,       // File size after decompression
  rawdata,  // Raw compressed file data
  data,     // Uncompressed file content
  method    // ZIP compression method code
);


So, for example, if you wanted to see the compression efficiency
(expressed as the size of the compressed content relative to the
original uncompressed file size) for all files in the ZIP archive,
sorted from most compressed to least compressed, you could run a
query like this:

sqlite&gt; SELECT name, (100.0*length(rawdata))/sz FROM zip ORDER BY 2;


Or using file I/O functions, you can extract elements of the
ZIP archive:

sqlite&gt; SELECT writefile(name,content) FROM zip
   ...&gt; WHERE name LIKE 'docProps/%';


9.1. How ZIP archive access is implemented

The command-line shell uses the Zipfile virtual table to
access ZIP archives.  You can see this by running the ".schema"
command when a ZIP archive is open:

sqlite&gt; .schema
CREATE VIRTUAL TABLE zip USING zipfile('document.docx')
/* zip(name,mode,mtime,sz,rawdata,data,method) */;


When opening a file, if the command-line client discovers that the
file is ZIP archive instead of an SQLite database, it actually opens
an in-memory database and then in that in-memory database it creates
an instance of the Zipfile virtual table that is attached to the
ZIP archive.

The special processing for opening ZIP archives is a trick of the
command-line shell, not the core SQLite library.  So if you want to
open a ZIP archive as a database in your application, you will need to
activate the Zipfile virtual table module then run an appropriate
CREATE VIRTUAL TABLE statement.




10. Converting An Entire Database To A Text File

Use the ".dump" command to convert the entire contents of a
database into a single UTF-8 text file.  This file can be converted
back into a database by piping it back into sqlite3.

A good way to make an archival copy of a database is this:


$ sqlite3 ex1 .dump | gzip -c &gt;ex1.dump.gz



This generates a file named ex1.dump.gz that contains everything
you need to reconstruct the database at a later time, or on another
machine.  To reconstruct the database, just type:


$ zcat ex1.dump.gz | sqlite3 ex2



The text format is pure SQL so you
can also use the .dump command to export an SQLite database
into other popular SQL database engines.  Like this:


$ createdb ex2
$ sqlite3 ex1 .dump | psql ex2




11. Recover Data From a Corrupted Database

Like the ".dump" command, ".recover" attempts to convert the entire
contents of a database file to text. The difference is that instead of
reading data using the normal SQL database interface, ".recover"
attempts to reassemble the database based on data extracted directly from
as many database pages as possible. If the database is corrupt, ".recover"
is usually able to recover data from all uncorrupted parts of the database,
whereas ".dump" stops when the first sign of corruption is encountered.

If the ".recover" command recovers one or more rows that it cannot
attribute to any database table, the output script creates a "lost_and_found"
table to store the orphaned rows. The schema of the lost_and_found
table is as follows:

CREATE TABLE lost_and_found(
    rootpgno INTEGER,             -- root page of tree pgno is a part of
    pgno INTEGER,                 -- page number row was found on
    nfield INTEGER,               -- number of fields in row
    id INTEGER,                   -- value of rowid field, or NULL
    c0, c1, c2, c3...             -- columns for fields of row
);


The "lost_and_found" table contains one row for each orphaned row recovered
from the database. Additionally, there is one row for each recovered index
entry that cannot be attributed to any SQL index. This is because, in an
SQLite database, the same format is used to store SQL index entries and
WITHOUT ROWID table entries.


  ColumnContents
  rootpgno Even though it may not be possible to attribute the
      row to a specific database table, it may be part of a tree structure
      within the database file. In this case, the root page number of that
      tree structure is stored in this column. Or, if the page the row was
      found on is not part of a tree structure, this column stores a copy of
      the value in column "pgno" - the page number of the page the row was
      found on. In many, although not all, cases, all rows in the
      lost_and_found table with the same value in this column belong to the
      same table.

  pgno The page number of the page on which this row was found.

  nfield The number of fields in this row.

  id If the row comes from a WITHOUT ROWID table, this column
      contains NULL. Otherwise, it contains the 64-bit integer rowid value for
      the row.

  c0,&nbsp;c1,&nbsp;c2... The values for each column of the row
      are stored in these columns. The ".recover" command creates the
      lost_and_found table with as many columns as required by the longest
      orphaned row.


If the recovered database schema already contains a table named
"lost_and_found", the ".recover" command uses the name "lost_and_found0". If
the name "lost_and_found0" is also already taken, "lost_and_found1", and so
on. The default name "lost_and_found" may be overridden by invoking ".recover"
with the --lost-and-found switch. For example, to have the output script call
the table "orphaned_rows":

sqlite&gt; .recover --lost-and-found orphaned_rows




12. Loading Extensions

You can add new custom application-defined SQL functions,
collating sequences, virtual tables, and VFSes to the command-line
shell at run-time using the ".load" command.  First, build the
extension as a DLL or shared library (as described in the
Run-Time Loadable Extensions document) then type:

sqlite&gt; .load /path/to/my_extension


Note that SQLite automatically adds the appropriate extension suffix
(".dll" on windows, ".dylib" on Mac, ".so" on most other unixes) to the
extension filename.  It is generally a good idea to specify the full
pathname of the extension.

SQLite computes the entry point for the extension based on the extension
filename.  To override this choice, simply add the name of the extension
as a second argument to the ".load" command.

Source code for several useful extensions can be found in the
ext/misc
subdirectory of the SQLite source tree.  You can use these extensions
as-is, or as a basis for creating your own custom extensions to address
your own particular needs.




13. Cryptographic Hashes Of Database Content

The ".sha3sum" dot-command computes a
SHA3 hash of the content
of the database.  To be clear, the hash is computed over the database content,
not its representation on disk.  This means, for example, that a VACUUM
or similar data-preserving transformation does not change the hash.

The ".sha3sum" command supports options "--sha3-224", "--sha3-256",
"--sha3-384", and "--sha3-512" to define which variety of SHA3 to use
for the hash.  The default is SHA3-256.

The database schema (in the sqlite_schema table) is not normally
included in the hash, but can be added by the "--schema" option.

The ".sha3sum" command takes a single optional argument which is a
LIKE pattern.  If this option is present, only tables whose names match
the LIKE pattern will be hashed.

The ".sha3sum" command is implemented with the help of the
extension function "sha3_query()"
that is included with the command-line shell.



14. Database Content Self-Tests

The ".selftest" command attempts to verify that a database is
intact and is not corrupt.
The .selftest command looks for a table in schema named "selftest"
and defined as follows:

CREATE TABLE selftest(
  tno INTEGER PRIMARY KEY,  -- Test number
  op TEXT,                  -- 'run' or 'memo'
  cmd TEXT,                 -- SQL command to run, or text of "memo"
  ans TEXT                  -- Expected result of the SQL command
);


The .selftest command reads the rows of the selftest table in
selftest.tno order.
For each 'memo' row, it writes the text in 'cmd' to the output.  For
each 'run' row, it runs the 'cmd' text as SQL and compares the result
to the value in 'ans', and shows an error message if the results differ.

If there is no selftest table, the ".selftest" command runs
PRAGMA integrity_check.

The ".selftest --init" command creates the selftest table if it
does not already exists, then appends entries that check the SHA3
hash of the content of all tables.  Subsequent runs of ".selftest"
will verify that the database has not been changed in any way.  To
generate tests to verify that a subset of the tables is unchanged,
simply run ".selftest --init" then DELETE the selftest rows that
refer to tables that are not constant.



15. SQLite Archive Support

The ".archive" dot-command and the "-A" command-line option
provide built-in support for the
SQLite Archive format. The interface is similar to
that of the "tar" command on unix systems. Each invocation of the ".ar"
command must specify a single command option. The following commands
are available for ".archive":


  OptionLong&nbsp;OptionPurpose
  -c--createCreate a new archive containing specified files.
  -x--extractExtract specified files from archive.
  -i--insertAdd files to existing archive.
  -r--removeRemove files from the archive.
  -t--listList the files in the archive.
  -u--updateAdd files to existing archive if they have changed.


As well as the command option, each invocation of ".ar" may specify
one or more modifier options. Some modifier options require an argument,
some do not. The following modifier options are available:


  OptionLong&nbsp;OptionPurpose
  -v--verboseList each file as it is processed.
  -f FILE--file FILEIf specified, use file FILE as the
  archive. Otherwise, assume that the current "main" database is the
  archive to be operated on.
  -a FILE--append FILELike --file, use file FILE as the
  archive, but open the file using the
  apndvfs VFS so that
  the archive will be appended to the end of FILE if FILE already exists.
  -C DIR--directory DIRIf specified, interpret all relative
  paths as relative to DIR, instead of the current working directory.
  -g--globUse glob(Y,X) to match arguments
  against names in the archive.
  -n--dryrunShow the SQL that would be run to carry out the
                       archive operation, but do not actually change anything.
  ----All subsequent command line words are command arguments,
  not options.



For command-line usage, add the short style command-line options immediately
following the "-A", without an intervening space.  All subsequent arguments
are considered to be part of the .archive command.  For example, the following
commands are equivalent:

sqlite3 new_archive.db -Acv file1 file2 file3
sqlite3 new_archive.db ".ar -cv file1 file2 file3"



Long and short style options may be mixed. For example, the following are
equivalent:

-- Two ways to create a new archive named "new_archive.db" containing
-- files "file1", "file2" and "file3".
.ar -c --file new_archive.db file1 file2 file3
.ar -f new_archive.db --create file1 file2 file3


Alternatively, the first argument following to ".ar" may be the concatenation
of the short form of all required options (without the "-" characters). In
this case arguments for options requiring them are read from the command line
next, and any remaining words are considered command arguments. For example:

-- Create a new archive "new_archive.db" containing files "file1" and
-- "file2" from directory "dir1".
.ar cCf dir1 new_archive.db file1 file2 file3


15.1.  SQLite Archive Create Command 

Create a new archive, overwriting any existing archive (either in the current
"main" db or in the file specified by a --file option). Each argument following
the options is a file to add to the archive. Directories are imported
recursively. See above for examples.

15.2.  SQLite Archive Extract Command 

Extract files from the archive (either to the current working directory or
to the directory specified by a --directory option).
Files or directories whose names match the arguments,
as affected by the --glob option, are extracted.
Or, if no arguments follow the options, all files and directories are extracted.
Any specified directories are extracted recursively. It is an error if any
specified names or match patterns cannot be found in the archive.

-- Extract all files from the archive in the current "main" db to the
-- current working directory. List files as they are extracted. 
.ar --extract --verbose

-- Extract file "file1" from archive "ar.db" to directory "dir1".
.ar fCx ar.db dir1 file1

-- Extract files with ".h" extension to directory "headers".
.ar -gCx headers *.h


15.3.  SQLite Archive List Command 

List the contents of the archive. If no arguments are specified, then all
files are listed. Otherwise, only those which match the arguments,
as affected by the --glob option, are listed. Currently,
the --verbose option does not change the behaviour of this command. That may
change in the future.

-- List contents of archive in current "main" db..
.ar --list




15.4.  SQLite Archive Insert And Update Commands 

 The --update and --insert commands work like --create command, except that
they do not delete the current archive before commencing. New versions of
files silently replace existing files with the same names, but otherwise
the initial contents of the archive (if any) remain intact.

 For the --insert command, all files listed are inserted into the archive.
For the --update command, files are only inserted if they do not previously
exist in the archive, or if their "mtime" or "mode" is different from what
is currently in the archive.

 Compatibility node:  Prior to SQLite version 3.28.0 (2019-04-16) only
the --update option was supported but that option worked like --insert in that
it always reinserted every file regardless of whether or not it had changed.

15.5.  SQLite Archive Remove Command 

 The --remove command deletes files and directories which match the
provided arguments (if any) as affected by the --glob option.
It is an error to provide arguments which match nothing in the archive.

15.6.  Operations On ZIP Archives 

If FILE is a ZIP archive rather than an SQLite Archive, the ".archive"
command and the "-A" command-line option still work.  This is accomplished
using of the zipfile extension.
Hence, the following commands are roughly equivalent,
differing only in output formatting:


  Traditional CommandEquivalent sqlite3.exe Command
  unzip archive.zipsqlite3 -Axf archive.zip
  unzip -l archive.zipsqlite3 -Atvf archive.zip
  zip -r archive2.zip dirsqlite3 -Acf archive2.zip dir


15.7.  SQL Used To Implement SQLite Archive Operations 

The various SQLite Archive Archive commands are implemented using SQL statements.
Application developers can easily add SQLite Archive Archive reading and writing
support to their own projects by running the appropriate SQL.

To see what SQL statements are used to implement an SQLite Archive
operation, add the --dryrun or -n option.  This causes the SQL to be
displayed but inhibits the execution of the SQL.

The SQL statements used to implement SQLite Archive operations make use of
various loadable extensions.  These extensions are all available in
the SQLite source tree in the
ext/misc/ subfolder.
The extensions needed for full SQLite Archive support include:



fileio.c &mdash;
This extension adds SQL functions readfile() and writefile() for
reading and writing content from files on disk.  The fileio.c
extension also includes fsdir() table-valued function for listing
the contents of a directory and the lsmode() function for converting
numeric st_mode integers from the stat() system call into human-readable
strings after the fashion of the "ls -l" command.


sqlar.c &mdash;
This extension adds the sqlar_compress() and sqlar_uncompress()
functions that are needed to compress and uncompress file content
as it is inserted and extracted from an SQLite Archive.


zipfile.c &mdash;
This extension implements the "zipfile(FILE)" table-valued function
which is used to read ZIP archives.  This extension is only needed
when reading ZIP archives instead of SQLite archives.


appendvfs.c &mdash;
This extension implements a new VFS that allows an SQLite database
to be appended to some other file, such as an executable.  This
extension is only needed if the --append option to the .archive
command is used.




16. SQL Parameters

SQLite allows bound parameters to appear in an SQL statement anywhere
that a literal value is allowed.  The values for these parameters are set
using the sqlite3_bind_...() family of APIs.

Parameters can be either named or unnamed.  An unnamed parameter is a single
question mark ("?").  Named parameters are a "?" followed immediately by a number
(ex: "?15" or "?123") or one of the characters "$", ":", or "@" followed by an
alphanumeric name (ex: "$var1", ":xyz", "@bingo").

This command-line shell leaves unnamed parameters unbound, meaning that they
will have a value of an SQL NULL, but named parameters might be assigned values.
If there exists a TEMP table named "sqlite_parameters" with a schema like this:

CREATE TEMP TABLE sqlite_parameters(
  key TEXT PRIMARY KEY,
  value
) WITHOUT ROWID;


And if there is an entry in that table where the key column exactly matches
the name of parameter (including the initial "?", "$", ":", or "@" character)
then the parameter is assigned the value of the value column.  If no entry exists,
the parameter defaults to NULL.

The ".parameter" command exists to simplify managing this table.  The
".parameter init" command (often abbreviated as just ".param init") creates
the temp.sqlite_parameters table if it does not already exist.  The ".param list"
command shows all entries in the temp.sqlite_parameters table.  The ".param clear"
command drops the temp.sqlite_parameters table.  The ".param set KEY VALUE" and
".param unset KEY" commands create or delete entries from the
temp.sqlite_parameters table.

The VALUE passed to ".param set KEY VALUE" can be either a SQL literal
or any other SQL expression or query which can be evaluated to yield a value.
This allows values of differing types to be set.
If such evaluation fails, the provided VALUE is instead quoted and inserted
as text.
Because such initial evaluation may or may not fail depending upon
the VALUE content, the reliable way to get a text value is to enclose it
with single-quotes protected from the above-described command-tail parsing.
For example, (unless one intends a value of -1365):
.parameter init
.parameter set @phoneNumber "'202-456-1111'"

Note that the double-quotes serve to protect the single-quotes
and ensure that the quoted text is parsed as one argument.

The temp.sqlite_parameters table only provides values for parameters in the
command-line shell.  The temp.sqlite_parameter table has no effect on queries
that are run directly using the SQLite C-language API.  Individual applications
are expected to implement their own parameter binding.  You can search for
"sqlite_parameters" in the
command-line shell source code
to see how the command-line shell does parameter binding, and use that as
a hint for how to implement it yourself.



17. Index Recommendations (SQLite Expert)

Note: This command is experimental. It may be removed or the
interface modified in incompatible ways at some point in the future.

For most non-trivial SQL databases, the key to performance is creating
the right SQL indexes. In this context "the right SQL indexes" means those
that cause the queries that an application needs to optimize run fast. The
".expert" command can assist with this by proposing indexes that might
assist with specific queries, were they present in the database.

The ".expert" command is issued first, followed by the SQL query
on a separate line. For example, consider the following session:

sqlite&gt; CREATE TABLE x1(a, b, c);                  -- Create table in database 
sqlite&gt; .expert
sqlite&gt; SELECT * FROM x1 WHERE a=? AND b&gt;?;        -- Analyze this SELECT 
CREATE INDEX x1_idx_000123a7 ON x1(a, b);

0|0|0|SEARCH TABLE x1 USING INDEX x1_idx_000123a7 (a=? AND b&gt;?)

sqlite&gt; CREATE INDEX x1ab ON x1(a, b);             -- Create the recommended index 
sqlite&gt; .expert
sqlite&gt; SELECT * FROM x1 WHERE a=? AND b&gt;?;        -- Re-analyze the same SELECT 
(no new indexes)

0|0|0|SEARCH TABLE x1 USING INDEX x1ab (a=? AND b&gt;?)


In the above, the user creates the database schema (a single table - "x1"),
and then uses the ".expert" command to analyze a query, in this case
"SELECT * FROM x1 WHERE a=? AND b&gt;?". The shell tool recommends that the
user create a new index (index "x1_idx_000123a7") and outputs the plan
that the query would use in EXPLAIN QUERY PLAN format. The user then creates
an index with an equivalent schema and runs the analysis on the same query
again. This time the shell tool does not recommend any new indexes, and
outputs the plan that SQLite will use for the query given the existing
indexes.

The ".expert" command accepts the following options:


 Option  Purpose
 &#8209;&#8209;verbose
     If present, output a more verbose report for each query analyzed.
 &#8209;&#8209;sample&nbsp;PERCENT
     This parameter defaults to 0, causing the ".expert" command to
         recommend indexes based on the query and database schema alone.
         This is similar to the way the SQLite query planner selects
         indexes for queries if the user has not run the ANALYZE command
         on the database to generate data distribution statistics.
         
         If this option is passed a non-zero argument, the ".expert" command
         generates similar data distribution statistics for all indexes
         considered based on PERCENT percent of the rows currently stored in
         each database table. For databases with unusual data distributions,
         this may lead to better index recommendations, particularly if the
         application intends to run ANALYZE.
         
         For small databases and modern CPUs, there is usually no reason not
         to pass "--sample 100". However, gathering data distribution
         statistics can be expensive for large database tables. If the
         operation is too slow, try passing a smaller value for the --sample
         option.


The functionality described in this section may be integrated into other
applications or tools using the

SQLite expert extension code.

A database schema which incorporate SQL custom functions made available
via the extension load mechanism may need special provision to work with
the .expert feature. Because the feature uses additional connections to
implement its functionality, those custom functions must be made available
to those additional connections. This can be done by means of the extension
load/usage options described at

Automatically Load Statically Linked Extensions
and 
Persistent Loadable Extensions.



18. Working With Multiple Database Connections


Beginning with version 3.37.0 (2021-11-27), the CLI has the ability to
hold multiple database connections open at once.  Only one database connection
is active at a time.  The inactive connections are still open but are idle.


Use the ".connection" dot-command (often abbreviated as just ".conn") to see a
list of database connections and an indication of which one is currently active.
Each database connection is identified by an integer between 0 and 9.  (There
can be at most 10 simultaneously open connections.)  Change to another database
connection, creating it if it does not already exist, by typing the ".conn"
command followed by its number.  Close a database connection by typing
".conn close N" where N is the connection number.


Though the underlying SQLite database connections are completely independent
of one another, many of the CLI settings, such as the output format, are
shared across all database connections.  Thus, changing the output mode in
one connection will change it in them all.  On the other hand, some
dot-commands such as .open only affect the current connection.



19. Miscellaneous Extension Features


The CLI is built with several SQLite extensions that are not
included with the SQLite library. A few add features
not described in the preceding sections, namely:

the UINT collating sequence which treats
unsigned integers embedded in text according to
their value, along with other text, for ordering;
decimal arithmetic as provided by the decimal extension;
the generate_series() table-valued function;




the base64() and base85() functions which encode a
blob to base64 or base85 text or decode the same to a blob; and
support for POSIX extended regular expressions
bound to the REGEXP operator.




20. Other Dot Commands

There are many other dot-commands available in the command-line
shell.  See the ".help" command for a complete list for any particular
version and build of SQLite.



21. Using sqlite3 in a shell script


One way to use sqlite3 in a shell script is to use "echo" or
"cat" to generate a sequence of commands in a file, then invoke sqlite3
while redirecting input from the generated command file.  This
works fine and is appropriate in many circumstances.  But as
an added convenience, sqlite3 allows a single SQL command to be
entered on the command line as a second argument after the
database name.  When the sqlite3 program is launched with two
arguments, the second argument is passed to the SQLite library
for processing, the query results are printed on standard output
in list mode, and the program exits.  This mechanism is designed
to make sqlite3 easy to use in conjunction with programs like
"awk".  For example:

$ sqlite3 ex1 'select * from tbl1' \
&gt;  | awk '{printf "&lt;tr&gt;&lt;td&gt;%s&lt;td&gt;%s\n",$1,$2 }'
&lt;tr&gt;&lt;td&gt;hello&lt;td&gt;10
&lt;tr&gt;&lt;td&gt;goodbye&lt;td&gt;20
$




22. Marking The End Of An SQL Statement


SQLite commands are normally terminated by a semicolon.  In the CLI
you can also use the word "GO" (case-insensitive) or a slash character
"/" on a line by itself to end a command.  These are used by SQL Server
and Oracle, respectively, and are supported by the SQLite CLI for
compatibility.  These won't work in sqlite3_exec(),
because the CLI translates these inputs into a semicolon before passing
them down into the SQLite core.



23. Command-line Options


There are many command-line options available to the CLI.  Use the --help
command-line option to see a list:

$ sqlite3 --help
Usage: ./sqlite3 &#91;OPTIONS&#93; FILENAME &#91;SQL&#93;
FILENAME is the name of an SQLite database. A new database is created
if the file does not previously exist.
OPTIONS include:
   -A ARGS...           run ".archive ARGS" and exit
   -append              append the database to the end of the file
   -ascii               set output mode to 'ascii'
   -bail                stop after hitting an error
   -batch               force batch I/O
   -box                 set output mode to 'box'
   -column              set output mode to 'column'
   -cmd COMMAND         run "COMMAND" before reading stdin
   -csv                 set output mode to 'csv'
   -deserialize         open the database using sqlite3_deserialize()
   -echo                print commands before execution
   -init FILENAME       read/process named file
   -&#91;no&#93;header          turn headers on or off
   -help                show this message
   -html                set output mode to HTML
   -interactive         force interactive I/O
   -json                set output mode to 'json'
   -line                set output mode to 'line'
   -list                set output mode to 'list'
   -lookaside SIZE N    use N entries of SZ bytes for lookaside memory
   -markdown            set output mode to 'markdown'
   -maxsize N           maximum size for a --deserialize database
   -memtrace            trace all memory allocations and deallocations
   -mmap N              default mmap size set to N
   -newline SEP         set output row separator. Default: '\n'
   -nofollow            refuse to open symbolic links to database files
   -nonce STRING        set the safe-mode escape nonce
   -nullvalue TEXT      set text string for NULL values. Default ''
   -pagecache SIZE N    use N slots of SZ bytes each for page cache memory
   -quote               set output mode to 'quote'
   -readonly            open the database read-only
   -safe                enable safe-mode
   -separator SEP       set output column separator. Default: '|'
   -stats               print memory stats before each finalize
   -table               set output mode to 'table'
   -tabs                set output mode to 'tabs'
   -no-utf8             do not setup for console I/O using UTF-8
   -version             show SQLite version
   -vfs NAME            use NAME as the default VFS
   -zip                 open the file as a ZIP Archive


The CLI is flexible regarding command-line option formatting.
Either one or two leading "-" characters are permitted.
Thus "-box" and "--box" mean the same thing.
Command-line options are processed from left to right.
Hence a "--box" option will override a prior "--quote" option.


Most of the command-line options are self-explanatory, but a few merit additional
discussion below.



23.1. The --safe command-line option

The --safe command-line option attempts to disable all features of the CLI that
might cause any changes to the host computer other than changes to the specific database
file named on the command-line.  The idea is that if you receive a large SQL script
from an unknown or untrusted source, you can run that script to see what it does without
risking an exploit by using the --safe option.  The --safe option disables (among other
things):


 The .open command, unless the --hexdb option is used or the filename is ":memory:".
     This prevents the script from reading or writing any database files not named on
     the original command-line.
 The ATTACH SQL command.
 SQL functions that have potentially harmful side-effects, such as
     edit(), fts3_tokenizer(), load_extension(), readfile() and writefile().
 The .archive command.
 The .backup and .save commands.
 The .import command.
 The .load command.
 The .log command.
 The .shell and .system commands.
 The .excel, .once and .output commands.
 Other commands that can have deleterious side effects.



Basically, any feature of the CLI that reads or writes from a file on disk other
than the main database file is disabled.

23.1.1. Bypassing --safe restrictions for specific commands

If the "--nonce NONCE" option is also included on the command-line, for some
large and arbitrary NONCE string, then the ".nonce NONCE" command (with the
same large nonce string) will permit the next SQL statement or dot-command
to bypass the --safe restrictions.

Suppose you want to run a suspicious script and the script requires one or
two of the features that --safe normally disables.  For example, suppose it
needs to ATTACH one additional database.  Or suppose the script needs to load
a specific extension. This can be accomplished by preceding the (carefully
audited) ATTACH statement or the ".load" command with an appropriate ".nonce"
command and supplying the same nonce value using the "--nonce" command-line
option.  Those specific commands will then be allowed to execute normally,
but all other unsafe commands will still be restricted.

The use of ".nonce" is dangerous in the sense that a mistake can allow a
hostile script to damage your system.  Therefore, use ".nonce" carefully,
sparingly, and as a last resort when there are no other ways to get a
script to run under --safe mode.



23.2. The --unsafe-testing command-line option

The --unsafe-testing command-line option supports use of the CLI for
internal testing of the SQLite library. It is not needed or useful for using
the CLI as a utility for creating, modifying or querying SQLite databases.
Its intended use is to permit scripted testing with direct schema changes,
defensive measures defeated, and certain special-purpose, undocumented,
test-oriented dot-commands enabled.

Misbehavior which requires use of the --unsafe-testing option to be induced
will generally not be considered a bug for that reason alone. CLI behavior
with --unsafe-testing is not supported or defined.

23.3. The --no-utf8 command-line option

On the Windows platform, when the console is used for input or output,
its input or output code page, or both are normally set for UTF-8 I/O
during the session, then restored afterward.
This allows copy and paste to transfer UTF-8
content properly and screen output to render properly.

This console UTF-8 I/O is not done if the CLI is invoked
with the --no-utf8 option or if the operating system
does not support UTF-8 console I/O. In either case,
I/O is translated between the CLI's internal UTF-8
encoding and the MBCS encoding in effect for the console.

In all cases, non-console text I/O is UTF-8 encoded.

On non-Windows platforms, this option is ignored and invisible.



24. Compiling the sqlite3 program from sources


To compile the command-line shell on unix systems and on Windows with MinGW,
the usual configure-make command works:

sh configure; make



The configure-make works whether you are building from the canonical sources
from the source tree, or from an amalgamated bundle.  There are few
dependencies.  When building from canonical sources, a working
tclsh is required.
If using an amalgamation bundle, all the preprocessing work normally
done by tclsh will have already been carried out and only normal build
tools are required.


A working zlib compression library is
needed in order for the .archive command to operate.


On Windows with MSVC, use nmake with the Makefile.msc:

nmake /f Makefile.msc



For correct operation of the .archive command, make a copy of the
zlib source code into the compat/zlib subdirectory
of the source tree and compile this way:

nmake /f Makefile.msc USE_ZLIB=1


24.1.  Do-It-Yourself Builds 


The source code to the sqlite3 command line interface is in a single
file named "shell.c".  The shell.c source file is generated from other
sources, but most of the code for shell.c can be found in
src/shell.c.in.
(Regenerate shell.c by typing "make shell.c" from the canonical source tree.)
Compile the shell.c file (together
with the sqlite3 library source code) to generate
the executable.  For example:

gcc -o sqlite3 shell.c sqlite3.c -ldl -lpthread -lz -lm



The following additional compile-time options are recommended in order to
provide a full-featured command-line shell:


 -DSQLITE_THREADSAFE=0
 -DSQLITE_ENABLE_EXPLAIN_COMMENTS
 -DSQLITE_HAVE_ZLIB
 -DSQLITE_INTROSPECTION_PRAGMAS
 -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION
 -DSQLITE_ENABLE_STMTVTAB
 -DSQLITE_ENABLE_DBPAGE_VTAB
 -DSQLITE_ENABLE_DBSTAT_VTAB
 -DSQLITE_ENABLE_OFFSET_SQL_FUNC
 -DSQLITE_ENABLE_JSON1
 -DSQLITE_ENABLE_RTREE
 -DSQLITE_ENABLE_FTS4
 -DSQLITE_ENABLE_FTS5

This page last modified on  2023-10-26 19:49:47 UTC


c2: About SQLite
c5: about.html
c4:
About SQLite


Executive Summary

 Full-featured SQL
 Billions and billions of deployments
 Single-file database
 Public domain source code
 All source code in one file (sqlite3.c)
 Small footprint
 Max DB size: 281 terabytes
     (248 bytes)
 Max row size: 1 gigabyte
 Faster than direct file I/O
 Aviation-grade quality and testing
 Zero-configuration
 ACID transactions, even after power loss
 Stable, enduring file format
 Extensive, detailed documentation
 Long-term support


SQLite is an in-process library that implements a
self-contained, 
serverless,
zero-configuration,
transactional
SQL database engine.
The code for SQLite is in the
public domain and is thus free for
use for any purpose, commercial or private.
SQLite is the most widely deployed database in the world
with more applications than we can
count, including several high-profile projects.

SQLite is an embedded SQL database engine.
Unlike most other SQL databases, SQLite does not have a separate
server process.  SQLite reads and writes directly to ordinary disk
files.  A complete SQL database with multiple tables, indices,
triggers, and views, is contained in a single disk file.
The database file format is cross-platform - you can freely copy a database
between 32-bit and 64-bit systems or between 
big-endian and
little-endian
architectures.  These features make SQLite a popular choice as
an Application File Format.  SQLite database files are a
recommended storage format by the US Library of Congress.
Think of SQLite not as a replacement for 
Oracle but
as a replacement for fopen()

SQLite is a compact library.
With all features enabled, the library size can be less than 750KiB,
depending on the target platform and compiler optimization settings.
(64-bit code is larger.  And some compiler optimizations
such as aggressive function inlining and loop unrolling can cause the
object code to be much larger.)
There is a tradeoff between memory usage and speed.  
SQLite generally runs faster the more memory
you give it.  Nevertheless, performance is usually quite good even
in low-memory environments.  Depending on how it is used, SQLite can be
faster than direct filesystem I/O.

SQLite is 
very carefully tested prior to every
release and has a reputation for being very reliable.
Most of the SQLite source code is devoted purely to testing and
verification.  An automated test suite runs millions and millions of
test cases involving hundreds of millions of individual SQL statements
and achieves 100% branch test coverage.  
SQLite responds gracefully to memory
allocation failures and disk I/O errors.  Transactions are
ACID
even if interrupted by system crashes or power failures.  
All of this is verified by
the automated tests using special test harnesses which simulate 
system failures.
Of course, even with all this testing, there are still bugs.
But unlike some similar projects (especially commercial competitors)
SQLite is open and honest about all bugs and provides
bugs lists and
minute-by-minute 
chronologies of code changes.

The SQLite code base is supported by an
international team of developers who work on
SQLite full-time.
The developers continue to expand the capabilities of SQLite
and enhance its reliability and performance while maintaining
backwards compatibility with the 
published interface spec,
SQL syntax, and database file format.
The source code is absolutely free to anybody who wants it,
but professional support is also available.

The SQLite project was started on 
2000-05-09.
The future is always hard to predict, but the intent of the developers
is to support SQLite through the year 2050.  Design decisions are made
with that objective in mind.

We the developers hope that you find SQLite useful and we
entreat you to use it well: to make good and beautiful products that
are fast, reliable, and simple to use.  Seek forgiveness for yourself
as you forgive others.  And just as you have received SQLite for free,
so also freely give, paying the debt forward.


c2: Javascript Required
c5: hp1.html
c4:
Note:
The hyperlinks on the download page only work if you have Javascript
enabled in your web browser.
This page last modified on  2022-01-08 05:02:57 UTC


c2: 35% Faster Than The Filesystem
c5: fasterthanfs.html
c4:
1. Summary

SQLite reads and writes small blobs (for example, thumbnail images)
35% faster&sup1; than the same blobs
can be read from or written to individual files on disk using
fread() or fwrite().

Furthermore, a single SQLite database holding
10-kilobyte blobs uses about 20% less disk space than
storing the blobs in individual files.

The performance difference arises (we believe) because when
working from an SQLite database, the open() and close() system calls
are invoked only once, whereas
open() and close() are invoked once for each blob
when using blobs stored in individual files.  It appears that the
overhead of calling open() and close() is greater than the overhead
of using the database.  The size reduction arises from the fact that
individual files are padded out to the next multiple of the filesystem
block size, whereas the blobs are packed more tightly into an SQLite
database.


The measurements in this article were made during the week of 2017-06-05
using a version of SQLite in between 3.19.2 and 3.20.0.  You may expect
future versions of SQLite to perform even better.

1.1. Caveats



&sup1;The 35% figure above is approximate.  Actual timings vary
depending on hardware, operating system, and the
details of the experiment, and due to random performance fluctuations
on real-world hardware.  See the text below for more detail.
Try the experiments yourself.  Report significant deviations on
the SQLite forum.



The 35% figure is based on running tests on every machine
that the author has easily at hand.
Some reviewers of this article report that SQLite has higher 
latency than direct I/O on their systems.  We do not yet understand
the difference.  We also see indications that SQLite does not
perform as well as direct I/O when experiments are run using
a cold filesystem cache.


So let your take-away be this: read/write latency for
SQLite is competitive with read/write latency of individual files on
disk.  Often SQLite is faster.  Sometimes SQLite is almost
as fast.  Either way, this article disproves the common
assumption that a relational database must be slower than direct
filesystem I/O.

1.2. Related Studies


A 2022 study
(alternative link on GitHub) found that
SQLite is roughly twice as fast at real-world workloads compared to Btrfs and Ext4 on Linux.


Jim Gray
and others studied the read performance of BLOBs
versus file I/O for Microsoft SQL Server and found that reading BLOBs 
out of the 
database was faster for BLOB sizes less than between 250KiB and 1MiB.
(Paper).
In that study, the database still stores the filename of the content even
if the content is held in a separate file.  So the database is consulted
for every BLOB, even if it is only to extract the filename.  In this
article, the key for the BLOB is the filename, so no preliminary database
access is required.  Because the database is never used at all when
reading content from individual files in this article, the threshold
at which direct file I/O becomes faster is smaller than it is in Gray's
paper.


The Internal Versus External BLOBs article on this website is an
earlier investigation (circa 2011) that uses the same approach as the
Jim Gray paper &mdash; storing the blob filenames as entries in the
database &mdash; but for SQLite instead of SQL Server.



2. How These Measurements Are Made

I/O performance is measured using the
kvtest.c program
from the SQLite source tree.
To compile this test program, first gather the kvtest.c source file
into a directory with the SQLite amalgamation source
files "sqlite3.c" and "sqlite3.h".  Then on unix, run a command like
the following:

gcc -Os -I. -DSQLITE_DIRECT_OVERFLOW_READ &#92;
  kvtest.c sqlite3.c -o kvtest -ldl -lpthread


Or on Windows with MSVC:

cl -I. -DSQLITE_DIRECT_OVERFLOW_READ kvtest.c sqlite3.c


Instructions for compiling for Android
are shown below.


Use the resulting "kvtest" program to
generate a test database with 100,000 random uncompressible
blobs, each with a random
size between 8,000 and 12,000 bytes
using a command like this:

./kvtest init test1.db --count 100k --size 10k --variance 2k



If desired, you can verify the new database by running this command:

./kvtest stat test1.db



Next, make copies of all the blobs into individual files in a directory
using a command like this:

./kvtest export test1.db test1.dir



At this point, you can measure the amount of disk space used by
the test1.db database and the space used by the test1.dir directory
and all of its content.  On a standard Ubuntu Linux desktop, the
database file will be 1,024,512,000 bytes in size and the test1.dir
directory will use 1,228,800,000 bytes of space (according to "du -k"),
about 20% more than the database.


The "test1.dir" directory created above puts all the blobs into a single
folder.  It was conjectured that some operating systems would perform 
poorly when a single directory contains 100,000 objects.  To test this,
the kvtest program can also store the blobs in a hierarchy of folders with no
more than 100 files and/or subdirectories per folder.  The alternative
on-disk representation of the blobs can be created using the --tree
command-line option to the "export" command, like this:

./kvtest export test1.db test1.tree --tree



The test1.dir directory will contain 100,000 files
with names like "000000", "000001", "000002" and so forth but the
test1.tree directory will contain the same files in subdirectories like
"00/00/00", "00/00/01", and so on.  The test1.dir and test1.test
directories take up approximately the same amount of space, though
test1.test is very slightly larger due to the extra directory entries.


All of the experiments that follow operate the same with either 
"test1.dir" or "test1.tree".  Very little performance difference is
measured in either case, regardless of operating system.


Measure the performance for reading blobs from the database and from
individual files using these commands:

./kvtest run test1.db --count 100k --blob-api
./kvtest run test1.dir --count 100k --blob-api
./kvtest run test1.tree --count 100k --blob-api



Depending on your hardware and operating system, you should see that reads 
from the test1.db database file are about 35% faster than reads from 
individual files in the test1.dir or test1.tree folders.  Results can vary
significantly from one run to the next due to caching, so it is advisable
to run tests multiple times and take an average or a worst case or a best
case, depending on your requirements.

The --blob-api option on the database read test causes kvtest to use
the sqlite3_blob_read() feature of SQLite to load the content of the
blobs, rather than running pure SQL statements.  This helps SQLite to run
a little faster on read tests.  You can omit that option to compare the
performance of SQLite running SQL statements.
In that case, the SQLite still out-performs direct reads, though
by not as much as when using sqlite3_blob_read().
The --blob-api option is ignored for tests that read from individual disk
files.


Measure write performance by adding the --update option.  This causes
the blobs are overwritten in place with another random blob of
exactly the same size.

./kvtest run test1.db --count 100k --update
./kvtest run test1.dir --count 100k --update
./kvtest run test1.tree --count 100k --update



The writing test above is not completely fair, since SQLite is doing
power-safe transactions whereas the direct-to-disk writing is not.
To put the tests on a more equal footing, add either the --nosync
option to the SQLite writes to disable calling fsync() or
FlushFileBuffers() to force content to disk, or using the --fsync option
for the direct-to-disk tests to force them to invoke fsync() or
FlushFileBuffers() when updating disk files.


By default, kvtest runs the database I/O measurements all within
a single transaction.  Use the --multitrans option to run each blob
read or write in a separate transaction.  The --multitrans option makes
SQLite much slower, and uncompetitive with direct disk I/O.  This
option proves, yet again, that to get the most performance out of
SQLite, you should group as much database interaction as possible within
a single transaction.


There are many other testing options, which can be seen by running
the command:

./kvtest help


2.1. Read Performance Measurements

The chart below shows data collected using 
kvtest.c on five different
systems:


Win7: A circa-2009 Dell Inspiron laptop, Pentium dual-core
    at 2.30GHz, 4GiB RAM, Windows7.
Win10: A 2016 Lenovo YOGA 910, Intel i7-7500 at 2.70GHz,
    16GiB RAM, Windows10.
Mac: A 2015 MacBook Pro, 3.1GHz intel Core i7, 16GiB RAM,
    MacOS 10.12.5
Ubuntu: Desktop built from Intel i7-4770K at 3.50GHz, 32GiB RAM,
    Ubuntu 16.04.2 LTS
Android: Galaxy S3, ARMv7, 2GiB RAM


All machines use SSD except Win7 which has a
hard-drive. The test database is 100K blobs with sizes uniformly
distributed between 8K and 12K, for a total of about 1 gigabyte
of content.  The database page size
is 4KiB.  The -DSQLITE_DIRECT_OVERFLOW_READ compile-time option was
used for all of these tests.
Tests were run multiple times.
The first run was used to warm up the cache and its timings were discarded.


The chart below shows average time to read a blob directly from the
filesystem versus the time needed to read the same blob from the SQLite 
database.
The actual timings vary considerably from one system to another 
(the Ubuntu desktop is much
faster than the Galaxy S3 phone, for example).  
This chart shows the ratio of the
times needed to read blobs from a file divided by the time needed to
from the database.  The left-most column in the chart is the normalized
time to read from the database, for reference.


In this chart, an SQL statement ("SELECT v FROM kv WHERE k=?1") 
is prepared once.  Then for each blob, the blob key value is bound 
to the ?1 parameter and the statement is evaluated to extract the
blob content.


The chart shows that on Windows10, content can be read from the SQLite
database about 5 times faster than it can be read directly from disk.
On Android, SQLite is only about 35% faster than reading from disk.






Chart 1:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg 10KB each, random order using SQL



The performance can be improved slightly by bypassing the SQL layer
and reading the blob content directly using the
sqlite3_blob_read() interface, as shown in the next chart:






Chart 2:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg size 10KB, random order
using sqlite3_blob_read().



Further performance improves can be made by using the
memory-mapped I/O feature of SQLite.  In the next chart, the
entire 1GB database file is memory mapped and blobs are read
(in random order) using the sqlite3_blob_read() interface.
With these optimizations, SQLite is twice as fast as Android
or MacOS-X and over 10 times faster than Windows.






Chart 3:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg size 10KB, random order
using sqlite3_blob_read() from a memory-mapped database.



The third chart shows that reading blob content out of SQLite can be
twice as fast as reading from individual files on disk for Mac and
Android, and an amazing ten times faster for Windows.

2.2. Write Performance Measurements


Writes are slower.
On all systems, using both direct I/O and SQLite, write performance is
between 5 and 15 times slower than reads.


Write performance measurements were made by replacing (overwriting)
an entire blob with a different blob.  All of the blobs in these
experiment are random and incompressible.  Because writes are so much
slower than reads, only 10,000 of the 100,000 blobs in the database
are replaced.  The blobs to be replaced are selected at random and
are in no particular order.


The direct-to-disk writes are accomplished using fopen()/fwrite()/fclose().
By default, and in all the results shown below, the OS filesystem buffers are
never flushed to persistent storage using fsync() or
FlushFileBuffers().  In other words, there is no attempt to make the
direct-to-disk writes transactional or power-safe.
We found that invoking fsync() or FlushFileBuffers() on each file
written causes direct-to-disk storage
to be about 10 times or more slower than writes to SQLite.


The next chart compares SQLite database updates in WAL mode
against raw direct-to-disk overwrites of separate files on disk.
The PRAGMA synchronous setting is NORMAL.
All database writes are in a single transaction.
The timer for the database writes is stopped after the transaction
commits, but before a checkpoint is run.
Note that the SQLite writes, unlike the direct-to-disk writes,
are transactional and power-safe, though because the synchronous
setting is NORMAL instead of FULL, the transactions are not durable.






Chart 4:  SQLite write latency relative to direct filesystem writes.
10K blobs, avg size 10KB, random order,
WAL mode with synchronous NORMAL,
exclusive of checkpoint time



The android performance numbers for the write experiments are omitted
because the performance tests on the Galaxy S3 are so random.  Two
consecutive runs of the exact same experiment would give wildly different
times.  And, to be fair, the performance of SQLite on android is slightly
slower than writing directly to disk.


The next chart shows the performance of SQLite versus direct-to-disk
when transactions are disabled (PRAGMA journal_mode=OFF)
and PRAGMA synchronous is set to OFF.  These settings put SQLite on an
equal footing with direct-to-disk writes, which is to say they make the
data prone to corruption due to system crashes and power failures.






Chart 5:  SQLite write latency relative to direct filesystem writes.
10K blobs, avg size 10KB, random order,
journaling disabled, synchronous OFF.



In all of the write tests, it is important to disable anti-virus software
prior to running the direct-to-disk performance tests.  We found that
anti-virus software slows down direct-to-disk by an order of magnitude
whereas it impacts SQLite writes very little.  This is probably due to the
fact that direct-to-disk changes thousands of separate files which all need
to be checked by anti-virus, whereas SQLite writes only changes the single
database file.

2.3. Variations

The -DSQLITE_DIRECT_OVERFLOW_READ compile-time option causes SQLite
to bypass its page cache when reading content from overflow pages.  This
helps database reads of 10K blobs run a little faster, but not all that much
faster.  SQLite still holds a speed advantage over direct filesystem reads
without the SQLITE_DIRECT_OVERFLOW_READ compile-time option.

Other compile-time options such as using -O3 instead of -Os or
using -DSQLITE_THREADSAFE=0 and/or some of the other
recommended compile-time options might help SQLite to run even faster
relative to direct filesystem reads.

The size of the blobs in the test data affects performance.
The filesystem will generally be faster for larger blobs, since
the overhead of open() and close() is amortized over more bytes of I/O,
whereas the database will be more efficient in both speed and space
as the average blob size decreases.


3. General Findings



SQLite is competitive with, and usually faster than, blobs stored in
separate files on disk, for both reading and writing.


SQLite is much faster than direct writes to disk on Windows
when anti-virus protection is turned on.  Since anti-virus software
is and should be on by default in Windows, that means that SQLite
is generally much faster than direct disk writes on Windows.


Reading is about an order of magnitude faster than writing, for all
systems and for both SQLite and direct-to-disk I/O.


I/O performance varies widely depending on operating system and hardware.
Make your own measurements before drawing conclusions.


Some other SQL database engines advise developers to store blobs in separate
files and then store the filename in the database.  In that case, where
the database must first be consulted to find the filename before opening
and reading the file, simply storing the entire blob in the database
gives much faster read and write performance with SQLite.
See the Internal Versus External BLOBs article for more information.



4. Additional Notes


4.1. Compiling And Testing on Android


The kvtest program is compiled and run on Android as follows.
First install the Android SDK and NDK.  Then prepare a script
named "android-gcc" that looks approximately like this:

#!/bin/sh
#
NDK=/home/drh/Android/Sdk/ndk-bundle
SYSROOT=$NDK/platforms/android-16/arch-arm
ABIN=$NDK/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin
GCC=$ABIN/arm-linux-androideabi-gcc
$GCC --sysroot=$SYSROOT -fPIC -pie $*


Make that script executable and put it on your $PATH.  Then
compile the kvtest program as follows:

android-gcc -Os -I. kvtest.c sqlite3.c -o kvtest-android


Next, move the resulting kvtest-android executable to the Android
device:

adb push kvtest-android /data/local/tmp


Finally use "adb shell" to get a shell prompt on the Android device,
cd into the /data/local/tmp directory, and begin running the tests
as with any other unix host.
This page last modified on  2023-10-28 11:35:47 UTC


c2: Dynamic Memory Allocation In SQLite
c5: malloc.html
c4:
Overview 

SQLite uses dynamic memory allocation to obtain
memory for storing various objects
(ex: database connections and prepared statements) and to build
a memory cache of the database file and to hold the results of queries.
Much effort has gone into making the dynamic memory allocation subsystem
of SQLite reliable, predictable, robust, secure, and efficient.

This document provides an overview of dynamic memory allocation within 
SQLite.  The target audience is software engineers who are tuning their
use of SQLite for peak performance in demanding environments.
Nothing in this document is required knowledge for using SQLite.  The
default settings and configuration for SQLite will work well in most
applications.  However, the information contained in this document may
be useful to engineers who are tuning SQLite to comply with special
requirements or to run under unusual circumstances.


1.  Features

The SQLite core and its memory allocation subsystem provides the 
following capabilities:



Robust against allocation failures.
If a memory allocation ever fails (that is to say, 
if malloc() or realloc() ever return NULL)
then SQLite will recover gracefully.   SQLite will first attempt
to free memory from unpinned cache pages then retry the allocation
request.  
Failing that, SQLite will either stop what
it is doing and return the
SQLITE_NOMEM error code back up to the application or it will
make do without the requested memory.



No memory leaks.
The application is responsible for destroying any objects it allocates.
(For example, the application must use sqlite3_finalize() on 
every prepared statement and sqlite3_close() on every 
database connection.)   But as long as
the application cooperates, SQLite will never leak memory.  This is
true even in the face of memory allocation failures or other system
errors.



Memory usage limits.
The sqlite3_soft_heap_limit64() mechanism allows the application to
set a memory usage limit that SQLite strives to stay below.  SQLite
will attempt to reuse memory from its caches rather than allocating new
memory as it approaches the soft limit.



Zero-malloc option.
The application can optionally provide SQLite with several buffers of bulk memory
at startup and SQLite will then use those provided buffers for all of
its memory allocation needs and never call system malloc() or free().



Application-supplied memory allocators.
The application can provide SQLite with pointers to alternative 
memory allocators at start-time.  The alternative memory allocator
will be used in place of system malloc() and free().



Proof against breakdown and fragmentation.
SQLite can be configured so that, subject to certain usage constraints
detailed below, it is guaranteed to never fail a memory allocation
or fragment the heap.
This property is important to long-running, high-reliability
embedded systems where a memory allocation error could contribute
to an overall system failure.



Memory usage statistics.
Applications can see how much memory they are using and detect when
memory usage is approaching or exceeding design boundaries.




Plays well with memory debuggers.
Memory allocation in SQLite is structured so that standard
third-party memory debuggers (such as dmalloc or 
valgrind) can be used to verify correct
memory allocation behavior.


Minimal calls to the allocator.
The system malloc() and free() implementations are inefficient
on many systems.  SQLite strives to reduce overall processing time
by minimizing its use of malloc() and free().



Open access.
Pluggable SQLite extensions or even the application itself can 
access to the same underlying memory allocation
routines used by SQLite through the
sqlite3_malloc(), sqlite3_realloc(), and sqlite3_free() interfaces.





2.  Testing

Most of the code in the SQLite source tree is devoted purely to 
testing and verification.  Reliability is important to SQLite.
Among the tasks of the test infrastructure is to ensure that
SQLite does not misuse dynamically allocated memory, that SQLite
does not leak memory, and that SQLite responds
correctly to a dynamic memory allocation failure.

The test infrastructure verifies that SQLite does not misuse
dynamically allocated memory by using a specially instrumented
memory allocator.  The instrumented memory allocator is enabled
at compile-time using the SQLITE_MEMDEBUG option.  The instrumented
memory allocator is much slower than the default memory allocator and
so its use is not recommended in production.  But when
enabled during testing, 
the instrumented memory allocator performs the following checks:


Bounds checking.
The instrumented memory allocator places sentinel values at both ends
of each memory allocation to verify that nothing within SQLite writes
outside the bounds of the allocation.

Use of memory after freeing.
When each block of memory is freed, every byte is overwritten with a
nonsense bit pattern.  This helps to ensure that no memory is ever
used after having been freed.

Freeing memory not obtained from malloc.
Each memory allocation from the instrumented memory allocator contains
sentinels used to verify that every allocation freed came
from prior malloc.

Uninitialized memory.
The instrumented memory allocator initializes each memory allocation
to a nonsense bit pattern to help ensure that the user makes no
assumptions about the content of allocation memory.


Regardless of whether or not the instrumented memory allocator is
used, SQLite keeps track of how much memory is currently checked out.
There are hundreds of test scripts used for testing SQLite.  At the
end of each script, all objects are destroyed and a test is made to
ensure that all  memory has been freed.  This is how memory
leaks are detected.  Notice that memory leak detection is in force at
all times, during test builds and during production builds.  Whenever
one of the developers runs any individual test script, memory leak
detection is active.  Hence memory leaks that do arise during development
are quickly detected and fixed.


The response of SQLite to out-of-memory (OOM) errors is tested using
a specialized memory allocator overlay that can simulate memory failures.
The overlay is a layer that is inserted in between the memory allocator
and the rest of SQLite.  The overlay passes most memory allocation
requests straight through to the underlying allocator and passes the
results back up to the requester.  But the overlay can be set to 
cause the Nth memory allocation to fail.  To run an OOM test, the overlay
is first set to fail on the first allocation attempt.  Then some test
script is run and verification that the allocation was correctly caught
and handled is made.  Then the overlay is set to fail on the second
allocation and the test repeats.  The failure point continues to advance
one allocation at a time until the entire test procedure runs to
completion without hitting a memory allocation error.  This whole
test sequence run twice.  On the first pass, the
overlay is set to fail only the Nth allocation.  On the second pass,
the overlay is set to fail the Nth and all subsequent allocations.

Note that the memory leak detection logic continues to work even
when the OOM overlay is being used.  This verifies that SQLite
does not leak memory even when it encounters memory allocation errors.
Note also that the OOM overlay can work with any underlying memory
allocator, including the instrumented memory allocator that checks
for memory allocation misuse.  In this way it is verified that 
OOM errors do not induce other kinds of memory usage errors.

Finally, we observe that the instrumented memory allocator and the
memory leak detector both work over the entire SQLite test suite and
the TCL test suite provides over 99% statement test coverage and that
the TH3 test harness provides 100% branch test coverage
with no leak leaks. This is
strong evidence that dynamic memory allocation is used correctly
everywhere within SQLite.


2.1.  Use of reallocarray()

The reallocarray() interface is a recent innovation (circa 2014)
from the OpenBSD community that grow out of efforts to prevent the
next "heartbleed" bug by avoiding 32-bit integer
arithmetic overflow on memory allocation size computations.  The
reallocarray() function has both unit-size and count parameters.
To allocate memory sufficient to hold an array of N elements each X-bytes
in size, one calls "reallocarray(0,X,N)".  This is preferred over
the traditional technique of invoking "malloc(X*N)" as reallocarray()
eliminates the risk that the X*N multiplication will overflow and
cause malloc() to return a buffer that is a different size from what
the application expected.

SQLite does not use reallocarray().  The reason is that reallocarray()
is not useful to SQLite.  It turns out that SQLite never does memory
allocations that are the simple product of two integers.  Instead, SQLite
does allocations of the form "X+C" or "N*X+C" or "M*N*X+C" or
"N*X+M*Y+C", and so forth.  The reallocarray() interface is not helpful
in avoiding integer overflow in those cases.

Nevertheless, integer overflow in the computation of memory allocation
sizes is a concern that SQLite would like to deal with.  To prevent
problems, all SQLite internal memory allocations occur using thin wrapper
functions that take a signed 64-bit integer size parameter.  The SQLite 
source code is audited to ensure that all size computations are carried 
out using 64-bit signed integers as well.  SQLite will
refuse to allocate more than about 2GB of memory at one go.  (In common
use, SQLite seldom ever allocates more than about 8KB of memory at a time
so a 2GB allocation limit is not a burden.)  So the 64-bit size parameter
provides lots of headroom for detecting overflows.  The same audit that
verifies that all size computations are done as 64-bit signed integers
also verifies that it is impossible to overflow a 64-bit integer
during the computation.

The code audits used to ensure that memory allocation size computations
do not overflow in SQLite are repeated prior to every SQLite release.


3.  Configuration

The default memory allocation settings in SQLite are appropriate
for most applications.  However, applications with unusual or particularly
strict requirements may want to adjust the configuration to more closely
align SQLite to their needs.
Both compile-time and start-time configuration options are available.



3.1.  Alternative low-level memory allocators

The SQLite source code includes several different memory allocation
modules that can be selected at compile-time, or to a limited extent
at start-time.



3.1.1. The default memory allocator

By default, SQLite uses the malloc(), realloc(), and free() routines
from the standard C library for its memory allocation needs.  These routines
are surrounded by a thin wrapper that also provides a "memsize()" function
that will return the size of an existing allocation.  The memsize() function
is needed to keep an accurate count of the number of bytes of outstanding
memory; memsize() determines how many bytes to remove from the outstanding
count when an allocation is freed.  The default allocator implements
memsize() by always allocating 8 extra bytes on each malloc() request and
storing the size of the allocation in that 8-byte header.

The default memory allocator is recommended for most applications.
If you do not have a compelling need to use an alternative memory
allocator, then use the default.



3.1.2. The debugging memory allocator

If SQLite is compiled with the SQLITE_MEMDEBUG compile-time option,
then a different, heavy wrapper is used around system malloc(), realloc(), 
and free().
The heavy wrapper allocates around 100 bytes of extra space
with each allocation.  The extra space is used to place sentinel values 
at both ends of the allocation returned to the SQLite core.  When an
allocation is freed,
these sentinels are checked to make sure the SQLite core did not overrun
the buffer in either direction.  When the system library is GLIBC, the 
heavy wrapper also makes use of the GNU backtrace() function to examine
the stack and record the ancestor functions of the malloc() call.  When
running the SQLite test suite, the heavy wrapper also records the name of
the current test case.  These latter two features are useful for
tracking down the source of memory leaks detected by the test suite.

The heavy wrapper that is used when SQLITE_MEMDEBUG is set also
makes sure each new allocation is filled with nonsense data prior to
returning the allocation to the caller.  And as soon as an allocation
is free, it is again filled with nonsense data.  These two actions help
to ensure that the SQLite core does not make assumptions about the state
of newly allocated memory and that memory allocations are not used after
they have been freed.

The heavy wrapper employed by SQLITE_MEMDEBUG is intended for use
only during testing, analysis, and debugging of SQLite.  The heavy wrapper
has a significant performance and memory overhead and probably should not
be used in production.



3.1.3. The Win32 native memory allocator

If SQLite is compiled for Windows with the SQLITE_WIN32_MALLOC
compile-time option, then a different, thin wrapper is used around
HeapAlloc(), HeapReAlloc(), and HeapFree().  The thin wrapper uses the
configured SQLite heap, which will be different from the default process
heap if the SQLITE_WIN32_HEAP_CREATE compile-time option is used.  In
addition, when an allocation is made or freed, HeapValidate() will be
called if SQLite is compiled with assert() enabled and the
SQLITE_WIN32_MALLOC_VALIDATE compile-time option.



3.1.4. Zero-malloc memory allocator

When SQLite is compiled with the SQLITE_ENABLE_MEMSYS5 option, an
alternative memory allocator that does not use malloc() is included in the
build.  The SQLite developers refer to this alternative memory allocator
as "memsys5".  Even when it is included in the build, memsys5 is 
disabled by default.
To enable memsys5, the application must invoke the following SQLite 
interface at start-time:


sqlite3_config(SQLITE_CONFIG_HEAP, pBuf, szBuf, mnReq);


In the call above, pBuf is a pointer to a large, contiguous chunk
of memory space that SQLite will use to satisfy all of its memory
allocation needs.   pBuf might point to a static array or it might
be memory obtained from some other application-specific mechanism.
szBuf is an integer that is the number of bytes of memory space
pointed to by pBuf.  mnReq is another integer that is the
minimum size of an allocation.  Any call to sqlite3_malloc(N) where
N is less than mnReq will be rounded up to mnReq.  mnReq must be
a power of two.  We shall see later that the mnReq parameter is
important in reducing the value of n and hence the minimum memory
size requirement in the Robson proof.

The memsys5 allocator is designed for use on embedded systems, 
though there is nothing to prevent its use on workstations.
The szBuf is typically between a few hundred kilobytes up to a few
dozen megabytes, depending on system requirements and memory budget.

The algorithm used by memsys5 can be called "power-of-two,
first-fit".  The sizes of all memory allocation 
requests are rounded up to a power of two and the request is satisfied
by the first free slot in pBuf that is large enough.  Adjacent freed
allocations are coalesced using a buddy system. When used appropriately,
this algorithm provides mathematical guarantees against fragmentation and
breakdown, as described further below.



3.1.5. Experimental memory allocators

The name "memsys5" used for the zero-malloc memory allocator implies
that there are several additional memory allocators available, and indeed
there are.  The default memory allocator is "memsys1".  The debugging
memory allocator is "memsys2".  Those have already been covered.

If SQLite is compiled with SQLITE_ENABLE_MEMSYS3 then another
zero-malloc memory allocator, similar to memsys5, is included in the
source tree.  The memsys3 allocator, like memsys5, must be activated
by a call to sqlite3_config(SQLITE_CONFIG_HEAP,...).  Memsys3
uses the memory buffer supplied as its source for all memory allocations.
The difference between memsys3 and memsys5 is that memsys3 uses a
different memory allocation algorithm that seems to work well in
practice, but which does not provide mathematical
guarantees against memory fragmentation and breakdown.  Memsys3 was
a predecessor to memsys5.  The SQLite developers now believe that 
memsys5 is superior to
memsys3 and that all applications that need a zero-malloc memory
allocator should use memsys5 in preference to memsys3.  Memsys3 is
considered both experimental and deprecated and will likely be removed 
from the source tree in a future release of SQLite.

Memsys4 and memsys6 were experimental memory allocators
introduced in around 2007 and subsequently removed from the
source tree in around 2008, after it became clear that they
added no new value.

Other experimental memory allocators might be added in future releases
of SQLite.  One may anticipate that these will be called memsys7, memsys8,
and so forth.


3.1.6. Application-defined memory allocators

New memory allocators do not have to be part of the SQLite source tree
nor included in the sqlite3.c amalgamation.  Individual applications can
supply their own memory allocators to SQLite at start-time.

To cause SQLite to use a new memory allocator, the application
simply calls:


sqlite3_config(SQLITE_CONFIG_MALLOC, pMem);


In the call above, pMem is a pointer to an sqlite3_mem_methods object
that defines the interface to the application-specific memory allocator.
The sqlite3_mem_methods object is really just a structure containing
pointers to functions to implement the various memory allocation primitives.


In a multi-threaded application, access to the sqlite3_mem_methods
is serialized if and only if SQLITE_CONFIG_MEMSTATUS is enabled.
If SQLITE_CONFIG_MEMSTATUS is disabled then the methods in
sqlite3_mem_methods must take care of their own serialization needs.


3.1.7. Memory allocator overlays

An application can insert layers or "overlays" in between the
SQLite core and the underlying memory allocator.
For example, the out-of-memory test logic
for SQLite uses an overlay that can simulate memory allocation
failures.

An overlay can be created by using the


sqlite3_config(SQLITE_CONFIG_GETMALLOC, pOldMem);


interface to obtain pointers to the existing memory allocator.
The existing allocator is saved by the overlay and is used as
a fallback to do real memory allocation.  Then the overlay is
inserted in place of the existing memory allocator using
the sqlite3_config(SQLITE_CONFIG_MALLOC,...) as described
above.


3.1.8. No-op memory allocator stub

If SQLite is compiled with the SQLITE_ZERO_MALLOC option, then
the default memory allocator is omitted and replaced by a stub
memory allocator that never allocates any memory.  Any calls to the
stub memory allocator will report back that no memory is available.

The no-op memory allocator is not useful by itself.  It exists only
as a placeholder so that SQLite has a memory allocator to link against
on systems that may not have malloc(), free(), or realloc() in their
standard library.
An application that is compiled with SQLITE_ZERO_MALLOC will need to
use sqlite3_config() together with SQLITE_CONFIG_MALLOC or
SQLITE_CONFIG_HEAP to specify a new alternative memory allocator
before beginning to use SQLite.



3.2.  Page cache memory

In most applications, the database page cache subsystem within 
SQLite uses more dynamically allocated memory than all other parts
of SQLite combined.  It is not unusual to see the database page cache
consume over 10 times more memory than the rest of SQLite combined.

SQLite can be configured to make page cache memory allocations from
a separate and distinct memory pool of fixed-size
slots.  This can have two advantages:



Because allocations are all the same size, the memory allocator can
operate much faster.  The allocator need not bother with coalescing 
adjacent free slots or searching for a slot
of an appropriate size.  All unallocated memory slots can be stored on
a linked list.  Allocating consists of removing the first entry from the
list.  Deallocating is simply adding an entry to the beginning of the list.



With a single allocation size, the n parameter in the
Robson proof is 1, and the total memory space required by the allocator
(N) is exactly equal to maximum memory used (M).  
No additional memory is required to cover fragmentation overhead, thus 
reducing memory requirements.  This is particularly important for the
page cache memory since the page cache constitutes the largest component
of the memory needs of SQLite.



The page-cache memory allocator is disabled by default.
An application can enable it at start-time as follows:


sqlite3_config(SQLITE_CONFIG_PAGECACHE, pBuf, sz, N);


The pBuf parameter is a pointer to a contiguous range of bytes that
SQLite will use for page-cache memory allocations.  The buffer must be
at least sz*N bytes in size.  The "sz" parameter
is the size of each page-cache allocation.  N is the maximum 
number of available allocations.

If SQLite needs a page-cache entry that is larger than "sz" bytes or
if it needs more than N entries, it falls back to using the
general-purpose memory allocator.



3.3.  Lookaside memory allocator

SQLite database connections make many
small and short-lived memory allocations.
This occurs most commonly when compiling SQL statements using
sqlite3_prepare_v2() but also to a lesser extent when running
prepared statements using sqlite3_step().  These small memory
allocations are used to hold things such as the names of tables
and columns, parse tree nodes, individual query results values,
and B-Tree cursor objects.  There are consequently
many calls to malloc() and free() - so many calls that malloc() and
free() end up using a significant fraction of the CPU time assigned
to SQLite.

SQLite version 3.6.1 (2008-08-06)
introduced the lookaside memory allocator to
help reduce the memory allocation load.  In the lookaside allocator,
each database connection preallocates a single large chunk of memory
(typically in the range of 60 to 120 kilobytes) and divides that chunk
up into small fixed-size "slots" of around 100 to 1000 byte each.  This
becomes the lookaside memory pool.  Thereafter, memory allocations
associated with the database connection and that are not too large
are satisfied using one of the lookaside pool slots rather than by calling
the general-purpose memory allocator.  Larger allocations continue to
use the general-purpose memory allocator, as do allocations that occur
when the lookaside pool slots are all checked out.  
But in many cases, the memory
allocations are small enough and there are few enough outstanding that
the new memory requests can be satisfied from the lookaside
pool.

Because lookaside allocations are always the same size, the allocation
and deallocation algorithms are very quick.  There is no
need to coalesce adjacent free slots or search for a slot
of a particular size.  Each database connection maintains a singly-linked
list of unused slots.  Allocation requests simply pull the first
element of this list.  Deallocations simply push the element back onto
the front of the list.
Furthermore, each database connection is assumed to already be
running in a single thread (there are mutexes already in
place to enforce this) so no additional mutexing is required to 
serialize access to the lookaside slot freelist.
Consequently, lookaside memory
allocations and deallocations are very fast.  In speed tests on
Linux and Mac OS X workstations, SQLite has shown overall performance
improvements as high as 10% and 15%, depending on the workload how
and lookaside is configured.

The size of the lookaside memory pool has a global default value
but can also be configured on a connection-by-connection basis.
To change the default size of the lookaside memory pool at
compile-time, use the 
-DSQLITE_DEFAULT_LOOKASIDE=SZ,N
option.
To change the default size of the lookaside memory pool at
start-time, use the sqlite3_config() interface:


sqlite3_config(SQLITE_CONFIG_LOOKASIDE, sz, cnt);


The "sz" parameter is the size in bytes of each lookaside slot.
The "cnt" parameter is
the total number of lookaside memory slots per database connection.
The total amount
of lookaside memory allocated to each database connection is
sz*cnt bytes. 


The lookaside pool can be changed for an individual
database connection "db" using this call:


sqlite3_db_config(db, SQLITE_DBCONFIG_LOOKASIDE, pBuf, sz, cnt);


The "pBuf" parameter is a pointer to memory space that will be
used for the lookaside memory pool.  If pBuf is NULL, then SQLite
will obtain its own space for the memory pool using sqlite3_malloc().
The "sz" and "cnt" parameters are the size of each lookaside slot
and the number of slots, respectively.  If pBuf is not NULL, then it
must point to at least sz*cnt bytes of memory.

The lookaside configuration can only be changed while there are
no outstanding lookaside allocations for the database connection.
Hence, the configuration should be set immediately after creating the 
database connection using sqlite3_open() (or equivalent) and before
evaluating any SQL statements on the connection.

3.3.1. Two-Size Lookaside


Beginning with SQLite version 3.31.0 (2020-01-22),
lookaside supports two memory pools, each with a different size
slot.  The small-slot pool uses 128-byte slots and the large-slot
pool uses whatever size is specified by SQLITE_DBCONFIG_LOOKASIDE
(defaulting to 1200 bytes).  Splitting the pool in two like this
allows memory allocations to be covered by lookaside more often
while at the same time reducing per-database-connection heap usage
from 120KB down to 48KB.


Configuration continues to use the SQLITE_DBCONFIG_LOOKASIDE or
SQLITE_CONFIG_LOOKASIDE configuration options, as described above,
with parameters "sz" and "cnt".  The total heap space used for
lookaside continues to be sz*cnt bytes.  But the space is allocated
between the small-slot lookaside and big-slot lookaside, with
preference given to small-slot lookaside.  The total number of
slots will usually exceed "cnt", since "sz" is typically much
larger than the small-slot size of 128 bytes.


The default lookaside configuration has changed from 100 slots
of 1200 bytes each (120KB) to be 40 slots of 1200 bytes each
(48KB).  This space ends up being allocated as 93 slots of
128 bytes each and 30 slots of 1200 bytes each.  So more lookaside
slots are available but much less heap space is used.


The default lookaside configuration, the size of the small-slots,
and the details of how heap space is allocated between small-slots
and big-slots, are all subject to change from one release to the
next.




3.4.  Memory status

By default, SQLite keeps statistics on its memory usage.  These
statistics are useful in helping to determine how much memory an
application really needs.  The statistics can also be used in
high-reliability system to determine
if the memory usage is coming close to or exceeding the limits 
of the Robson proof and hence that the memory allocation subsystem is 
liable to breakdown.

Most memory statistics are global, and therefore the tracking of
statistics must be serialized with a mutex.  Statistics are turned 
on by default, but an option exists to disable them.  By disabling 
memory statistics,
SQLite avoids entering and leaving a mutex on each memory allocation
and deallocation.  That savings can be noticeable on systems where
mutex operations are expensive.  To disable memory statistics, the
following interface is used at start-time:


sqlite3_config(SQLITE_CONFIG_MEMSTATUS, onoff);


The "onoff" parameter is true to enable the tracking of memory
statistics and false to disable statistics tracking.

Assuming statistics are enabled, the following routine can be used
to access them:


sqlite3_status(verb, &amp;current, &amp;highwater, resetflag);


The "verb" argument determines what statistic is accessed.
There are various verbs defined.  The
list is expected to grow as the sqlite3_status() interface matures.
The current value the selected parameter is written into integer 
"current" and the highest historical value
is written into integer "highwater".  If resetflag is true, then
the high-water mark is reset down to the current value after the call
returns.

A different interface is used to find statistics associated with a
single database connection:


sqlite3_db_status(db, verb, &amp;current, &amp;highwater, resetflag);


This interface is similar except that it takes a pointer to
a database connection as its first argument and returns statistics about
that one object rather than about the entire SQLite library.
The sqlite3_db_status() interface currently only recognizes a
single verb SQLITE_DBSTATUS_LOOKASIDE_USED, though additional verbs
may be added in the future.

The per-connection statistics do not use global variables and hence
do not require mutexes to update or access.  Consequently the
per-connection statistics continue to function even if
SQLITE_CONFIG_MEMSTATUS is turned off.


3.5.  Setting memory usage limits

The sqlite3_soft_heap_limit64() interface can be used to set an
upper bound on the total amount of outstanding memory that the
general-purpose memory allocator for SQLite will allow to be outstanding
at one time.  If attempts are made to allocate more memory than specified
by the soft heap limit, then SQLite will first attempt to free cache
memory before continuing with the allocation request.  The soft heap
limit mechanism only works if memory statistics are enabled and
it works best
if the SQLite library is compiled with the SQLITE_ENABLE_MEMORY_MANAGEMENT
compile-time option.

The soft heap limit is "soft" in this sense:  If SQLite is not able
to free up enough auxiliary memory to stay below the limit, it goes
ahead and allocates the extra memory and exceeds its limit.  This occurs
under the theory that it is better to use additional memory than to fail
outright.

As of SQLite version 3.6.1 (2008-08-06), 
the soft heap limit only applies to the
general-purpose memory allocator.  The soft heap limit does not know
about or interact with
the pagecache memory allocator or the lookaside memory allocator.
This deficiency will likely be addressed in a future release.



4.  Mathematical Guarantees Against Memory Allocation Failures

The problem of dynamic memory allocation, and specifically the
problem of a memory allocator breakdown, has been studied by
J. M. Robson and the results published as:


J. M. Robson.  "Bounds for Some Functions Concerning Dynamic
Storage Allocation".  Journal of the Association for
Computing Machinery, Volume 21, Number 8, July 1974,
pages 491-499.


Let us use the following notation (similar but not identical to
Robson's notation):



N

The amount of raw memory needed by the memory allocation system
in order to guarantee that no memory allocation will ever fail.

M

The maximum amount of memory that the application ever has checked out
at any point in time.

n

The ratio of the largest memory allocation to the smallest.  We assume
that every memory allocation size is an integer multiple of the smallest memory
allocation size.




Robson proves the following result:


N = M*(1 + (log2 n)/2) - n + 1


Colloquially, the Robson proof shows that in order to guarantee
breakdown-free operation, any memory allocator must use a memory pool
of size N which exceeds the maximum amount of memory ever
used M by a multiplier that depends on n, 
the ratio of the largest to the smallest allocation size.  In other
words, unless all memory allocations are of exactly the same size
(n=1) then the system needs access to more memory than it will
ever use at one time.  Furthermore, we see that the amount of surplus
memory required grows rapidly as the ratio of largest to smallest
allocations increases, and so there is strong incentive to keep all
allocations as near to the same size as possible.

Robson's proof is constructive. 
He provides an algorithm for computing a sequence of allocation
and deallocation operations that will lead to an allocation failure due to
memory fragmentation if available memory is as much as one byte
less than N.
And, Robson shows that a power-of-two first-fit memory allocator
(such as implemented by memsys5) will never fail a memory allocation
provided that available memory is N or more bytes.

The values M and n are properties of the application.
If an application is constructed in such a way that both M and
n are known, or at least have known upper bounds, and if the
application uses
the memsys5 memory allocator and is provided with N bytes of
available memory space using SQLITE_CONFIG_HEAP
then Robson proves that no memory allocation request will ever fail
within the application.
To put this another way, the application developer can select a value
for N that will guarantee that no call to any SQLite interface
will ever return SQLITE_NOMEM.  The memory pool will never become
so fragmented that a new memory allocation request cannot be satisfied.
This is an important property for
applications where a software fault could cause injury, physical harm, or
loss of irreplaceable data.

4.1.  Computing and controlling parameters M and n

The Robson proof applies separately to each of the memory allocators
used by SQLite:


The general-purpose memory allocator (memsys5).
The pagecache memory allocator.
The lookaside memory allocator.


For allocators other than memsys5,
all memory allocations are of the same size.  Hence, n=1
and therefore N=M.  In other words, the memory pool need
be no larger than the largest amount of memory in use at any given moment.

The usage of pagecache memory is somewhat harder to control in
SQLite version 3.6.1, though mechanisms are planned for subsequent
releases that will make controlling pagecache memory much easier.
Prior to the introduction of these new mechanisms, the only way
to control pagecache memory is using the cache_size pragma.

Safety-critical applications will usually want to modify the
default lookaside memory configuration so that when the initial
lookaside memory buffer is allocated during sqlite3_open() the
resulting memory allocation is not so large as to force the n
parameter to be too large.  In order to keep n under control,
it is best to try to keep the largest memory allocation below 2 or 4
kilobytes.  Hence, a reasonable default setup for the lookaside
memory allocator might any one of the following:


sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 32, 32);  /* 1K */
sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 64, 32);  /* 2K */
sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 32, 64);  /* 2K */
sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 64, 64);  /* 4K */


Another approach is to initially disable the lookaside memory
allocator:


sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 0, 0);


Then let the application maintain a separate pool of larger
lookaside memory buffers that it can distribute to database connections
as they are created.  In the common case, the application will only
have a single database connection and so the lookaside memory pool
can consist of a single large buffer.


sqlite3_db_config(db, SQLITE_DBCONFIG_LOOKASIDE, aStatic, 256, 500);


The lookaside memory allocator is really intended as performance
optimization, not as a method for assuring breakdown-free memory allocation,
so it is not unreasonable to completely disable the lookaside memory
allocator for safety-critical operations.

The general purpose memory allocator is the most difficult memory pool
to manage because it supports allocations of varying sizes.  Since 
n is a multiplier on M we want to keep n as small
as possible.  This argues for keeping the minimum allocation size for
memsys5 as large as possible.  In most applications, the
lookaside memory allocator is able to handle small allocations.  So
it is reasonable to set the minimum allocation size for memsys5 to
2, 4 or even 8 times the maximum size of a lookaside allocation.  
A minimum allocation size of 512 is a reasonable setting.

Further to keeping n small, one desires to keep the size of
the largest memory allocations under control.
Large requests to the general-purpose memory allocator
might come from several sources:


SQL table rows that contain large strings or BLOBs.
Complex SQL queries that compile down to large prepared statements.
SQL parser objects used internally by sqlite3_prepare_v2().
Storage space for database connection objects.
Page cache memory allocations that overflow into the general-purpose
    memory allocator.
Lookaside buffer allocations for new database connections.


The last two allocations can be controlled and/or eliminated by
configuring the pagecache memory allocator,
and lookaside memory allocator appropriately, as described above.
The storage space required for database connection objects depends
to some extent on the length of the filename of the database file, but
rarely exceeds 2KB on 32-bit systems.  (More space is required on
64-bit systems due to the increased size of pointers.)
Each parser object uses about 1.6KB of memory.  Thus, elements 3 through 6
above can easily be controlled to keep the maximum memory allocation
size below 2KB.

If the application is designed to manage data in small pieces,
then the database should never contain any large strings or BLOBs
and hence element 1 above should not be a factor.  If the database
does contain large strings or BLOBs, they should be read using
incremental BLOB I/O and rows that contain the
large strings or BLOBs should never be update by any means other
than incremental BLOB I/O.  Otherwise, the 
sqlite3_step() routine will need to read the entire row into
contiguous memory at some point, and that will involve at least
one large memory allocation.

The final source of large memory allocations is the space to hold
the prepared statements that result from compiling complex SQL
operations.  Ongoing work by the SQLite developers is reducing the
amount of space required here.  But large and complex queries might
still require prepared statements that are several kilobytes in
size.  The only workaround at the moment is for the application to
break complex SQL operations up into two or more smaller and simpler 
operations contained in separate prepared statements.

All things considered, applications should normally be able to
hold their maximum memory allocation size below 2K or 4K.  This
gives a value for log2(n) of 2 or 3.  This will
limit N to between 2 and 2.5 times M.

The maximum amount of general-purpose memory needed by the application
is determined by such factors as how many simultaneous open 
database connection and prepared statement objects the application
uses, and on the complexity of the prepared statements.  For any
given application, these factors are normally fixed and can be
determined experimentally using SQLITE_STATUS_MEMORY_USED.
A typical application might only use about 40KB of general-purpose
memory.  This gives a value of N of around 100KB.

4.2.  Ductile failure

If the memory allocation subsystems within SQLite are configured
for breakdown-free operation but the actual memory usage exceeds
design limits set by the Robson proof, SQLite will usually continue 
to operate normally.
The pagecache memory allocator
and the lookaside memory allocator automatically failover
to the memsys5 general-purpose memory allocator.  And it is usually the
case that the memsys5 memory allocator will continue to function
without fragmentation even if M and/or n exceeds the limits
imposed by the Robson proof.  The Robson proof shows that it is 
possible for a memory allocation to break down and fail in this 
circumstance, but such a failure requires an especially
despicable sequence of allocations and deallocations - a sequence that
SQLite has never been observed to follow.  So in practice it is usually
the case that the limits imposed by Robson can be exceeded by a
considerable margin with no ill effect.

Nevertheless, application developers are admonished to monitor
the state of the memory allocation subsystems and raise alarms when
memory usage approaches or exceeds Robson limits.  In this way,
the application will provide operators with abundant warning well
in advance of failure.
The memory statistics interfaces of SQLite provide the application with
all the mechanism necessary to complete the monitoring portion of
this task.


5.  Stability Of Memory Interfaces

Update: As of SQLite version 3.7.0 (2010-07-21), 
all of SQLite memory allocation interfaces
are considered stable and will be supported in future releases.
This page last modified on  2022-04-18 02:55:50 UTC


c2: The dbhash.exe Utility Program
c5: dbhash.html
c4:
1. Overview

The dbhash (or dbhash.exe on Windows) utility is a
command-line program that computes the SHA1 hash of the schema and content 
for an SQLite database.

Dbhash ignores extraneous formatting details and hashes only the database
schema and content.  Hence the hash is constant even if the database file
is modified by:


 VACUUM
 PRAGMA page_size
 PRAGMA journal_mode
 REINDEX
 ANALYZE
 copied via the backup API
 ... and so forth


The operations above can potentially cause vast changes the raw database file,
and hence cause very different SHA1 hashes at the file level.
But since the content represented in the database file is unchanged by these
operations, the hash computed by dbhash is also unchanged.

Dbhash can be used to compare two databases to confirm that they
are equivalent, even though their representation on disk is quite different.
Dbhash might also be used to verify the content of a remote database without having
to transmit the entire content of the remote database over a slow link.

2. Usage

Dbhash is a command-line utility.
To run it, type "dbhash" on a command-line prompt followed by the names of
one or more SQLite database files that are to be hashed.
The database hashes will be displayed on standard output.
For example:

drh@bella:~/sqlite/bld$ dbhash ~/Fossils/sqlite.fossil
8d3da9ff87196312aaa33076627ccb7943ef79e3 /home/drh/Fossils/sqlite.fossil


Dbhash supports command-line options that can restrict the tables of the
database file that are hashed, or restrict the hash to only content or only
the schema.  Run "dbhash --help" for further information.

3. Building

To build a copy of the dbhash utility program on unix, get a copy of the
canonical SQLite source code and enter:

./configure
make dbhash


On Windows, enter:

nmake /f makefile.msc dbhash.exe


The dbhash program is implemented by a single file of C-code
called dbhash.c.
To build the dbhash program manually, simply compile the dbhash.c source file
and link it against the SQLite library.
This page last modified on  2022-01-08 05:02:57 UTC


c2: Database File Format
c5: fileformat.html
c4:
This document describes and defines the on-disk database file
format used by all releases of SQLite since 
version 3.0.0 (2004-06-18).

1. The Database File

The complete state of an SQLite database is usually
contained in a single file on disk called the "main database file".

During a transaction, SQLite stores additional information 
in a second file called the "rollback journal", or if SQLite is in
WAL mode, a write-ahead log file.



1.1. Hot Journals
If the application or
host computer crashes before the transaction completes, then the rollback
journal or write-ahead log contains information needed 
to restore the main database file to a consistent state.  When a rollback 
journal or write-ahead log contains information necessary for recovering 
the state of the database, they are called a "hot journal" or "hot WAL file".
Hot journals and WAL files are only a factor during error recovery
scenarios and so are uncommon, but they are part of the state of an SQLite
database and so cannot be ignored.  This document defines the format
of a rollback journal and the write-ahead log file, but the focus is
on the main database file.

1.2. Pages

The main database file consists of one or more pages.  The size of a
page is a power of two between 512 and 65536 inclusive.  All pages within
the same database are the same size.  The page size for a database file
is determined by the 2-byte integer located at an offset of
16 bytes from the beginning of the database file.

Pages are numbered beginning with 1.  The maximum page number is
4294967294 (232 - 2).  The minimum size
SQLite database is a single 512-byte page.
The maximum size database would be 4294967294 pages at 65536 bytes per
page or 281,474,976,579,584 bytes (about 281 terabytes).  Usually SQLite will
hit the maximum file size limit of the underlying filesystem or disk
hardware long before it hits its own internal size limit.

In common use, SQLite databases tend to range in size from a few kilobytes
to a few gigabytes, though terabyte-size SQLite databases are known to exist
in production.

At any point in time, every page in the main database has a single
use which is one of the following:

The lock-byte page
A freelist page

A freelist trunk page
A freelist leaf page

A b-tree page

A table b-tree interior page
A table b-tree leaf page
An index b-tree interior page
An index b-tree leaf page

A payload overflow page
A pointer map page



All reads from and writes to the main database file begin at a page
boundary and all writes are an integer number of pages in size.  Reads
are also usually an integer number of pages in size, with the one exception
that when the database is first opened, the first 100 bytes of the
database file (the database file header) are read as a sub-page size unit.

Before any information-bearing page of the database is modified, 
the original unmodified content of that page is written into the 
rollback journal.  If a transaction is interrupted and needs to be 
rolled back, the rollback journal can then be used to restore the
database to its original state.  Freelist leaf pages bear no
information that would need to be restored on a rollback and so they
are not written to the journal prior to modification, in order to
reduce disk I/O.



1.3. The Database Header

The first 100 bytes of the database file comprise the database file 
header.  The database file header is divided into fields as shown by
the table below.  All multibyte fields in the database file header are
stored with the most significant byte first (big-endian).


Database Header Format

OffsetSizeDescription
016
The header string: "SQLite format 3\000"
162
The database page size in bytes.  Must be a power of two between 512
and 32768 inclusive, or the value 1 representing a page size of 65536.
181
File format write version.  1 for legacy; 2 for WAL.
191
File format read version.  1 for legacy; 2 for WAL.
201
Bytes of unused "reserved" space at the end of each page.  Usually 0.
211
Maximum embedded payload fraction.  Must be 64.
221
Minimum embedded payload fraction.  Must be 32.
231
Leaf payload fraction.  Must be 32.
244
File change counter.
284
Size of the database file in pages.  The "in-header database size".
324
Page number of the first freelist trunk page.
364
Total number of freelist pages.
404
The schema cookie.
444
The schema format number.  Supported schema formats are 1, 2, 3, and 4.
484
Default page cache size.
524
The page number of the largest root b-tree page when in auto-vacuum or
incremental-vacuum modes, or zero otherwise.
564
The database text encoding.  A value of 1 means UTF-8.  A value of 2
means UTF-16le.  A value of 3 means UTF-16be.
604
The "user version" as read and set by the user_version pragma.
644
True (non-zero) for incremental-vacuum mode.  False (zero) otherwise.
684
The "Application ID" set by PRAGMA application_id.
7220
Reserved for expansion.  Must be zero.
924
The version-valid-for number.
964
SQLITE_VERSION_NUMBER


1.3.1. Magic Header String

Every valid SQLite database file begins with the following 16 bytes 
(in hex): 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00.  This byte sequence
corresponds to the UTF-8 string "SQLite format 3" including the nul
terminator character at the end.

1.3.2. Page Size

The two-byte value beginning at offset 16 determines the page size of 
the database.  For SQLite versions 3.7.0.1 (2010-08-04)
and earlier, this value is 
interpreted as a big-endian integer and must be a power of two between
512 and 32768, inclusive.  Beginning with SQLite version 3.7.1
(2010-08-23), a page
size of 65536 bytes is supported.  The value 65536 will not fit in a
two-byte integer, so to specify a 65536-byte page size, the value
at offset 16 is 0x00 0x01.
This value can be interpreted as a big-endian
1 and thought of as a magic number to represent the 65536 page size.
Or one can view the two-byte field as a little endian number and say
that it represents the page size divided by 256.  These two 
interpretations of the page-size field are equivalent.



1.3.3. File format version numbers

The file format write version and file format read version at offsets
18 and 19 are intended to allow for enhancements of the file format
in future versions of SQLite.  In current versions of SQLite, both of
these values are 1 for rollback journalling modes and 2 for WAL
journalling mode.  If a version of SQLite coded to the current
file format specification encounters a database file where the read
version is 1 or 2 but the write version is greater than 2, then the database
file must be treated as read-only.  If a database file with a read version
greater than 2 is encountered, then that database cannot be read or written.



1.3.4. Reserved bytes per page

SQLite has the ability to set aside a small number of extra bytes at
the end of every page for use by extensions.  These extra bytes are
used, for example, by the SQLite Encryption Extension to store a nonce
and/or cryptographic checksum associated with each page.  The 
"reserved space" size in the 1-byte integer at offset 20 is the number
of bytes of space at the end of each page to reserve for extensions.
This value is usually 0.  The value can be odd.



The "usable size" of a database page is the page size specified by the
2-byte integer at offset 16 in the header less the "reserved" space size
recorded in the 1-byte integer at offset 20 in the header.  The usable
size of a page might be an odd number.  However, the usable size is not
allowed to be less than 480.  In other words, if the page size is 512,
then the reserved space size cannot exceed 32.

1.3.5. Payload fractions

The maximum and minimum embedded payload fractions and the leaf
payload fraction values must be 64, 32, and 32.  These values were
originally intended to be tunable parameters that could be used to
modify the storage format of the b-tree algorithm.  However, that
functionality is not supported and there are no current plans to add
support in the future.  Hence, these three bytes are fixed at the
values specified.

1.3.6. File change counter



The file change counter is a 4-byte big-endian integer at
offset 24 that is incremented whenever the database file is unlocked
after having been modified.
When two or more processes are reading the same database file, each 
process can detect database changes from other processes by monitoring 
the change counter.
A process will normally want to flush its database page cache when
another process modified the database, since the cache has become stale.
The file change counter facilitates this.

In WAL mode, changes to the database are detected using the wal-index
and so the change counter is not needed.  Hence, the change counter might
not be incremented on each transaction in WAL mode.

1.3.7. In-header database size



The 4-byte big-endian integer at offset 28 into the header 
stores the size of the database file in pages.  If this in-header
datasize size is not valid (see the next paragraph), then the database 
size is computed by looking
at the actual size of the database file. Older versions of SQLite
ignored the in-header database size and used the actual file size
exclusively.  Newer versions of SQLite use the in-header database
size if it is available but fall back to the actual file size if
the in-header database size is not valid.

The in-header database size is only considered to be valid if
it is non-zero and if the 4-byte change counter at offset 24
exactly matches the 4-byte version-valid-for number at offset 92.
The in-header database size is always valid 
when the database is only modified using recent versions of SQLite,
versions 3.7.0 (2010-07-21) and later.
If a legacy version of SQLite writes to the database, it will not
know to update the in-header database size and so the in-header
database size could be incorrect.  But legacy versions of SQLite
will also leave the version-valid-for number at offset 92 unchanged
so it will not match the change-counter.  Hence, invalid in-header
database sizes can be detected (and ignored) by observing when
the change-counter does not match the version-valid-for number.

1.3.8. Free page list

Unused pages in the database file are stored on a freelist.  The
4-byte big-endian integer at offset 32 stores the page number of
the first page of the freelist, or zero if the freelist is empty.
The 4-byte big-endian integer at offset 36 stores the total 
number of pages on the freelist.

1.3.9. Schema cookie

The schema cookie is a 4-byte big-endian integer at offset 40
that is incremented whenever the database schema changes.  A 
prepared statement is compiled against a specific version of the
database schema.  When the database schema changes, the statement
must be reprepared.  When a prepared statement runs, it first checks
the schema cookie to ensure the value is the same as when the statement
was prepared and if the schema cookie has changed, the statement either
automatically reprepares and reruns or it aborts with an SQLITE_SCHEMA 
error.



1.3.10. Schema format number

The schema format number is a 4-byte big-endian integer at offset 44.
The schema format number is similar to the file format read and write
version numbers at offsets 18 and 19 except that the schema format number
refers to the high-level SQL formatting rather than the low-level b-tree
formatting.  Four schema format numbers are currently defined:


Format 1 is understood by all versions of SQLite back to
version 3.0.0 (2004-06-18).
Format 2 adds the ability of rows within the same table
to have a varying number of columns, in order to support the
ALTER TABLE ... ADD COLUMN functionality.  Support for
reading and writing format 2 was added in SQLite 
version 3.1.3 on 2005-02-20.
Format 3 adds the ability of extra columns added by
ALTER TABLE ... ADD COLUMN to have non-NULL default
values.  This capability was added in SQLite version 3.1.4
on 2005-03-11.
Format 4 causes SQLite to respect the
DESC keyword on
index declarations.  (The DESC keyword is ignored in indexes for 
formats 1, 2, and 3.)
Format 4 also adds two new boolean record type values (serial types
8 and 9).  Support for format 4 was added in SQLite 3.3.0 on
2006-01-10.


New database files created by SQLite use format 4 by default.
The legacy_file_format pragma can be used to cause SQLite
to create new database files using format 1.
The format version number can be made to default to 1 instead of 4 by
setting SQLITE_DEFAULT_FILE_FORMAT=1 at compile-time.


1.3.11. Suggested cache size

The 4-byte big-endian signed integer at offset 48 is the suggested
cache size in pages for the database file.  The value is a suggestion
only and SQLite is under no obligation to honor it.  The absolute value
of the integer is used as the suggested size.  The suggested cache size
can be set using the default_cache_size pragma.

1.3.12. Incremental vacuum settings

The two 4-byte big-endian integers at offsets 52 and 64 are used
to manage the auto_vacuum and incremental_vacuum modes.  If
the integer at offset 52 is zero then pointer-map (ptrmap) pages are
omitted from the database file and neither auto_vacuum nor
incremental_vacuum are supported.  If the integer at offset 52 is
non-zero then it is the page number of the largest root page in the
database file, the database file will contain ptrmap pages, and the
mode must be either auto_vacuum or incremental_vacuum.  In this latter
case, the integer at offset 64 is true for incremental_vacuum and
false for auto_vacuum.  If the integer at offset 52 is zero then
the integer at offset 64 must also be zero.



1.3.13. Text encoding

The 4-byte big-endian integer at offset 56 determines the encoding
used for all text strings stored in the database.  
A value of 1 means UTF-8.
A value of 2 means UTF-16le.
A value of 3 means UTF-16be.
No other values are allowed.
The sqlite3.h header file defines C-preprocessor macros SQLITE_UTF8 as 1,
SQLITE_UTF16LE as 2, and SQLITE_UTF16BE as 3, to use in place of
the numeric codes for the text encoding.

1.3.14. User version number

The 4-byte big-endian integer at offset 60 is the user version which
is set and queried by the user_version pragma.  The user version is
not used by SQLite.



1.3.15. Application ID

The 4-byte big-endian integer at offset 68 is an "Application ID" that
can be set by the PRAGMA application_id command in order to identify the
database as belonging to or associated with a particular application.
The application ID is intended for database files used as an
application file-format.  The application ID can be used by utilities 
such as file(1) to determine the specific
file type rather than just reporting "SQLite3 Database".  A list of
assigned application IDs can be seen by consulting the
magic.txt
file in the SQLite source repository.



1.3.16. Write library version number and version-valid-for number

The 4-byte big-endian integer at offset 96 stores the 
SQLITE_VERSION_NUMBER value for the SQLite library that most
recently modified the database file.  The 4-byte big-endian integer at
offset 92 is the value of the change counter when the version number
was stored.  The integer at offset 92 indicates which transaction
the version number is valid for and is sometimes called the
"version-valid-for number".

1.3.17. Header space reserved for expansion

All other bytes of the database file header are reserved for
future expansion and must be set to zero.



1.4. The Lock-Byte Page

The lock-byte page is the single page of the database file
that contains the bytes at offsets between 1073741824 and 1073742335,
inclusive.  A database file that is less than or equal to 1073741824 bytes 
in size contains no lock-byte page.  A database file larger than
1073741824 contains exactly one lock-byte page.


The lock-byte page is set aside for use by the operating-system specific
VFS implementation in implementing the database file locking primitives.
SQLite does not use the lock-byte page.  The SQLite core 
will never read or write the lock-byte page,
though operating-system specific VFS 
implementations may choose to read or write bytes on the lock-byte 
page according to the 
needs and proclivities of the underlying system.  The unix and win32
VFS implementations that come built into SQLite do not write to the
lock-byte page, but third-party VFS implementations for
other operating systems might.

The lock-byte page arose from the need to support Win95 which was the
predominant operating system when this file format was designed and which 
only supported mandatory file locking.  All modern operating systems that
we know of support advisory file locking, and so the lock-byte page is
not really needed any more, but is retained for backwards compatibility.



1.5. The Freelist

A database file might contain one or more pages that are not in
active use.  Unused pages can come about, for example, when information
is deleted from the database.  Unused pages are stored on the freelist
and are reused when additional pages are required.

The freelist is organized as a linked list of freelist trunk pages
with each trunk page containing page numbers for zero or more freelist
leaf pages.

A freelist trunk page consists of an array of 4-byte big-endian integers.
The size of the array is as many integers as will fit in the usable space
of a page.  The minimum usable space is 480 bytes so the array will always
be at least 120 entries in length.  The first integer on a freelist trunk
page is the page number of the next freelist trunk page in the list or zero 
if this is the last freelist trunk page.  The second integer on a freelist
trunk page is the number of leaf page pointers to follow.  
Call the second integer on a freelist trunk page L.
If L is greater than zero then integers with array indexes between 2 and
L+1 inclusive contain page numbers for freelist leaf pages.

Freelist leaf pages contain no information.  SQLite avoids reading or
writing freelist leaf pages in order to reduce disk I/O.

A bug in SQLite versions prior to 3.6.0 (2008-07-16)
caused the database to be
reported as corrupt if any of the last 6 entries in the freelist trunk page 
array contained non-zero values.  Newer versions of SQLite do not have
this problem.  However, newer versions of SQLite still avoid using the 
last six entries in the freelist trunk page array in order that database
files created by newer versions of SQLite can be read by older versions
of SQLite.

The number of freelist pages is stored as a 4-byte big-endian integer
in the database header at an offset of 36 from the beginning of the file.
The database header also stores the page number of the first freelist trunk
page as a 4-byte big-endian integer at an offset of 32 from the beginning
of the file.



1.6. B-tree Pages

The b-tree algorithm provides key/data storage with unique and
ordered keys on page-oriented storage devices.
For background information on b-trees, see
Knuth, The Art Of Computer Programming, Volume 3 "Sorting
and Searching", pages 471-479.  Two variants of b-trees are used by
SQLite.  "Table b-trees" use a 64-bit signed integer key and store
all data in the leaves.  "Index b-trees" use arbitrary keys and store no
data at all.

A b-tree page is either an interior page or a leaf page.
A leaf page contains keys and in the case of a table b-tree each
key has associated data.  An interior page contains
K keys together with K+1 pointers to child b-tree pages.
A "pointer" in an interior b-tree page is just the 32-bit 
unsigned integer page number of the child page.

The number of keys on an interior b-tree page, K,
is almost always at least 2 and is usually much more than 2.
The only exception is when page 1 is an interior b-tree page.
Page 1 has 100 fewer bytes of storage space available,
due to the presence of the database header at the beginning of that page,
and so sometimes (rarely) if page 1 is an interior b-tree page, it can
end up holding just a a single key.  In all other cases, K is 2 or more.
The upper bound on K is as many keys as will fit on the page. Large keys
on index b-trees are split up into overflow pages so that no single key
uses more than one fourth of the available storage space on the page
and hence every internal page is able to store at least 4 keys.
The integer keys of table b-trees are never large enough to
require overflow, so key overflow only occurs on index b-trees.

Define the depth
of a leaf b-tree to be 1 and the depth of any interior b-tree to be one
more than the maximum depth of any of its children.  In a well-formed
database, all children of an interior b-tree have the same depth.

In an interior b-tree page, the pointers and keys logically alternate 
with a pointer on both ends. (The previous sentence is to be understood
conceptually - the actual layout of the keys and
pointers within the page is more complicated and will be described in
the sequel.)  All keys within the same page are unique and are logically
organized in ascending order from left to right.  (Again, this ordering
is logical, not physical.  The actual location of keys within the page
is arbitrary.) For any key X, pointers to the left
of a X refer to b-tree pages on which all keys are less than or equal to X.
Pointers to the right of X refer to pages where all keys are 
greater than X.

Within an interior b-tree page, each key and the pointer to its
immediate left are combined into a structure called a "cell".  The
right-most pointer is held separately.  A leaf b-tree page has no
pointers, but it still uses the cell structure to hold keys for
index b-trees or keys and content for table b-trees.  Data is also
contained in the cell.


Every b-tree page has at most one parent b-tree page.
A b-tree page without a parent is called a root page.  A root b-tree page
together with the closure of its children form a complete b-tree.
It is possible (and in fact rather common) to have a complete b-tree
that consists of a single page that is both a leaf and the root.
Because there are pointers from parents to children, every page of a
complete b-tree can be located if only the root page is known.  Hence,
b-trees are identified by their root page number.



A b-tree page is either a table b-tree page or an index b-tree page.
All pages within each complete b-tree are of the same type: either table
or index.  There is one table b-trees in the database file
for each rowid table in the database schema, including system tables
such as sqlite_schema.  There is one index b-tree
in the database file for each index in the schema, including implied indexes
created by uniqueness constraints.  There are no b-trees associated with
virtual tables.  Specific virtual table implementations might make use
of shadow tables for storage, but those shadow tables will have separate
entries in the database schema.  WITHOUT ROWID tables use index b-trees
rather than a table b-trees, so there is one
index b-tree in the database file for each WITHOUT ROWID table.
The b-tree corresponding to the sqlite_schema table is always a table
b-tree and always has a root page of 1.
The sqlite_schema table contains the root page number for every other 
table and index in the database file.

Each entry in a table b-tree consists of a 64-bit signed integer key
and up to 2147483647 bytes of arbitrary data.  (The key of a table b-tree
corresponds to the rowid of the SQL table that the b-tree implements.)
Interior table b-trees hold only keys and pointers to children.
All data is contained in the table b-tree leaves.

Each entry in an index b-tree consists of an arbitrary key of up
to 2147483647 bytes in length and no data.



Define the "payload" of a cell to be the arbitrary length section
of the cell.  For an index b-tree, the key is always arbitrary in length
and hence the payload is the key.  There are no arbitrary length elements
in the cells of interior table b-tree pages and so those cells have no
payload.  Table b-tree leaf pages contain arbitrary length content and
so for cells on those pages the payload is the content.
When the size of payload for a cell exceeds a certain threshold (to
be defined later) then only the first few bytes of the payload
are stored on the b-tree page and the balance is stored in a linked list
of content overflow pages.

A b-tree page is divided into regions in the following order:


The 100-byte database file header (found on page 1 only)
The 8 or 12 byte b-tree page header
The cell pointer array
Unallocated space
The cell content area
The reserved region.



The 100-byte database file header is found only on page 1, which is
always a table b-tree page.  All other b-tree pages in the database file
omit this 100-byte header.

The reserved region is an area of unused space at the end of every
page (except the locking page) that extensions can use to hold per-page
information.  The size of the reserved region is determined by the one-byte
unsigned integer found at an offset of 20 into the database file header.
The size of the reserved region is usually zero.

The b-tree page header is 8 bytes in size for leaf pages and 12
bytes for interior pages.  All multibyte values in the page header
are big-endian.
The b-tree page header is composed of the following fields:


B-tree Page Header Format

OffsetSizeDescription
01
The one-byte flag at offset 0 indicating the b-tree page type.
A value of 2 (0x02) means the page is an interior index b-tree page.
A value of 5 (0x05) means the page is an interior table b-tree page.
A value of 10 (0x0a) means the page is a leaf index b-tree page.
A value of 13 (0x0d) means the page is a leaf table b-tree page.
Any other value for the b-tree page type is an error.
12
The two-byte integer at offset 1 gives the start of the
first freeblock on the page, or is zero if there are no freeblocks.
32
The two-byte integer at offset 3 gives the number of cells on the page.
52
The two-byte integer at offset 5 designates the start of the cell content
area.  A zero value for this integer is interpreted as 65536.
71
The one-byte integer at offset 7 gives the number of fragmented free
bytes within the cell content area.
84
The four-byte page number at offset 8 is the right-most pointer.  This
value appears in the header of interior b-tree pages only and is omitted from
all other pages.


The cell pointer array of a b-tree page immediately follows the b-tree
page header.  Let K be the number of cells on the btree.  The cell pointer
array consists of K 2-byte integer offsets to the cell contents.  The
cell pointers are arranged in key order with left-most cell (the cell with the
smallest key) first and the right-most cell (the cell with the largest
key) last.

Cell content is stored in the cell content region of the b-tree page.
SQLite strives to place cells as far toward the end of the b-tree page as
it can, in order to leave space for future growth of the cell pointer array.
The area in between the last cell pointer array entry and the beginning of
the first cell is the unallocated region.


If a page contains no cells (which is only possible for a root page
of a table that contains no rows) then the offset to the
cell content area will equal the page size minus the bytes of reserved space.
If the database uses a 65536-byte page size and the reserved space is zero
(the usual value for reserved space) then the cell content offset of an
empty page wants to be 65536.  
However, that integer is too large to be stored in a
2-byte unsigned integer, so a value of 0 is used in its place.

A freeblock is a structure used to identify unallocated space within
a b-tree page.  Freeblocks are organized as a chain.  The first 2 bytes of
a freeblock are a big-endian integer which is the offset in the b-tree page
of the next freeblock in the chain, or zero if the freeblock is the last on
the chain.  The third and fourth bytes of each freeblock form
a big-endian integer which is the size of the freeblock in bytes, including
the 4-byte header.  Freeblocks are always connected in order 
of increasing offset.  The second field of the b-tree page header is the
offset of the first freeblock, or zero if there are no freeblocks on the
page.  In a well-formed b-tree page, there will always be at least one cell
before the first freeblock.

A freeblock requires at least 4 bytes of space.  If there is an isolated
group of 1, 2, or 3 unused bytes within the cell content area, those bytes
comprise a fragment.  The total number of bytes in all fragments is stored
in the fifth field of the b-tree page header.  In a well-formed b-tree page,
the total number of bytes in fragments may not exceed 60.

The total amount of free space on a b-tree page consists of the size
of the unallocated region plus the total size of all freeblocks plus the
number of fragmented free bytes.  SQLite may from time to time reorganize
a b-tree page so that there are no freeblocks or fragment bytes, all
unused bytes are contained in the unallocated space region, and all
cells are packed tightly at the end of the page.  This is called 
"defragmenting" the b-tree page.




A variable-length integer or "varint" is a static Huffman encoding
of 64-bit twos-complement integers that uses less space for small positive 
values. 
A varint is between 1 and 9 bytes in length.  The varint consists of either
zero or more bytes which have the high-order bit set followed by a single byte
with the high-order bit clear, or nine bytes, whichever is shorter.
The lower seven bits of each of the first eight bytes and all 8 bits of
the ninth byte are used to reconstruct the 64-bit twos-complement integer.
Varints are big-endian: bits taken from the earlier byte of the varint
are more significant than bits taken from the later bytes. 

The format of a cell depends on which kind of b-tree page the cell
appears on.  The following table shows the elements of a cell, in
order of appearance, for the various b-tree page types.


Table B-Tree Leaf Cell (header 0x0d):

A varint which is the total number of bytes of payload, including any
overflow
A varint which is the integer key, a.k.a. "rowid"
The initial portion of the payload that does not spill to overflow
pages.
A 4-byte big-endian integer page number for the first page of the
overflow page list - omitted if all payload fits on the b-tree page.


Table B-Tree Interior Cell (header 0x05):

A 4-byte big-endian page number which is the left child pointer.
A varint which is the integer key


Index B-Tree Leaf Cell (header 0x0a):

A varint which is the total number of bytes of key payload, including any
overflow
The initial portion of the payload that does not spill to overflow
pages.
A 4-byte big-endian integer page number for the first page of the
overflow page list - omitted if all payload fits on the b-tree page.


Index B-Tree Interior Cell (header 0x02):

A 4-byte big-endian page number which is the left child pointer.
A varint which is the total number of bytes of key payload, including any
overflow
The initial portion of the payload that does not spill to overflow
pages.
A 4-byte big-endian integer page number for the first page of the
overflow page list - omitted if all payload fits on the b-tree page.



The information above can be recast into a table format as follows:




B-tree Cell Format

Datatype
    Appears in...
    Description
Table Leaf (0x0d)
    Table Interior (0x05)
    Index Leaf (0x0a)
    Index Interior (0x02)
4-byte integer
    &nbsp;
    &#x2714;
    &nbsp;
    &#x2714;
    Page number of left child
varint
    &#x2714;
    &nbsp;
    &#x2714;
    &#x2714;
    Number of bytes of payload
varint
    &#x2714;
    &#x2714;
    &nbsp;
    &nbsp;
    Rowid
byte array
    &#x2714;
    &nbsp;
    &#x2714;
    &#x2714;
    Payload
4-byte integer
    &#x2714;
    &nbsp;
    &#x2714;
    &#x2714;
    Page number of first overflow page



The amount of payload that spills onto overflow pages also depends on
the page type.  For the following computations, let U be the usable size
of a database page, the total page size less the reserved space at the
end of each page.  And let P be the payload size.  In the following,
symbol X represents the maximum amount of payload that can be stored directly
on the b-tree page without spilling onto an overflow page and symbol M
represents the minimum amount of payload that must be stored on the btree
page before spilling is allowed.


Table B-Tree Leaf Cell:

Let X be U-35.  If the payload size P is less than or equal to X then
the entire payload is stored on the b-tree leaf page.
Let M be ((U-12)*32/255)-23 and let K be M+((P-M)%(U-4)).
If P is greater than X
then the number of bytes stored on the table b-tree leaf page is K
if K is less or equal to X or M otherwise.
The number of bytes stored on the leaf page is never less than M.


Table B-Tree Interior Cell:

Interior pages of table b-trees have no payload and so there is never
any payload to spill.


Index B-Tree Leaf Or Interior Cell:

Let X be ((U-12)*64/255)-23.  If the payload size P is less than
or equal to X then the entire payload is stored on the b-tree page.
Let M be ((U-12)*32/255)-23 and let K be M+((P-M)%(U-4)).
If P is greater than X then the number
of bytes stored on the index b-tree page is K if K is less than or
equal to X or M otherwise.
The number of bytes stored on the index page is never less than M.



Here is an alternative description of the same computation:


X is U-35 for table btree leaf pages or
    ((U-12)*64/255)-23 for index pages.
M is always ((U-12)*32/255)-23.
Let K be M+((P-M)%(U-4)).
If P&lt;=X then all P bytes of payload are stored directly on the 
    btree page without overflow.
If P&gt;X and K&lt;=X then the first K bytes of P are stored on the 
    btree page and the remaining P-K bytes are stored on overflow pages.
If P&gt;X and K&gt;X then the first M bytes of P are stored on the
    btree page and the remaining P-M bytes are stored on overflow pages.


The overflow thresholds are designed to give a minimum fanout of
4 for index b-trees and to make sure enough of the payload
is on the b-tree page that the record header can usually be accessed
without consulting an overflow page.  In hindsight, the designer of
the SQLite b-tree logic realized that these thresholds could have been
made much simpler.  However, the computations cannot be changed
without resulting in an incompatible file format.  And the current computations
work well, even if they are a little complex.



1.7. Cell Payload Overflow Pages

When the payload of a b-tree cell is too large for the b-tree page,
the surplus is spilled onto overflow pages.  Overflow pages form a linked
list.  The first four bytes of each overflow page are a big-endian
integer which is the page number of the next page in the chain, or zero
for the final page in the chain.  The fifth byte through the last usable
byte are used to hold overflow content.

1.8. Pointer Map or Ptrmap Pages

Pointer map or ptrmap pages are extra pages inserted into the database
to make the operation of auto_vacuum and incremental_vacuum modes
more efficient.  Other page types in the database typically have pointers
from parent to child.  For example, an interior b-tree page contains pointers
to its child b-tree pages and an overflow chain has a pointer
from earlier to later links in the chain.  A ptrmap page contains linkage
information going in the opposite direction, from child to parent.

Ptrmap pages must exist in any database file which has a non-zero
largest root b-tree page value at offset 52 in the database header.
If the largest root b-tree page value is zero, then the database must not
contain ptrmap pages.

In a database with ptrmap pages, the first ptrmap page is page 2.
A ptrmap page consists of an array of 5-byte entries.  Let J be the
number of 5-byte entries that will fit in the usable space of a page.
(In other words, J=U/5.)  The first ptrmap page will contain back pointer
information for pages 3 through J+2, inclusive.  The second pointer map
page will be on page J+3 and that ptrmap page will provide back pointer
information for pages J+4 through 2*J+3 inclusive.  And so forth for
the entire database file.

In a database that uses ptrmap pages, all pages at locations identified
by the computation in the previous paragraph must be ptrmap page and no
other page may be a ptrmap page.  Except, if the byte-lock page happens to
fall on the same page number as a ptrmap page, then the ptrmap is moved
to the following page for that one case.

Each 5-byte entry on a ptrmap page provides back-link information about 
one of the pages that immediately follow the pointer map.  If page B is a
ptrmap page then back-link information about page B+1 is provided by
the first entry on the pointer map.  Information about page B+2 is
provided by the second entry.  And so forth.

Each 5-byte ptrmap entry consists of one byte of "page type" information
followed by a 4-byte big-endian page number.  Five page types are recognized:



A b-tree root page.  The
page number should be zero.
A freelist page.  The page number should be
zero.
The first page of a
cell payload overflow chain.  The page number is the b-tree page that
contains the cell whose content has overflowed.
A page in an overflow chain
other than the first page.  The page number is the prior page of the
overflow chain.
A non-root b-tree page.  The
page number is the parent b-tree page.


In any database file that contains ptrmap pages, all b-tree root pages 
must come before any non-root b-tree page, cell payload overflow page, or
freelist page.  This restriction ensures that a root page will never
be moved during an auto-vacuum or incremental-vacuum.  The auto-vacuum
logic does not know how to update the root_page field of the sqlite_schema
table and so it is necessary to prevent root pages from being moved
during an auto-vacuum in order to preserve the integrity of the
sqlite_schema table.  Root pages are moved to the beginning of the
database file by the CREATE TABLE, CREATE INDEX, DROP TABLE, and
DROP INDEX operations.

2. Schema Layer

The foregoing text describes low-level aspects of the SQLite file
format.  The b-tree mechanism provides a powerful and efficient means of
accessing a large data set.  This section will describe how the
low-level b-tree layer is used to implement higher-level SQL
capabilities.



2.1. Record Format

The data for a table b-tree leaf page and the key
of an index b-tree page was characterized above
as an arbitrary sequence of bytes.
The prior discussion mentioned one key being less than another, but
did not define what "less than" meant.  The current section will address
these omissions.

Payload, either table b-tree data or index b-tree keys, 
is always in the "record format".
The record format defines a sequence of values corresponding
to columns in a table or index.  The record format specifies the number
of columns, the datatype of each column, and the content of each column.

The record format makes extensive use of the 
variable-length integer or varint
representation of 64-bit signed integers defined above.



A record contains a header and a body, in that order.  
The header begins with a single varint which determines the total number
of bytes in the header.  The varint value is the size of the header in
bytes including the size varint itself.  Following the size varint are
one or more additional varints, one per column.  These additional varints
are called "serial type" numbers and
determine the datatype of each column, according to the following chart:


Serial Type Codes Of The Record Format

Serial TypeContent SizeMeaning
00
Value is a NULL.
11
Value is an 8-bit twos-complement integer.
22
Value is a big-endian 16-bit twos-complement integer.
33
Value is a big-endian 24-bit twos-complement integer.
44
Value is a big-endian 32-bit twos-complement integer.
56
Value is a big-endian 48-bit twos-complement integer.
68
Value is a big-endian 64-bit twos-complement integer.
78
Value is a big-endian IEEE 754-2008 64-bit floating point number.
80
Value is the integer 0. (Only available for schema format 4 and higher.)
90
Value is the integer 1. (Only available for schema format 4 and higher.)
10,11
    variable
Reserved for internal use.  These serial type codes will
   never appear in a well-formed database file, but they
   might be used in transient and temporary database files
   that SQLite sometimes generates for its own use.
   The meanings of these codes can shift from one release
   of SQLite to the next.
N&#x2265;12 and even
    (N-12)/2
Value is a BLOB that is (N-12)/2 bytes in length.
N&#x2265;13 and odd
    (N-13)/2
Value is a string in the text encoding and (N-13)/2 bytes in length.
The nul terminator is not stored.


The header size varint
and serial type varints will usually consist of a single byte.  The
serial type varints for large strings and BLOBs might extend to two or three
byte varints, but that is the exception rather than the rule. 
The varint format is very efficient at coding the record header.

The values for each column in the record immediately follow the header.
For serial types 0, 8, 9, 12, and 13, the value is zero bytes in
length.  If all columns are of these types then the body section of the
record is empty.

A record might have fewer values than the number of columns in the
corresponding table.  This can happen, for example, after an
ALTER TABLE ... ADD COLUMN SQL statement has increased
the number of columns in the table schema without modifying preexisting rows
in the table.
Missing values at the end of the record are filled in using the
default value for the corresponding columns defined in the table schema.



2.2. Record Sort Order

The order of keys in an index b-tree is determined by the sort order of
the records that the keys represent.  Record comparison progresses column
by column.  Columns of a record are examined from left to right.  The
first pair of columns that are not equal determines the relative order
of the two records.  The sort order of individual columns is as
follows:


NULL values (serial type 0) sort first.
Numeric values (serial types 1 through 9) sort after NULLs
      and in numeric order.
Text values (odd serial types 13 and larger) sort after numeric
    values in the order determined by the columns collating function.
BLOB values (even serial types 12 and larger) sort last and in the order 
    determined by memcmp().


A collating function for each column is necessary in order to compute
the order of text fields.
SQLite defines three built-in collating functions:



BINARY
     The built-in BINARY collation compares strings byte by byte
        using the memcmp() function
        from the standard C library.
NOCASE
     The NOCASE collation is like BINARY except that uppercase
        ASCII characters ('A' through 'Z')
        are folded into their lowercase equivalents prior to running the
        comparison.  Only ASCII characters are case-folded.
        NOCASE
        does not implement a general purpose unicode caseless comparison.
RTRIM
     RTRIM is like BINARY except that extra spaces at the end of either
         string do not change the result.  In other words, strings will
         compare equal to one another as long as they
         differ only in the number of spaces at the end.


Additional application-specific collating functions can be added to
SQLite using the sqlite3_create_collation() interface.

The default collating function for all strings is BINARY.
Alternative collating functions for table columns can be specified in the
CREATE TABLE statement using the COLLATE clause on the column definition.
When a column is indexed, the same collating function specified in the
CREATE TABLE statement is used for the column in the index, by default,
though this can be overridden using a COLLATE clause in the 
CREATE INDEX statement.



2.3. Representation Of SQL Tables

 Each ordinary SQL table in the database schema is represented on-disk
by a table b-tree.  Each entry in the table b-tree corresponds to a row
of the SQL table.  The rowid of the SQL table is the 64-bit signed
integer key for each entry in the table b-tree.

 The content of each SQL table row is stored in the database file by
first combining the values in the various columns into a byte array
in the record format, then storing that byte array as the payload in
an entry in the table b-tree.  The order of values in the record is
the same as the order of columns in the SQL table definition.
When an SQL table includes an
INTEGER PRIMARY KEY column (which aliases the rowid) then that
column appears in the record as a NULL value.  SQLite will always use
the table b-tree key rather than the NULL value when referencing the
INTEGER PRIMARY KEY column.

 If the affinity of a column is REAL and that column contains a
value that can be converted to an integer without loss of information
(if the value contains no fractional part and is not too large to be
represented as an integer) then the column may be stored in the record
as an integer.  SQLite will convert the value back to floating
point when extracting it from the record.

2.4. Representation of WITHOUT ROWID Tables

If an SQL table is created using the "WITHOUT ROWID" clause at the
end of its CREATE TABLE statement, then that table is a WITHOUT ROWID
table and uses a different on-disk representation.  A WITHOUT ROWID
table uses an index b-tree rather than a table b-tree for storage.
The key for each entry in the WITHOUT ROWID b-tree is a record composed
of the columns of the PRIMARY KEY followed by all remaining columns of
the table.  The primary key columns appear in the order that they were
declared in the PRIMARY KEY clause and the remaining columns appear in
the order they occur in the CREATE TABLE statement.

Hence, the content encoding for a WITHOUT ROWID table is the same
as the content encoding for an ordinary rowid table, except that the
order of the columns is rearranged so that PRIMARY KEY columns come
first, and the content is used as the key in an index b-tree rather
than as the data in a table b-tree.
The special encoding rules for columns with REAL affinity
apply to WITHOUT ROWID tables the same as they do with rowid tables.

2.4.1. Suppression of redundant columns in the PRIMARY KEY
    of WITHOUT ROWID tables

If the PRIMARY KEY of a WITHOUT ROWID tables uses the same columns
with the same collating sequence more than once, then the second and
subsequent occurrences of that column in the PRIMARY KEY definition are
ignored.  For example, the following CREATE TABLE statements all specify
the same table, which will have the exact same representation on disk:


CREATE TABLE t1(a,b,c,d,PRIMARY KEY(a,c)) WITHOUT ROWID;
CREATE TABLE t1(a,b,c,d,PRIMARY KEY(a,c,a,c)) WITHOUT ROWID;
CREATE TABLE t1(a,b,c,d,PRIMARY KEY(a,A,a,C)) WITHOUT ROWID;
CREATE TABLE t1(a,b,c,d,PRIMARY KEY(a,a,a,a,c)) WITHOUT ROWID;


The first example above is the preferred definition of the table,
of course.  All of the examples create a WITHOUT ROWID table with
two PRIMARY KEY columns, "a" and "c", in that order, followed by
two data columns "b" and "d", also in that order.

2.5. Representation Of SQL Indices

Each SQL index, whether explicitly declared via a CREATE INDEX statement
or implied by a UNIQUE or PRIMARY KEY constraint, corresponds to an 
index b-tree in the database file.
Each entry in the index b-tree corresponds to a single row in the 
associated SQL table.
The key to an index b-tree is
a record composed of the columns that are being indexed followed by the
key of the corresponding table row.  For ordinary tables, the row key is
the rowid, and for WITHOUT ROWID tables the row key is the PRIMARY KEY.
Because every row in the table has a unique row key,
all keys in an index are unique.

In a normal index, there is a one-to-one mapping between rows in a 
table and entries in each index associated with that table.
However, in a partial index, the index b-tree only contains entries
corresponding to table rows for which the WHERE clause expression on the
CREATE INDEX statement is true.
Corresponding rows in the index and table b-trees share the same rowid
or primary key values and contain the same value for all indexed columns.

2.5.1. Suppression of redundant columns in WITHOUT ROWID secondary indexes


 In an index on a WITHOUT ROWID table, if a column of the PRIMARY KEY
is also a column in the index and has a matching collating sequence, then the
indexed column is not repeated in the table-key suffix on the
end of the index record.  As an example, consider the following SQL:


CREATE TABLE ex25(a,b,c,d,e,PRIMARY KEY(d,c,a)) WITHOUT rowid;
CREATE INDEX ex25ce ON ex25(c,e);
CREATE INDEX ex25acde ON ex25(a,c,d,e);
CREATE INDEX ex25ae ON ex25(a COLLATE nocase,e);


Each row in the ex25ce index is a record
with these columns: c, e, d, a.  The first two columns are
the columns being indexed, c and e.  The remaining columns are the primary
key of the corresponding table row.  Normally, the primary key would be
columns d, c, and a, but because column c already appears earlier in the
index, it is omitted from the key suffix.

In the extreme case where the columns being indexed cover all columns
of the PRIMARY KEY, the index will consist of only the columns being
indexed.  The ex25acde example above demonstrates this.  Each entry in
the ex25acde index consists of only the columns a, c, d, and e, in that
order.

Each row in ex25ae contains five columns: a, e, d, c, a.  The "a"
column is repeated since the first occurrence of "a" has a collating
function of "nocase" and the second has a collating sequence of "binary".
If the "a" column is not repeated and if the table contains two or more
entries with the same "e" value and where "a" differs only in case, then
all of those table entries would correspond to a single entry in the
index, which would break the one-to-one correspondence between the table
and the index.

 The suppression of redundant columns in the key suffix of an index
entry only occurs in WITHOUT ROWID tables.  In an ordinary rowid table,
the index entry always ends with the rowid even if the INTEGER PRIMARY KEY
column is one of the columns being indexed.



2.6. Storage Of The SQL Database Schema

Page 1 of a database file is the root page of a table b-tree that
holds a special table named "sqlite_schema".  This b-tree is known
as the "schema table" since it stores the complete
database schema.  The structure of the sqlite_schema table is as
if it had been created using the following SQL:


CREATE TABLE sqlite_schema(
  type text,
  name text,
  tbl_name text,
  rootpage integer,
  sql text
);


The sqlite_schema table contains one row for each table, index, view,
and trigger (collectively "objects") in the database schema, except there
is no entry for the sqlite_schema table itself.  The sqlite_schema table
contains entries for internal schema objects in addition to application-
and programmer-defined objects.


The sqlite_schema.type column will be one
of the following text strings:  'table', 'index', 'view', or 'trigger'
according to the type of object defined.  The 'table' string is used
for both ordinary and virtual tables.

The sqlite_schema.name column will hold the name of the object.
UNIQUE and PRIMARY KEY constraints on tables cause SQLite to create
internal indexes with names of the form "sqlite_autoindex_TABLE_N"
where TABLE is replaced by the name of the table that contains the
constraint and N is an integer beginning with 1 and increasing by one
with each constraint seen in the table definition.
In a WITHOUT ROWID table, there is no sqlite_schema entry for the
PRIMARY KEY, but the "sqlite_autoindex_TABLE_N" name is set aside
for the PRIMARY KEY as if the sqlite_schema entry did exist.  This
will affect the numbering of subsequent UNIQUE constraints.
The "sqlite_autoindex_TABLE_N" name is never allocated for an
INTEGER PRIMARY KEY, either in rowid tables or WITHOUT ROWID tables.


The sqlite_schema.tbl_name column holds the name of a table or view
that the object is associated with.  For a table or view, the
tbl_name column is a copy of the name column.  For an index, the tbl_name
is the name of the table that is indexed.  For a trigger, the tbl_name
column stores the name of the table or view that causes the trigger 
to fire.

The sqlite_schema.rootpage column stores the page number of the root
b-tree page for tables and indexes.  For rows that define views, triggers,
and virtual tables, the rootpage column is 0 or NULL.

The sqlite_schema.sql column stores SQL text that describes the
object.  This SQL text is a CREATE TABLE, CREATE VIRTUAL TABLE,
CREATE INDEX,
CREATE VIEW, or CREATE TRIGGER statement that if evaluated against
the database file when it is the main database of a database connection
would recreate the object.  The text is usually a copy of the original
statement used to create the object but with normalizations applied so
that the text conforms to the following rules:


The CREATE, TABLE, VIEW, TRIGGER, and INDEX keywords at the beginning
of the statement are converted to all upper case letters.
The TEMP or TEMPORARY keyword is removed if it occurs after the 
initial CREATE keyword.
Any database name qualifier that occurs prior to the name of the
object being created is removed.
Leading spaces are removed.
All spaces following the first two keywords are converted into a single
space.


The text in the sqlite_schema.sql column is a copy of the original
CREATE statement text that created the object, except normalized as
described above and as modified by subsequent ALTER TABLE statements.
The sqlite_schema.sql is NULL for the internal indexes that are
automatically created by UNIQUE or PRIMARY KEY constraints.

2.6.1. Alternative Names For The Schema Table

The name "sqlite_schema" does not appear anywhere in the file format.
That name is just a convention used by the database implementation.
Due to historical and operational considerations, the
"sqlite_schema" table can also sometimes be called by one of the
following aliases:


 sqlite_master
 sqlite_temp_schema
 sqlite_temp_master


Because the name of the schema table does not appear anywhere in
the file format, the meaning of the database file is not changed if
the application chooses to refer to the schema table by one of
these alternative names.



2.6.2. Internal Schema Objects

In addition to the tables, indexes, views, and triggers created by
the application and/or the developer using CREATE statements SQL, the
sqlite_schema table may contain zero or more entries for 
internal schema objects that are created by SQLite for its 
own internal use.  The names of internal schema objects
always begin with "sqlite_" and any table, index, view, or trigger
whose name begins with "sqlite_" is an internal schema object.
SQLite prohibits applications from creating objects whose names begin
with "sqlite_".  

Internal schema objects used by SQLite may include the following:


Indices with names of the form "sqlite_autoindex_TABLE_N" that
       are used to implement UNIQUE and PRIMARY KEY constraints on
       ordinary tables.

A table with the name "sqlite_sequence" that is used to keep track
       of the maximum historical INTEGER PRIMARY KEY for a table
       using AUTOINCREMENT.

Tables with names of the form "sqlite_statN" where N is an integer.
       Such tables store database statistics gathered by the ANALYZE
       command and used by the query planner to help determine the best
       algorithm to use for each query.


New internal schema objects names, always beginning with "sqlite_",
may be added to the SQLite file format in future releases.



2.6.3. The sqlite_sequence table

The sqlite_sequence table is an internal table used to help implement
AUTOINCREMENT.  The sqlite_sequence table is created automatically
whenever any ordinary table with an AUTOINCREMENT integer primary
key is created.  Once created, the sqlite_sequence table exists in the
sqlite_schema table forever; it cannot be dropped.
The schema for the sqlite_sequence table is:


CREATE TABLE sqlite_sequence(name,seq);


There is a single row in the sqlite_sequence table for each ordinary
table that uses AUTOINCREMENT.  The name of the table (as it appears in
sqlite_schema.name) is in the sqlite_sequence.name field and the largest
INTEGER PRIMARY KEY ever inserted into that table is 
in the sqlite_sequence.seq field.  
New automatically generated integer primary keys for AUTOINCREMENT
tables are guaranteed to be larger than the sqlite_sequence.seq field for
that table.
If the sqlite_sequence.seq field of an AUTOINCREMENT table is already at
the largest integer value (9223372036854775807) then attempts to add new
rows to that table with an automatically generated integer primary will fail
with an SQLITE_FULL error.
The sqlite_sequence.seq field is automatically updated if required when
new entries are inserted to an AUTOINCREMENT table.  
The sqlite_sequence row for an AUTOINCREMENT table is automatically deleted
when the table is dropped.
If the sqlite_sequence row for an AUTOINCREMENT table does not exist when
the AUTOINCREMENT table is updated, then a new sqlite_sequence row is created.
If the sqlite_sequence.seq value for an AUTOINCREMENT table is manually 
set to something other than an integer and there is a subsequent attempt to
insert the or update the AUTOINCREMENT table, then the behavior is undefined.

Application code is allowed to modify the sqlite_sequence table, to add
new rows, to delete rows, or to modify existing rows.  However, application
code cannot create the sqlite_sequence table if it does not already exist.
Application code can delete all entries from the sqlite_sequence table,
but application code cannot drop the sqlite_sequence table.



2.6.4. The sqlite_stat1 table

The sqlite_stat1 is an internal table created by the ANALYZE command
and used to hold supplemental information about tables and indexes that the
query planner can use to help it find better ways of performing queries.
Applications can update, delete from, insert into or drop the sqlite_stat1
table, but may not create or alter the sqlite_stat1 table.
The schema of the sqlite_stat1 table is as follows:


CREATE TABLE sqlite_stat1(tbl,idx,stat);


 There is normally one row per index, with the index identified by the
name in the sqlite_stat1.idx column.  The sqlite_stat1.tbl column is
the name of the table to which the index belongs.  In each such row, 
the sqlite_stat.stat column will be
a string consisting of a list of integers followed by zero or more 
arguments.  The first integer in this
list is the approximate number of rows in the index.  (The number of
rows in the index is the same as the number of rows in the table,
except for partial indexes.)
The second integer is the approximate number of rows in the index
that have the same value in the first column of the index.  The third 
integer is the number of rows in the index that have 
the same value for the first two columns.  The N-th integer (for N>1) 
is the estimated average number of rows in 
the index which have the same value for the first N-1 columns.  For
a K-column index, there will be K+1 integers in the stat column.  If
the index is unique, then the last integer will be 1.

The list of integers in the stat column can optionally be followed
by arguments, each of which is a sequence of non-space characters.
All arguments are preceded by a single space.
Unrecognized arguments are silently ignored.

If the "unordered" argument is present, then the query planner assumes
that the index is unordered and will not use the index for a range query
or for sorting.

The "sz=NNN" argument (where NNN represents a sequence of 1 or more digits)
means that the average row size over all records of the table or
index is NNN bytes per row.  The SQLite query planner might use the
estimated row size information provided by the "sz=NNN" token
to help it choose smaller tables and indexes that require less disk I/O.

The presence of the "noskipscan" token on the sqlite_stat1.stat field
of an index prevents that index from being used with the
skip-scan optimization.

New text tokens may be added to the end of the stat column in future
enhancements to SQLite.  For compatibility, unrecognized tokens at the end
of the stat column are silently ignored.

If the sqlite_stat1.idx column is NULL, then the sqlite_stat1.stat
column contains a single integer which is the approximate number of
rows in the table identified by sqlite_stat1.tbl.
If the sqlite_stat1.idx column is the same as the sqlite_stat1.tbl
column, then the table is a WITHOUT ROWID table and the sqlite_stat1.stat
field contains information about the index btree that implements the
WITHOUT ROWID table.



2.6.5. The sqlite_stat2 table

The sqlite_stat2 is only created and is only used if SQLite is compiled
with SQLITE_ENABLE_STAT2 and if the SQLite version number is between
3.6.18 (2009-09-11) and 3.7.8 (2011-09-19).
The sqlite_stat2 table is neither read nor written by any
version of SQLite before 3.6.18 nor after 3.7.8.
The sqlite_stat2 table contains additional information
about the distribution of keys within an index.
The schema of the sqlite_stat2 table is as follows:


CREATE TABLE sqlite_stat2(tbl,idx,sampleno,sample);


The sqlite_stat2.idx column and the sqlite_stat2.tbl column in each 
row of the sqlite_stat2 table identify an index described by that row.
There are usually 10 rows in the sqlite_stat2
table for each index.

The sqlite_stat2 entries for an index that have sqlite_stat2.sampleno
between 0 and 9 inclusive are samples of the left-most key value in the
index taken at evenly spaced points along the index.
Let C be the number of rows in the index.
Then the sampled rows are given by


     rownumber = (i*C*2 + C)/20


The variable i in the previous expression varies between 0 and 9.
Conceptually, the index space is divided into
10 uniform buckets and the samples are the middle row from each bucket.

The format for sqlite_stat2 is recorded here for legacy reference.  
Recent versions of SQLite no longer support sqlite_stat2 and the
sqlite_stat2 table, if is exists, is simply ignored.



2.6.6. The sqlite_stat3 table

The sqlite_stat3 is only used if SQLite is compiled
with SQLITE_ENABLE_STAT3 or SQLITE_ENABLE_STAT4
and if the SQLite version number is 3.7.9 (2011-11-01) or greater.
The sqlite_stat3 table is neither read nor written by any
version of SQLite before 3.7.9.
If the SQLITE_ENABLE_STAT4 compile-time option is used and the
SQLite version number is 3.8.1 (2013-10-17) or greater,
then sqlite_stat3 might be read but not written.
The sqlite_stat3 table contains additional information
about the distribution of keys within an index, information that the
query planner can use to devise better and faster query algorithms.
The schema of the sqlite_stat3 table is as follows:


CREATE TABLE sqlite_stat3(tbl,idx,nEq,nLt,nDLt,sample);


There are usually multiple entries in the sqlite_stat3 table for each index.
The sqlite_stat3.sample column holds the value of the left-most field of an
index identified by sqlite_stat3.idx and sqlite_stat3.tbl.
The sqlite_stat3.nEq column holds the approximate
number of entries in the index whose left-most column exactly matches
the sample.
The sqlite_stat3.nLt holds the approximate number of entries in the
index whose left-most column is less than the sample.
The sqlite_stat3.nDLt column holds the approximate
number of distinct left-most entries in the index that are less than
the sample.

There can be an arbitrary number of sqlite_stat3 entries per index.
The ANALYZE command will typically generate sqlite_stat3 tables
that contain between 10 and 40 samples that are distributed across
the key space and with large nEq values.

In a well-formed sqlite_stat3 table, the samples for any single
index must appear in the same order that they occur in the index.  
In other words, if the entry with left-most column S1 is earlier in
the index b-tree than the
entry with left-most column S2, then in the sqlite_stat3 table, 
sample S1 must have a smaller rowid than sample S2.



2.6.7. The sqlite_stat4 table

The sqlite_stat4 is only created and is only used if SQLite is compiled
with SQLITE_ENABLE_STAT4 and if the SQLite version number is
3.8.1 (2013-10-17) or greater.
The sqlite_stat4 table is neither read nor written by any
version of SQLite before 3.8.1.
The sqlite_stat4 table contains additional information
about the distribution of keys within an index or the distribution of
keys in the primary key of a WITHOUT ROWID table.
The query planner can sometimes use the additional information in
the sqlite_stat4 table to devise better and faster query algorithms.
The schema of the sqlite_stat4 table is as follows:


CREATE TABLE sqlite_stat4(tbl,idx,nEq,nLt,nDLt,sample);


There are typically between 10 to 40 entries in the sqlite_stat4 table for
each index for which statistics are available, however these limits are
not hard bounds.
The meanings of the columns in the sqlite_stat4 table are as follows:



tbl:
    The sqlite_stat4.tbl column holds name of the table that owns
    the index that the row describes

idx:
    The sqlite_stat4.idx column holds name of the index that the
    row describes, or in the case of
    an sqlite_stat4 entry for a WITHOUT ROWID table, the
    name of the table itself.

sample:
    The sqlite_stat4.sample column holds a BLOB
    in the record format that encodes the indexed columns followed by
    the rowid for a rowid table or by the columns of the primary key 
    for a WITHOUT ROWID table.
    The sqlite_stat4.sample BLOB for the WITHOUT ROWID table itself 
    contains just the columns of the primary key.
    Let the number of columns encoded by the sqlite_stat4.sample blob be N.
    For indexes on an ordinary rowid table, N will be one more than the number 
    of columns indexed.
    For indexes on WITHOUT ROWID tables, N will be the number of columns
    indexed plus the number of columns in the primary key.
    For a WITHOUT ROWID table, N will be the number of columns in the
    primary key.

nEq:
    The sqlite_stat4.nEq column holds a list of N integers where 
    the K-th integer is the approximate number of entries in the index
    whose left-most K columns exactly match the K left-most columns
    of the sample.

nLt:
    The sqlite_stat4.nLt column holds a list of N integers where
    the K-th integer is the approximate number of entries in the
    index whose K left-most columns are collectively less than the 
    K left-most columns of the sample.

nDLt:
    The sqlite_stat4.nDLt column holds a list of N integers where
    the K-th integer is the approximate
    number of entries in the index that are distinct in the first K columns and
    where the left-most K columns are collectively less than the left-most
    K columns of the sample.



The sqlite_stat4 is a generalization of the sqlite_stat3 table.  The
sqlite_stat3 table provides information about the left-most column of an
index whereas the sqlite_stat4 table provides information about all columns
of the index.

There can be an arbitrary number of sqlite_stat4 entries per index.
The ANALYZE command will typically generate sqlite_stat4 tables
that contain between 10 and 40 samples that are distributed across
the key space and with large nEq values.

In a well-formed sqlite_stat4 table, the samples for any single
index must appear in the same order that they occur in the index.  
In other words, if entry S1 is earlier in the index b-tree than 
entry S2, then in the sqlite_stat4 table, sample S1 must have a
smaller rowid than sample S2.



3. The Rollback Journal

The rollback journal is a file associated with each SQLite database
file that holds information used to restore the database file to its initial
state during the course of a transaction.
The rollback journal file is always located in the same 
directory as the database
file and has the same name as the database file but with the string
"-journal" appended.  There can only be a single rollback journal
associated with a give database and hence there can only be one write
transaction open against a single database at one time.

If a transaction is aborted due to an application crash, an operating
system crash, or a hardware power failure or crash, then the database may
be left in an inconsistent state.  The next time SQLite attempts to open
the database file, the presence of the rollback journal file will be 
detected and the journal will be automatically played back to restore the
database to its state at the start of the incomplete transaction.

A rollback journal is only considered to be valid if it exists and
contains a valid header.  Hence a transaction can be committed in one
of three ways:

The rollback journal file can be deleted,
The rollback journal file can be truncated to zero length, or
The header of the rollback journal can be overwritten with
invalid header text (for example, all zeros).


These three ways of committing a transaction correspond to the DELETE,
TRUNCATE, and PERSIST settings, respectively, of the journal_mode pragma.



A valid rollback journal begins with a header in the following format:


Rollback Journal Header Format

OffsetSizeDescription
0
    8
    Header string:  0xd9, 0xd5, 0x05, 0xf9, 0x20, 0xa1, 0x63, 0xd7
8
    4
    The "Page Count" - The number of pages in the next segment of the 
        journal, or -1 to
        mean all content to the end of the file
12
    4
    A random nonce for the checksum
16
    4
    Initial size of the database in pages
20
    4
    Size of a disk sector assumed by the process that wrote this
        journal.
24
    4
    Size of pages in this journal.



A rollback journal header is padded with zeros out to the size of a 
single sector (as defined by the sector size integer at offset 20).
The header is in a sector by itself so that if a power loss occurs while
writing the sector, information that follows the header will be
(hopefully) undamaged.

After the header and zero padding are zero or more page records.  Each
page record stores a copy of the content of a page from the database file
before it was changed.  The same page may not appear more than once
within a single rollback journal.
To rollback an incomplete transaction, a process
has merely to read the rollback journal from beginning to end and
write pages found in the journal back into the database file at the
appropriate location.

Let the database page size (the value of the integer at offset 24 
in the journal header) be N.
Then the format of a page record is as follows:


Rollback Journal Page Record Format

OffsetSizeDescription
0
    4
    The page number in the database file
4
    N
    Original content of the page prior to the start of the transaction
N+4
    4
    Checksum




The checksum is an unsigned 32-bit integer computed as follows:


Initialize the checksum to the checksum nonce value found in the
journal header at offset 12.
Initialize index X to be N-200 (where N is the size of a database page
in bytes.
Interpret the byte at offset X into the page as an 8-bit unsigned integer
 and add the value of that integer to the checksum.
Subtract 200 from X.
If X is greater than or equal to zero, go back to step 3.


The checksum value is used to guard against incomplete writes of
a journal page record following a power failure.  A different random nonce
is used each time a transaction is started in order to minimize the risk
that unwritten sectors might by chance contain data from the same page
that was a part of prior journals.  By changing the nonce for each
transaction, stale data on disk will still generate an incorrect checksum
and be detected with high probability.  The checksum only uses a sparse sample
of 32-bit words from the data record for performance reasons - design studies 
during the planning phases of SQLite 3.0.0 showed
a significant performance hit in checksumming the entire page.

Let the page count value at offset 8 in the journal header be M.
If M is greater than zero then after M page records the journal file
may be zero padded out to the next multiple of the sector size and another
journal header may be inserted.  All journal headers within the same
journal must contain the same database page size and sector size.

If M is -1 in the initial journal header, then the number of page records
that follow is computed by computing how many page records will fit in
the available space of the remainder of the journal file.



4. The Write-Ahead Log

Beginning with version 3.7.0 (2010-07-21), 
SQLite supports a new transaction
control mechanism called "write-ahead log" or "WAL".
When a database is in WAL mode, all connections to that database must
use the WAL.  A particular database will use either a rollback journal
or a WAL, but not both at the same time.
The WAL is always located in the same directory as the database
file and has the same name as the database file but with the string
"-wal" appended.

4.1. WAL File Format

A WAL file consists of a header followed by zero or more "frames".
Each frame records the revised content of a single page from the
database file.  All changes to the database are recorded by writing
frames into the WAL.  Transactions commit when a frame is written that
contains a commit marker.  A single WAL can and usually does record 
multiple transactions.  Periodically, the content of the WAL is
transferred back into the database file in an operation called a
"checkpoint".

A single WAL file can be reused multiple times.  In other words, the
WAL can fill up with frames and then be checkpointed and then new
frames can overwrite the old ones.  A WAL always grows from beginning
toward the end.  Checksums and counters attached to each frame are
used to determine which frames within the WAL are valid and which
are leftovers from prior checkpoints.

The WAL header is 32 bytes in size and consists of the following eight
big-endian 32-bit unsigned integer values:


WAL Header Format

OffsetSizeDescription
04
    Magic number.  0x377f0682 or 0x377f0683
44
    File format version.  Currently 3007000.
84
    Database page size.  Example: 1024
124
    Checkpoint sequence number
164
    Salt-1: random integer incremented with each checkpoint
204
    Salt-2: a different random number for each checkpoint
244
    Checksum-1: First part of a checksum on the first 24 bytes of header
284
    Checksum-2: Second part of the checksum on the first 24 bytes of header



Immediately following the wal-header are zero or more frames. Each
frame consists of a 24-byte frame-header followed by a page-size bytes
of page data. The frame-header is six big-endian 32-bit unsigned 
integer values, as follows:


WAL Frame Header Format

OffsetSizeDescription
04
    Page number
44
    For commit records, the size of the database file in pages
        after the commit.  For all other records, zero.
84
    Salt-1 copied from the WAL header
124
    Salt-2 copied from the WAL header
164
    Checksum-1:  Cumulative checksum up through and including this page
204
    Checksum-2:  Second half of the cumulative checksum.



A frame is considered valid if and only if the following conditions are
true:


The salt-1 and salt-2 values in the frame-header match
       salt values in the wal-header

The checksum values in the final 8 bytes of the frame-header
       exactly match the checksum computed consecutively on the
       first 24 bytes of the WAL header and the first 8 bytes and
       the content of all frames
       up to and including the current frame.




4.2. Checksum Algorithm

The checksum is computed by interpreting the input as
an even number of unsigned 32-bit integers: x(0) through x(N).
The 32-bit integers are big-endian if the
magic number in the first 4 bytes of the WAL header is 0x377f0683 and
the integers are little-endian if the magic number is 0x377f0682.
The checksum values are always stored in the frame header in a
big-endian format regardless of which byte order is used to compute
the checksum.

The checksum algorithm only works for content which is a multiple of
8 bytes in length.  In other words, if the inputs are x(0) through x(N)
then N must be odd.
The checksum algorithm is as follows:

 
s0 = s1 = 0
for i from 0 to n-1 step 2:
   s0 += x(i) + s1;
   s1 += x(i+1) + s0;
endfor
# result in s0 and s1


The outputs s0 and s1 are both weighted checksums using Fibonacci weights
in reverse order.  (The largest Fibonacci weight occurs on the first element
of the sequence being summed.)  The s1 value spans all 32-bit integer
terms of the sequence whereas s0 omits the final term.

4.3. Checkpoint Algorithm

On a checkpoint, the WAL is first flushed to persistent storage using
the xSync method of the VFS. 
Then valid content of the WAL is transferred into the database file.
Finally, the database is flushed to persistent storage using another
xSync method call.
The xSync operations serve as write barriers - all writes launched
before the xSync must complete before any write that launches after the
xSync begins.

A checkpoint need not run to completion.  It might be that some
readers are still using older transactions with data that is contained
in the database file.  In that case, transferring content for newer
transactions from the WAL file into the database would delete the content
out from under readers still using the older transactions.  To avoid that,
checkpoints only run to completion if all reader are using the
last transaction in the WAL.



4.4. WAL Reset

After a complete checkpoint, if no other connections are in transactions
that use the WAL, then subsequent write transactions can
overwrite the WAL file from the beginning.  This is called "resetting the
WAL".  At the start of the first new
write transaction, the WAL header salt-1 value is incremented
and the salt-2 value is randomized.  These changes to the salts invalidate
old frames in the WAL that have already been checkpointed but not yet
overwritten, and prevent them from being checkpointed again.

The WAL file can optionally be truncated on a reset, but it need not be.
Performance is usually a little better if the WAL is not truncated, since
filesystems generally will overwrite an existing file faster than they
will grow a file.



4.5. Reader Algorithm

To read a page from the database (call it page number P), a reader
first checks the WAL to see if it contains page P.  If so, then the
last valid instance of page P that is followed by a commit frame
or is a commit frame itself becomes the value read.  If the WAL
contains no copies of page P that are valid and which are a commit
frame or are followed by a commit frame, then page P is read from
the database file.

To start a read transaction, the reader records the number of value
frames in the WAL as "mxFrame".  (More detail)
The reader uses this recorded mxFrame value
for all subsequent read operations.  New transactions can be appended
to the WAL, but as long as the reader uses its original mxFrame value
and ignores subsequently appended content, the reader will see a 
consistent snapshot of the database from a single point in time.  
This technique allows multiple concurrent readers to view different 
versions of the database content simultaneously.

The reader algorithm in the previous paragraphs works correctly, but 
because frames for page P can appear anywhere within the WAL, the
reader has to scan the entire WAL looking for page P frames.  If the
WAL is large (multiple megabytes is typical) that scan can be slow,
and read performance suffers.  To overcome this problem, a separate
data structure called the wal-index is maintained to expedite the
search for frames of a particular page.



4.6. WAL-Index Format

Conceptually, the wal-index is shared memory, though the current
VFS implementations use a memory-mapped file for operating-system
portability.  The memory-mapped
file is in the same directory as the database and has the same name
as the database with a "-shm" suffix appended.  Because
the wal-index is shared memory, SQLite does not support 
journal_mode=WAL 
on a network filesystem when clients are on different machines, as
all clients of the database must be able to share the same memory.

The purpose of the wal-index is to answer this question quickly:


Given a page number P and a maximum WAL frame index M,
return the largest WAL frame index for page P that does not exceed M, 
or return NULL if there are no frames for page P that do not exceed M.


The M value in the previous paragraph is the "mxFrame" value
defined in section 4.4 that is read at the start 
of a transaction and which defines the maximum frame from the WAL that 
the reader will use.

The wal-index is transient.  After a crash, the wal-index is
reconstructed from the original WAL file.  The VFS is required
to either truncate or zero the header of the wal-index when the last
connection to it closes.  Because the wal-index is transient, it can
use an architecture-specific format; it does not have to be cross-platform.
Hence, unlike the database and WAL file formats which store all values
as big endian, the wal-index stores multi-byte values in the native
byte order of the host computer.

This document is concerned with the persistent state of the database
file, and since the wal-index is a transient structure, no further 
information about the format of the wal-index will be provided here.
Additional details on the format of the wal-index are contained in
the separate WAL-index File Format document.


c2: Hints for Debugging SQLite
c5: debugging.html
c4:
Debugging Hints


The following is a random assortment of techniques used by the
SQLite developers to trace, examine, and understand the behavior of the
core SQLite library.


These techniques are designed to aid in understanding the
core SQLite library itself, not applications that merely use SQLite.



Use the ".eqp full" option on the command-line shell

When you have a SQL script that you are debugging or trying
to understand, it is often useful to run it in the command-line shell
with the ".eqp full" setting.  When ".eqp" is set to FULL, the shell
automatically shows the EXPLAIN and EXPLAIN QUERY PLAN output for
each command prior to actually running that command.

For added readability, also set ".echo on" so that the output contains
the original SQL text.

The newer ".eqp trace" command does everything that ".eqp full" does
and also turns on VDBE tracing.



Use compile-time options to enable debugging features.

Suggested compile-time options include:

-DSQLITE_DEBUG
-DSQLITE_ENABLE_EXPLAIN_COMMENTS
-DSQLITE_ENABLE_TREETRACE
-DSQLITE_ENABLE_WHERETRACE



The SQLITE_ENABLE_TREETRACE and SQLITE_ENABLE_WHERETRACE options
are not documented in compile-time options document because they
are not officially supported.  What they do is activate the
".treetrace" and ".wheretrace" dot-commands in the command-line
shell, which provide low-level tracing output for the logic that
generates code for SELECT and DML statements and WHERE clauses, respectively.


Call sqlite3ShowExpr() and similar from the debugger.

When compiled with SQLITE_DEBUG, SQLite includes routines that will
print out various internal abstract syntax tree structures as ASCII-art graphs.
This can be very useful in a debugging in order to understand the variables
that SQLite is working with.  The following routines are available:


  void sqlite3ShowExpr(const Expr*);
  void sqlite3ShowExprList(const ExprList*);
  void sqlite3ShowIdList(const IdList*);
  void sqlite3ShowSrcList(const SrcList*);
  void sqlite3ShowSelect(const Select*);
  void sqlite3ShowWith(const With*);
  void sqlite3ShowUpsert(const Upsert*);
  void sqlite3ShowTrigger(const Trigger*);
  void sqlite3ShowTriggerList(const Trigger*);
  void sqlite3ShowTriggerStep(const TriggerStep*);
  void sqlite3ShowTriggerStepList(const TriggerStep*);
  void sqlite3ShowWindow(const Window*);
  void sqlite3ShowWinFunc(const Window*);


These routines are not APIs and are subject to change.  They are
for interactive debugging use only.



Breakpoints on test_addoptrace

When debugging the bytecode generator, it is often useful to know
where a particular opcode is being generated.  To find this easily,
run the script in a debugger.  Set a breakpoint on the "test_addoptrace"
routine.  Then run the "PRAGMA vdbe_addoptrace=ON;" followed by the
SQL statement in question.  Each opcode will be displayed as it is
appended to the VDBE program, and the breakpoint will fire immediately
thereafter.  Step until reaching the opcode then look backwards
in the stack to see where and how it was generated.

This only works when compiled with SQLITE_DEBUG.



Using the ".treetrace" and ".wheretrace" shell commands

When the command-line shell and the core SQLite library are
both compiled with SQLITE_DEBUG and
SQLITE_ENABLE_TREETRACE and SQLITE_ENABLE_WHERETRACE, then the
shell has two commands used to turn on debugging facilities for the
most intricate parts of the code generator - the logic dealing with
SELECT statements and WHERE clauses, respectively.
The ".treetrace" and ".wheretrace" commands each take a numeric
argument which can be expressed in hexadecimal.  Each bit turns on
various parts of debugging.  Values of "0xfff" and "0xff" are commonly
used.  Use an argument of "0" to turn all tracing output back off.



Using the ".breakpoint" shell command

The ".breakpoint" command in the CLI does nothing but invoke the
procedure named "test_breakpoint()", which is a no-op.

If you have a script and you want to start debugging at some point
half-way through that script, simply set a breakpoint in gdb (or whatever
debugger you are using) on the test_breakpoint() function, and add a
".breakpoint" command where you want to stop.  When you reach that first
breakpoint, set whatever additional breakpoints are variable traces you
need.


Disable the lookaside memory allocator

When looking for memory allocation problems (memory leaks, use-after-free
errors, buffer overflows, etc) it is sometimes useful to disable the
lookaside memory allocator then run the test under valgrind or MSAN or
some other heap memory debugging tool.
The lookaside memory allocator can 
be disabled at start-time using the SQLITE_CONFIG_LOOKASIDE
interface.  The command-line shell will use that interface to
disable lookaside if it is started with the "--lookaside 0 0"
command line option.


This page last modified on  2022-04-18 02:55:50 UTC


c2: Run-Time Loadable Extensions
c5: loadext.html
c4:
1. Overview

SQLite has the ability to load extensions (including new
application-defined SQL functions,
collating sequences, virtual tables, and VFSes) at run-time.
This feature allows the code for extensions to be developed and
tested separately from the application and then loaded
on an as-needed basis.

Extensions can also be statically linked with the application.
The code template shown below will work just as well as a statically
linked extension as it does as a run-time loadable extension except that
you should give the entry point function ("sqlite3_extension_init")
a different name to avoid name collisions if your application contains
two or more extensions.

2. Loading An Extension

An SQLite extension is a shared library or DLL.  To load it, you
need to supply SQLite with the name of the file containing the
shared library or DLL and an entry point to initialize the extension.
In C code, this information is supplied using the
sqlite3_load_extension() API.  See the documentation on that
routine for additional information.

Note that different operating systems use different filename
suffixes for their shared libraries.  Windows uses ".dll", Mac uses
".dylib", and most unixes other than mac use ".so".  If you want to
make your code portable, you can omit the suffix from the shared
library filename and the appropriate suffix will be added automatically
by the sqlite3_load_extension() interface.

There is also an SQL function that can be used to load extensions:
load_extension(X,Y).  It works just like the sqlite3_load_extension()
C interface.

Both methods for loading an extension allow you to specify
the name of an entry point for the extension.  
You can leave this argument blank - passing in
a NULL pointer for the sqlite3_load_extension() C-language interface
or omitting the second argument for the load_extension() SQL interface -
and the extension loader logic will attempt to figure out the entry point
on its own.  It will first try the generic extension name
"sqlite3_extension_init".  If that does not work, it constructs a
entry point using the template "sqlite3_X_init" where the X is replaced
by the lowercase equivalent of every ASCII character in the filename
after the last "/" and before the first following "." omitting the
first three characters if they happen to be "lib".  So, for example,
if the filename is "/usr/lib/libmathfunc-4.8.so" the entry point name
would be "sqlite3_mathfunc_init".  Or if the filename is 
"./SpellFixExt.dll" then the entry point would be called
"sqlite3_spellfixext_init".

For security reasons, extension loading is turned off by default.
In order to use either the C-language or SQL extension loading functions,
one must first enable extension loading using the
sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_LOAD_EXTENSION,1,NULL)
C-language API in your application.

From the command-line shell, extensions can be loaded using the
".load" dot-command.  For example:


.load ./YourCode


Note that the command-line shell program has already enabled
extension loading for you (by calling the sqlite3_enable_load_extension()
interface as part of its setup) so the command above works without
any special switches, setup, or other complications.

The ".load" command with one argument invokes sqlite3_load_extension()
with the zProc parameter set to NULL, causing SQLite to first look for
an entry point named "sqlite3_extension_init" and then "sqlite3_X_init"
where "X" is derived from the filename.  If your extension has an entry
point with a different name, simply supply that name as the second
argument.  For example:


.load ./YourCode nonstandard_entry_point




3. Compiling A Loadable Extension

Loadable extensions are C-code.  To compile them on
most unix-like operating
systems, the usual command is something like this:


gcc -g -fPIC -shared YourCode.c -o YourCode.so


Macs are unix-like, but they do not follow the usual shared library
conventions.  To compile a shared library on a Mac, use a command like
this:


gcc -g -fPIC -dynamiclib YourCode.c -o YourCode.dylib


If when you try to load your library you get back an error message
that says "mach-o, but wrong architecture" then you might need to add
command-line options "-arch i386" or "arch x86_64" to gcc, depending
on how your application is built.

To compile on Windows using MSVC, a command similar to the following
will usually work:


cl YourCode.c -link -dll -out:YourCode.dll


To compile for Windows using MinGW, the command line is just like it
is for unix except that the output file suffix is changed to ".dll" and
the -fPIC argument is omitted:


gcc -g -shared YourCode.c -o YourCode.dll




4. Programming Loadable Extensions

A template loadable extension contains the following three elements:



Use "#include &lt;sqlite3ext.h&gt;" at the top of your source
code files instead of "#include &lt;sqlite3.h&gt;".



Put the macro "SQLITE_EXTENSION_INIT1" on a line by itself 
right after the "#include &lt;sqlite3ext.h&gt;" line.



Add an extension loading entry point routine that looks like 
something the following:

#ifdef _WIN32
__declspec(dllexport)
#endif
int sqlite3_extension_init( /* &lt;== Change this name, maybe */
  sqlite3 *db, 
  char **pzErrMsg, 
  const sqlite3_api_routines *pApi
){
  int rc = SQLITE_OK;
  SQLITE_EXTENSION_INIT2(pApi);
  /* insert code to initialize your extension here */
  return rc;
}


You will do well to customize the name of your entry point to
correspond to the name of the shared library you will be generating,
rather than using the generic "sqlite3_extension_init" name.  Giving
your extension a custom entry point name will enable you to statically
link two or more extensions into the same program without a linker
conflict, if you later decide to use static linking rather than run-time
linking.
If your shared library ends up being named "YourCode.so" or
"YourCode.dll" or "YourCode.dylib" as shown in the compiler examples
above, then the correct entry point name would be
"sqlite3_yourcode_init".


Here is a complete template extension that you can copy/paste 
to get started:

/* Add your header comment here */
#include &lt;sqlite3ext.h&gt; /* Do not use &lt;sqlite3.h&gt;! */
SQLITE_EXTENSION_INIT1

/* Insert your extension code here */

#ifdef _WIN32
__declspec(dllexport)
#endif
/* TODO: Change the entry point name so that "extension" is replaced by
** text derived from the shared library filename as follows:  Copy every
** ASCII alphabetic character from the filename after the last "/" through
** the next following ".", converting each character to lowercase, and
** discarding the first three characters if they are "lib".
*/
int sqlite3_extension_init(
  sqlite3 *db, 
  char **pzErrMsg, 
  const sqlite3_api_routines *pApi
){
  int rc = SQLITE_OK;
  SQLITE_EXTENSION_INIT2(pApi);
  /* Insert here calls to
  **     sqlite3_create_function_v2(),
  **     sqlite3_create_collation_v2(),
  **     sqlite3_create_module_v2(), and/or
  **     sqlite3_vfs_register()
  ** to register the new features that your extension adds.
  */
  return rc;
}


4.1. Example Extensions

Many examples of complete and working loadable extensions can be 
seen in the SQLite source tree in the
ext/misc subdirectory.
Each file in that directory is a separate extension.  Documentation
is provided by a header comment on the file.
Here are brief notes on a few of the extensions in 
the ext/misc subdirectory:



carray.c &mdash;
Implementation of the carray table-valued function.

compress.c &mdash;
Implementation of application-defined SQL functions compress() and
uncompress() that do zLib compression of text or blob content.

json1.c &mdash;
Implementation of JSON SQL functions and table-valued functions.
This is a larger and more complex extension.

memvfs.c &mdash;
Implementation of a new VFS that stores all content in-memory.

rot13.c &mdash;
Implementation of a rot13() 
SQL function.  This is a very simple example of an extension function
and is useful as a template for creating new extensions.

series.c &mdash;
Implementation of the generate_series virtual table and
table-valued function.  This is a relatively simple example of a
virtual table implementation which can serve as a template for writing
new virtual tables.


Other and more complex extensions can be found in subfolders
under ext/ other than ext/misc/.



5. Persistent Loadable Extensions

The default behavior for a loadable extension is that it is unloaded
from process memory when the database connection that originally invoked
sqlite3_load_extension() closes.  (In other words, the xDlClose method
of the sqlite3_vfs object is called for all extensions when a database
connection closes.)  However, if the initialization procedure returns
SQLITE_OK_LOAD_PERMANENTLY instead of SQLITE_OK, then the extension will
not be unloaded (xDlClose will not be invoked) and the extension will remain
in process memory indefinitely.  The SQLITE_OK_LOAD_PERMANENTLY return
value is useful for extensions that want to register new VFSes.

To clarify: an extension for which the initialization function returns
SQLITE_OK_LOAD_PERMANENTLY continues to exist in memory after the database
connection closes.  However, the extension is not automatically
registered with subsequent database connections.  This makes it possible
to load extensions that implement new VFSes.
To persistently load and register an extension that implements new SQL
functions, collating sequences, and/or virtual tables, such that those
added capabilities are available to all subsequent database connections,
then the initialization routine should also invoke sqlite3_auto_extension()
on a subfunction that will register those services.

The vfsstat.c extension
show an example of a loadable extension that persistently registers both
a new VFS and a new virtual table.  The
sqlite3_vfsstat_init()
initialization routine in that extension is called only once, when the
extension is first loaded.  It registers the new "vfslog" VFS just that
one time, and it returns SQLITE_OK_LOAD_PERMANENTLY so that the code used
to implement the "vfslog" VFS will remain in memory. The initialization routine
also invokes sqlite3_auto_extension() on a pointer to the "vstatRegister()"
function so that all subsequent database connections will invoke the
"vstatRegister()" function as they start up, and hence register the
"vfsstat" virtual table.



6. Statically Linking A Run-Time Loadable Extension

The exact same source code can be used for both a run-time loadable
shared library or DLL and as a module that is statically linked with your
application.  This provides flexibility and allows you to reuse the same
code in different ways.

To statically link your extension, simply add the -DSQLITE_CORE
compile-time option.  The SQLITE_CORE macro causes the SQLITE_EXTENSION_INIT1
and SQLITE_EXTENSION_INIT2 macros to become no-ops.  Then modify your
application to invoke the entry point directly, passing in a NULL pointer
as the third "pApi" parameter.

It is particularly important to use an entry point name that is
based on the extension filename, rather than the generic
"sqlite3_extension_init" entry point name, if you will be statically
linking two or more extensions.  If you use the generic name, there
will be multiple definitions of the same symbol and the link will fail.

If you will be opening multiple database connections in your application,
rather than invoking the extension entry points for each database
connection separately, you might want to consider using the
sqlite3_auto_extension() interface to register your extensions and
to cause them to be automatically started as each database connection
is opened.  You only have to register each extension once, and you can
do so near the beginning of your main() routine.  Using the
sqlite3_auto_extension() interface to register your extensions makes
your extensions work as if they were built into the core SQLite - they
automatically exist whenever you open a new database connection
without needing to be initialized.  Just be sure to complete any
configuration you need to accomplish using sqlite3_config() before
registering your extensions, since the sqlite3_auto_extension()
interface implicitly calls sqlite3_initialize().

7. Implementation Details

SQLite implements run-time extension loading using the
xDlOpen(), xDlError(), xDlSym(), and xDlClose() methods of the
sqlite3_vfs object.  These methods are implemented using
the dlopen() library on unix (which explains why SQLite commonly
needs to be linked against the "-ldl" library on unix systems)
and using LoadLibrary() API on Windows.  In a custom VFS for
unusual systems, these methods can all be omitted, in which case
the run-time extension loading mechanism will not work (though
you will still be able to statically link the extension code, assuming
the entry pointers are uniquely named).
SQLite can be compiled with
SQLITE_OMIT_LOAD_EXTENSION to omit the extension loading code
from the build.
This page last modified on  2023-10-20 22:32:30 UTC


c2: Powersafe Overwrite
c5: psow.html
c4:
Powersafe Overwrite

"Powersafe overwrite" is a term used by the SQLite team to describe
a behavior of some filesystems and disk-controllers related to
data preservation during a power loss.  Powersafe overwrite
is a boolean property: either the storage system has it or it does not.

We say that a system has the powersafe overwrite property if the following
statement is true:


  When an application writes a range of bytes in a file, no
  bytes outside of that range will change, even if the write occurs
  just before a crash or power failure.


The powersafe overwrite property says nothing about the state of the
bytes that were written.  Those bytes might contain their old values,
their new values, random values, or some combination of these.  The powersafe
overwrite property merely states that writes cannot change bytes outside
of the range of bytes written.

In other words, powersafe overwrite means that there is no "collateral
damage" when a power loss occurs while writing.  Only those bytes actually
being written might be damaged.

In practical terms, what the powersafe write property means is that when
the disk controller detects an impending power loss, it finishes writing
whatever sector it is working on prior to parking the heads.  It means that
individual sector writes will complete once started, even if
there is a power loss.

Consider what would happen if disk sector writes are interrupted
by a power loss.  If an application writes two or three bytes in the middle
of some file, the operating system will implement this by first reading
the entire sector containing those bytes, making the change to the
sector in memory, then writing the entire sector back to the disk.  If a power
loss occurs during the writeback and the sector was not completely written,
then on the next read after reboot, error correcting codes
in the sector will probably detect irreparable damage and the disk 
controller will read out the sector as all zeros or all ones.  Thus
values will have changed outside of the range of the two or three bytes 
that were written at the application level - a violation of the powersafe
overwrite property.

SQLite Assumptions About Powersafe Overwrite

All versions of SQLite up to and including version 3.7.9
(2011-11-01) assume that
the filesystem does not provide powersafe overwrite.  SQLite 
has traditionally assumed that when any one byte of a file changes, all
other bytes within the same sector of that byte have the potential of
being corrupted on a power loss.  When writing, SQLite has made sure
to journal all bytes in the same sector of any modifications
and it pads journal files out to the next sector boundary so that
subsequent appends to that journal cannot damage prior records.
SQLite understands the sector size to be the value returned by the
xSectorSize method in the VFS.  The SQLite team has often referred
to the value returned by xSectorSize as the "blast radius" of a write,
since it expresses the range of bytes that might be damaged if a power
loss occurs during the write.
The default VFSes for unix and windows have always returned 512 as 
the sector size (or blast radius) for all versions of SQLite up to
and including version 3.7.9.

Newer disk drives have begun using 4096 byte sectors however.  Beginning
with SQLite version 3.7.10 (2012-01-16), 
the SQLite development team experimented with 
changes xSectorSize to report 4096 bytes as the blast radius.
This had the effect of increasing write overhead on
many databases.  For a database with a PRAGMA page_size of 1024
(a very common choice) making a change to a single page in the database
now requires SQLite to backup three other adjacent pages to the rollback
journal, whereas formerly it only had to backup the one page that was
changing.  In WAL mode, each transaction had to be padded out to the
next 4096-byte boundary in the WAL file, rather than the next 512-byte
boundary, resulting in thousands of extra bytes being written
per transaction.

The extra write overhead prompted a reexamination of assumptions about
powersafe overwrite.  With modern disk drives, the capacity has become
so large and the data density so great that a single sector is very
small and writing a single sector takes very little time.  We know that
disk drives can detect an impending power loss and continue
to operate for some small amount of time on residual energy because those
drives are able to park their heads before spinning down.  And
so if an impending power loss is detectable by the disk controller, it
seems reasonable that the controller will finish writing
whatever sector it is current working on when the imminent power loss 
is first detected, prior to parking the heads, as long as doing so
does not take too long, which it should not with
small and dense sectors.  Hence it seems reasonable
to assume powersafe overwrite for modern disks.  Indeed, BerkeleyDB has
made this assumption for decades, we are told.  Caution is advised
though. As Roger Binns noted on the SQLite developers mailing list:
"'poorly written' should be the main assumption about drive firmware."



Torn Pages

A torn page occurs when a database page is larger than a disk sector,
the database page is written to disk, but a power loss occurs prior to
all sectors of the database page being written.  Then, upon recovery, part of
the database page will have the old content while some other parts of the
page will have the new content.  Some database engines assume that 
page writes are atomic and hence a torn page is an unrecoverable error.


SQLite never assumes that database page writes are atomic,
regardless of the PSOW setting.(1)
And hence SQLite is always able to automatically recover from torn pages
induced by a crash.  Enabling PSOW does not decrease SQLite's ability
to recover from a torn page.

Changes In SQLite Version 3.7.10

The VFS for SQLite version 3.7.10 (2012-01-16)
adds a new device characteristic 
named SQLITE_IOCAP_POWERSAFE_OVERWRITE.  Database files that report this
characteristic are assumed to reside on storage systems that have the
powersafe overwrite property.
The default unix and windows VFSes now report
SQLITE_IOCAP_POWERSAFE_OVERWRITE if SQLite is compiled with
-DSQLITE_POWERSAFE_OVERWRITE=1 or they
make the legacy assumption that storage does not have the powersafe
overwrite property if compiled with
-DSQLITE_POWERSAFE_OVERWRITE=0.
For now, the default is for powersafe overwrite to be turned on, though
we may revisit this in the future and default it off.

The powersafe overwrite property for individual databases can be
specified as the database is opened using the "psow" query parameter
with a URI filename.  For example, to always assume powersafe
overwrite for a file (perhaps to ensure maximum write performance), 
open it as


   file:somefile.db?psow=1


Or to be extra safe with a database and to force SQLite to assume the
database lacks powersafe overwrite, open it using


   file:somefile.db?psow=0


There is also a new SQLITE_FCNTL_POWERSAFE_OVERWRITE opcode for
the sqlite3_file_control() that allows
an application to query the powersafe overwrite property for a database
file.


Notes:

SQLite never assumes atomic page writes in its default configurations.
But a custom VFS can set one of the 
SQLITE_IOCAP_ATOMIC bits in the result of the xDeviceCharacteristic()
method and then SQLite will assume that page writes are atomic.  The
application must supply a custom VFS to accomplish this, however, since
none of the standard VFSes will ever set any of the atomic bits in the
xDeviceCharacteristics() vector.

This page last modified on  2022-01-08 05:02:57 UTC


c2: How To Corrupt An SQLite Database File
c5: howtocorrupt.html
c4:
Overview 

An SQLite database is highly resistant to corruption.
If an application crash, or an operating-system crash, or even
a power failure occurs in the middle of a transaction, the partially
written transaction should be automatically rolled back the next time
the database file is accessed.  The recovery process is fully
automatic and does not require any action on the part of the user
or the application.


Though SQLite is resistant to database corruption, it is not immune.
This document describes the various ways that an SQLite database might
go corrupt.

1.  File overwrite by a rogue thread or process

SQLite database files are ordinary disk files.
That means that any process can open the file and 
overwrite it with garbage.  There is nothing that the SQLite
library can do to defend against this.



1.1.  Continuing to use a file descriptor after it has been closed

We have seen multiple cases where a file descriptor was open on a file,
then that file descriptor was closed and reopened on an SQLite database.
Later, some other thread continued to write into the
old file descriptor, not realizing that the original file had been closed
already.  But because the file descriptor had been reopened by SQLite,
the information that was intended to go into the original file ended up
overwriting parts of the SQLite database, leading to corruption of the
database.

One example of this occurred circa 2013-08-30 on the canonical repository
for the Fossil DVCS.  In that event,
file descriptor 2 (standard error) was being erroneously closed (by
stunnel, we suspect) prior to 
sqlite3_open_v2() so that the file descriptor used for the
repository database file was 2.  Later, an application 
bug caused an assert() statement to emit
an error message by invoking write(2,...).  But since file descriptor 2 was 
now connected to a database file, the error message
overwrote part of the database.  To guard against this kind of problem,
SQLite version 3.8.1 (2013-10-17)
and later refuse to use low-numbered file descriptors
for database files. 
(See SQLITE_MINIMUM_FILE_DESCRIPTOR for additional information.)

Another example of corruption caused by using a closed file
descriptor was 
reported by facebook engineers in a blog post on 2014-08-12.

Another example of this error was reported against
Fossil on 2019-07-11.  A file descriptor would
be opened for debugging output, but then closed and reopened by SQLite.
But the debugging logic continued to write into the original file
descriptor. See the
forum discussion
for the bug report and a link to the fix.

1.2.  Backup or restore while a transaction is active

Systems that run automatic backups in the background might try to
make a backup copy of an SQLite database file while it is in the middle
of a transaction.  The backup copy then might contain some old and some
new content, and thus be corrupt.

The best approach to make reliable backup copies of an SQLite database
is to make use of the backup API that is part of the SQLite library.
Failing that, it is safe to make a copy of an SQLite database file as long
as there are no transactions in progress by any process.  If the previous
transaction failed, then it is important that any rollback journal
(the *-journal file) or write-ahead log (the *-wal file)
be copied together with the database file itself.



1.3.  Deleting a hot journal

SQLite normally stores all content in a single disk file.  However,
while performing a transaction, information necessary to recover the
database following a crash or power failure is stored in auxiliary
journal files.  Such journal files are described as "hot".
The journal files have the same name as the
original database file with the addition
of -journal or -wal suffix.

SQLite must see the journal files in order to recover from a crash
or power failure.  If the hot journal files are moved, deleted, or renamed
after a crash or power failure, then automatic recovery will not work
and the database may go corrupt.

Another manifestation of this problem is
database corruption caused by inconsistent use of 8+3 filenames.



1.4.  Mispairing database files and hot journals 

 The previous example is a specific case of a more general problem:
The state of an SQLite database is controlled by both the
database file and the journal file.  In a quiescent state, the journal
file does not exist and only the database file matters.
But if the journal file does exist, it must be
kept together with the database to avoid corruption.  The following
actions are all likely to lead to corruption:

 Swapping journal files between two different databases.
 Overwritting a journal file with a different journal file.
 Moving a journal file from one database to another.
 Copying a database file without also copying its journal.
 Overwriting a database file with another without also
     deleting any hot journal associated with the original database.


2.  File locking problems

SQLite uses file locks on the database file, and on the 
write-ahead log or WAL file, to coordinate access between concurrent
processes.  Without coordination, two threads or processes might try
to make incompatible changes to a database file at the same time,
resulting in database corruption.

2.1.  Filesystems with broken or missing lock implementations

SQLite depends on the underlying filesystem to do locking as the
documentation says it will.  But some filesystems contain bugs in their
locking logic such that the locks do not always behave as advertised.
This is especially true of network filesystems and NFS in particular.
If SQLite is used on a filesystem where the locking primitives contain
bugs, and if two or more threads or processes try to access the same
database at the same time, then database corruption might result.



2.2.  Posix advisory locks canceled by a separate thread doing close()

The default locking mechanism used by SQLite on unix platforms is
POSIX advisory locking.  Unfortunately, POSIX advisory locking has design
quirks that make it prone to misuse and failure. In particular, any
thread in the same process with a file descriptor that is holding a POSIX
advisory lock can override that lock using a different file descriptor.
One particularly pernicious problem is that the close() system
call will cancel all POSIX advisory locks on the same file for all
threads and all file descriptors in the process.

So, for example, suppose a multi-thread process has
two or more threads with separate SQLite database connections to the
same database file.  Then a third thread comes along and wants to read
something out of that same database file on its own, without using the 
SQLite library.
The third thread does an open(), a read() and then
a close().
One would think this would be harmless.
But the close() system call caused the
locks held on the database by all the other threads to be dropped.  Those
other threads have no way of knowing that their locks have just been
trashed (POSIX does not provide any mechanism to determine this) and so
they keep on running under the assumption that their locks are still valid.
This can lead to two or more threads or processes trying to write to the
database at the same time, resulting in database corruption.

Note that it is perfectly safe for two or more threads to access the
same SQLite database file using the SQLite library.  The unix drivers for
SQLite know about the POSIX advisory locking quirks and work around them.
This problem only arises when a thread tries to bypass the SQLite library
and read the database file directly.

2.2.1. Multiple copies of SQLite linked into the same application

As pointed out in the previous paragraph, SQLite takes steps to work
around the quirks of POSIX advisory locking.  Part of that work-around involves
keeping a global list (mutex protected) of open SQLite database files.
But, if multiple copies of SQLite are linked into the same application,
then there will be multiple instances of this global list.
Database connections opened using one copy of the SQLite library
will be unaware of database connections opened using the other copy,
and will be unable to work around the POSIX advisory locking quirks.
A close() operation on one connection might unknowingly 
clear the locks on a different database connection, leading to database
corruption.

The scenario above sounds far-fetched.
But the SQLite developers are aware of at 
least one commercial product that was released
with exactly this bug.  The vendor came to the SQLite developers seeking
help in tracking down some infrequent database corruption issues they were
seeing on Linux and Mac.  The problem was eventually traced to the
fact that the application was linking against two separate copies of SQLite.
The solution was to change the application build procedures to link against
just one copy of SQLite instead of two.

2.3.  Two processes using different locking protocols

The default locking mechanism used by SQLite on unix platforms is
POSIX advisory locking, but there are other options.  By selecting an
alternative sqlite3_vfs using the sqlite3_open_v2() interface, an
application can make use of other locking protocols that might be more
appropriate to certain filesystems.  For example, dot-file locking might
be select for use in an application that has to run on an NFS filesystem
that does not support POSIX advisory locking.

It is important that all connections to the same database file use 
the same locking protocol.
If one application is using POSIX advisory locks and another application
is using dot-file locking, then the two applications will not see each
other's locks and will not be able to coordinate database access, possibly
leading to database corruption.



2.4.  Unlinking or renaming a database file while in use

If two processes have open connections to the same database file and
one process closes its connection, unlinks the file, then creates a new
database file in its place with the same name and reopens the new file,
then the two processes will be talking to different database files with 
the same name.  (Note that this is only possible on Posix and Posix-like
systems that permit a file to be unlinked while it is still open for
reading and writing.  Windows does not allow this to occur.)
Since rollback journals and WAL files are based on the name of the database
file, the two different database files will share the same rollback
journal or WAL file.  A rollback or recovery for one of the databases
might use content from the other database, resulting in corruption.
A similar problem occurs if a database file is renamed while it is
opened and a new file is created with the old name.

In other words, unlinking or renaming an open database file 
results in behavior that is undefined and probably undesirable.

Beginning with SQLite version 3.7.17 (2013-05-20),
the unix OS interface will
send SQLITE_WARNING messages to the error log if a database file is unlinked
while it is still in use.



2.5.  Multiple links to the same file

If a single database file has multiple links (either hard or soft links)
then that is just another way of saying that the file has multiple names.
If two or more processes open the database using different names, then
they will use different rollback journals and WAL files.  That means that
if one process crashes, the other process will be unable to recover the
transaction in progress because it will be looking in the wrong place
for the appropriate journal.

In other words, opening and using a database file that has two or
more names results in behavior that is undefined and probably undesirable.

Beginning with SQLite version 3.7.17 (2013-05-20),
the unix OS interface will
send SQLITE_WARNING messages to the error log if a database file has 
multiple hard links.

Beginning with SQLite version 3.10.0 (2016-01-06), 
the unix OS interface will
attempt to resolve symbolic links and open the database file by its
canonical name.  Prior to version 3.10.0, opening a database file 
through a symbolic link was similar to opening a database file
that had multiple hard links and resulted in undefined behavior.



2.6.  Carrying an open database connection across a fork() 

 Do not open an SQLite database connection, then fork(), then try
to use that database connection in the child process.  All kinds of
locking problems will result and you can easily end up with a corrupt
database.  SQLite is not designed to support that kind of behavior.
Any database connection that is used in a child process must be opened
in the child process, not inherited from the parent.

 Do not even call sqlite3_close() on a database connection from a
child process if the connection was opened in the parent.  It is safe
to close the underlying file descriptor, but the sqlite3_close()
interface might invoke cleanup activities that will delete content out
from under the parent, leading to errors and perhaps even database
corruption.

3.  Failure to sync

In order to guarantee that database files are always consistent, SQLite
will occasionally ask the operating system to flush all pending writes to
persistent storage then wait for that flush to complete.  This is 
accomplished using the fsync() system call under unix and
FlushFileBuffers() under Windows.  We call this flush of
pending writes a "sync".

Actually, if one is only concerned with atomic and consistent writes and
is willing to forego durable writes, the sync operation does not need
to wait until the content is completely stored on persistent media.  Instead,
the sync operation can be thought of as an I/O barrier.  As long as all
writes that occur before the sync are completed before any write that happens
after the sync, no database corruption will occur.  If sync is operating as
an I/O barrier and not as a true sync, then a power failure or system crash
might cause one or more previously committed transactions to roll back
(in violation of the "durable" property of "ACID") but the database will at
least continue to be consistent, and that is what most people care about.

3.1.  Disk drives that do not honor sync requests

Unfortunately, most consumer-grade mass storage devices lie about
syncing.  Disk drives will report that content is safely on persistent
media as soon as it reaches the track buffer and before actually being
written to oxide.  This makes the disk drives seem to operate faster
(which is vitally important to the manufacturer so that they can show
good benchmark numbers in trade magazines).  And in fairness, the lie
normally causes no harm, as long as there is no power loss or hard reset
prior to the track buffer actually being written to oxide.  But if a
power loss or hard reset does occur, and if that results in content that
was written after a sync reaching oxide while content written before
the sync is still in a track buffer, then database corruption can occur.

USB flash memory sticks seem to be especially pernicious liars 
regarding sync requests.  One can easily see this by committing a large
transaction to an SQLite database on a USB memory stick.  The COMMIT
command will return relatively quickly, indicating that the memory stick
has told the operating system and the operating system has told SQLite that
all content is safely in persistent storage, and yet the LED on the end
of the memory stick will continue flashing for several more seconds. 
Pulling out the memory stick while the LED is still flashing will frequently
result in database corruption.

Note that SQLite must believe whatever the operating system and hardware
tell it about the status of sync requests.  There is no way for SQLite to
detect that either is lying and that writes might be occurring out-of-order.
However, SQLite in WAL mode is far more forgiving of
out-of-order writes than in the default rollback journal modes.  In WAL
mode, the only time that a failed sync operation can cause database corruption
is during a checkpoint operation.  A sync failure during a COMMIT might
result in loss of durability but not in a corrupt database file.  Hence,
one line of defense against database corruption due to failed sync operations
is to use SQLite in WAL mode and to checkpoint as infrequently as possible.

3.2.  Disabling sync using PRAGMAs

The sync operations that SQLite performs to help ensure integrity
can be disabled at run-time using the synchronous pragma.  By setting
PRAGMA synchronous=OFF, all sync operations are omitted.  This makes
SQLite seem to run faster, but it also allows the operating system to freely
reorder writes, which could result in database corruption if a power failure
or hard reset occurs prior to all content reaching persistent storage.

For maximum reliability and for robustness against database corruption,
SQLite should always be run with its default synchronous setting of FULL.



4.  Disk Drive and Flash Memory Failures

An SQLite database can become corrupt if the file content changes 
due to a disk drive or flash memory failure.  It is very rare, but disks 
will occasionally flip a bit in the middle of a sector.

4.1.  Non-powersafe flash memory controllers

We are told that in some flash memory controllers the wear-leveling logic
can cause random filesystem damage if power is interrupted during a write.
This can manifest, for example, as random changes in the middle of a file
that was not even open at the time of the power loss.  So, for example,
a device would be writing content into an MP3 file in flash memory when a
power loss occurs, and that could result in an SQLite database being
corrupted even though the database was not even in use at the time of the
power loss.



4.2.  Fake capacity USB sticks

There are many fraudulent USB sticks in circulation that report to have
a high capacity (ex: 8GB) but are really only capable of storing a much
smaller amount (ex: 1GB).   Attempts to write on these devices will
often result in unrelated files being overwritten.  Any use of a fraudulent
flash memory device can easily lead to database corruption, therefore.
Internet searches such as "fake capacity usb" will turn up lots of
disturbing information about this problem.

5.  Memory corruption

SQLite is a C-library that runs in the same address space as the 
application that it serves.  That means that stray pointers, buffer
overruns, heap corruption, or other malfunctions in the application can
corrupt internal SQLite data structure and ultimately result in a
corrupt database file.  Normally these kinds of problems manifest themselves
as segfaults prior to any database corruption occurring, but there have
been instances where application code errors have caused SQLite to
malfunction subtly so as to corrupt the database file rather than
panicking.

The memory corruption problem becomes more acute when
using memory-mapped I/O.
When all or part of the database file is mapped into the application's
address space, then a stray pointer that overwrites any part of that
mapped space will immediately corrupt the database file, without
requiring the application to do a subsequent write() system call.

6.  Other operating system problems

Sometimes operating systems will exhibit non-standard behavior which
can lead to problems.  Sometimes this non-standard behavior is deliberate,
and sometimes it is a mistake in the implementation.  But in any event,
if the operating performs differently from they way SQLite expects it to
perform, the possibility of database corruption exists.

6.1.  Linux Threads

Some older versions of Linux used the LinuxThreads library for thread
support.  LinuxThreads is similar to Pthreads, but is subtly different
with respect to handling of POSIX advisory locks.  SQLite versions
2.2.3 through 3.6.23 recognized that LinuxThreads were being used at
runtime and took appropriate action to work around the non-standard
behavior of LinuxThreads.  But most modern Linux implementations make
use of the newer, and correct, NPTL implementation of Pthreads.  Beginning
with SQLite version 3.7.0 (2010-07-21), the use of NPTL is assumed.
No checks are 
made.  Hence, recent versions of SQLite will subtly malfunction and may 
corrupt database files if used in multi-threaded application that run
on older linux systems that make use of LinuxThreads.

6.2.  Failures of mmap() on QNX

There exists some subtle problem with mmap() on QNX such that making
a second mmap() call against a single file descriptor can cause
the memory obtained from the first mmap() call to be zeroed.  SQLite on
unix uses mmap() to create a shared memory region for transaction 
coordination in WAL mode, and it will call mmap() multiple times
for large transactions.  The QNX mmap() has been demonstrated to corrupt
database file under that scenario.  QNX engineers are aware of this problem
and are working on a solution; the problem may have already been fixed by
the time you read this.

When running on QNX, it is recommended that memory-mapped I/O never
be used.  Furthermore, to use WAL mode, it is recommended that applications
employ the exclusive locking mode in order to 
use WAL without shared memory.




6.3.  Filesystem Corruption

Since SQLite databases are ordinary disk files, any malfunction in the
filesystem can corrupt the database.  Filesystems in modern operating systems
are very reliable, but errors do still occur.  For example, on 2013-10-01
the SQLite database that holds the
Wiki for Tcl/Tk went corrupt a few days
after the host computer was moved to a dodgy build of the (linux) kernel
that had issues in the filesystem layer.  In that event, the filesystem
eventually became so badly corrupted that the machine was unusable, but
the earliest symptom of trouble was the corrupted SQLite database.



7. SQLite Configuration Errors

SQLite has many built-in protections against database corruption.
But many of these protections can be disabled by configuration options.
If protections are disabled, database corruption may occur.

The following are examples of disabling the built-in protection
mechanisms of SQLite:


Setting PRAGMA synchronous=OFF can cause the database to
go corrupt if there is an operating-system crash or power failure,
though this setting is safe from damage due to application crashes.

Changing the PRAGMA schema_version while other database
connections are open.

Using PRAGMA journal_mode=OFF or PRAGMA journal_mode=MEMORY
and taking an application crash in the middle of a write transaction.

Setting PRAGMA writable_schema=ON and then changing the
database schema using DML statements can render the database completely
unreadable, if not done carefully.


8.  Bugs in SQLite

SQLite is very carefully tested to help ensure that it is
as bug-free as possible.  Among the many tests that are carried out for
every SQLite version are tests that simulate power failures, I/O errors,
and out-of-memory (OOM) errors and verify that no database corruption occurs
during any of these events.  SQLite is also field-proven with approximately
two billion active deployments with no serious problems.

Nevertheless, no software is 100% perfect.  There have been a few
historical bugs in SQLite (now fixed) that could cause database corruption.
And there may be yet a few more that remain undiscovered.  Because of the
extensive testing and widespread use of SQLite, bugs that result in
database corruption tend to be very obscure.  The likelihood
of an application encountering an SQLite bug is small.  To illustrate this,
an account is given below 
of all database-corruption bugs found in SQLite during the
four-year period from 2009-04-01 to 2013-04-15.
This account should give the reader an intuitive sense of the
kinds of bugs in SQLite that manage to slip through testing procedures
and make it into a release.


8.1.  False corruption reports due to database shrinkage

If a database is written by SQLite version 3.7.0 or later and then
written again by SQLite version 3.6.23 or earlier in such a way as to
make the size of the database file decrease, then the next time that
SQLite version 3.7.0 access the database file, it might report that the
database file is corrupt.  The database file is not really corrupt, however.
Version 3.7.0 was simply being overly zealous in its corruption detection.

The problem was fixed on 2011-02-20.  The fix first appears in
SQLite version 3.7.6 (2011-04-12).

8.2.  Corruption following switches between rollback and WAL modes

Repeatedly switching an SQLite database in and out of WAL mode
and running the VACUUM command in between switches, in one process or
thread, can cause another process or thread that has the database file
open to miss the fact that the database has changed.  That second process
or thread might then try to modify the database using a stale cache and
cause database corruption.

This problem was discovered during internal testing and has never been
observed in the wild.  The problem was fixed on 2011-01-27 and in version
3.7.5.

8.3.  I/O error while obtaining a lock leads to corruption

If the operating system returns an I/O error while attempting to obtain
a certain lock on shared memory in WAL mode then SQLite might fail 
to reset its cache,
which could lead to database corruption if subsequent writes are attempted.

Note that this problem only occurs if the attempt to acquire the lock
resulted in an I/O error.  If the lock is simply not granted (because some
other thread or process is already holding a conflicting lock) then no
corruption will ever occur.  We are not aware of any operating systems that
will fail with an I/O error while attempting to get a file lock on shared
memory.  So this is a theoretical problem rather than a real problem.
Needless to say, this problem has never been observed in the wild.  The
problem was discovered while doing stress testing of SQLite in a test
harness that simulates I/O errors.

This problem was fixed on 2010-09-20 for SQLite version 3.7.3.

8.4.  Database pages leak from the free page list

When content is deleted from an SQLite database, pages that are no
longer used are added to a free list and are reused to hold content
added by subsequent inserts.  A bug in SQLite that was present in
version 3.6.16 through 3.7.2 might cause pages to go missing out of
the free list when incremental_vacuum was used.  This would not cause
data loss.  But it would result in the database file being larger than
necessary.  And it would cause the integrity_check pragma to report
pages missing from the free list.

This problem was fixed on 2010-08-23 for SQLite version 3.7.2.

8.5.  Corruption following alternating writes from 3.6 and 3.7

SQLite version 3.7.0 introduced a number of new enhancements to
the SQLite database file format (such as but not limited to WAL).
The 3.7.0 release was a shake-out release for these new features.
We expected to find problems and were not disappointed.

If a database were originally created using SQLite version 3.7.0,
then written by SQLite version 3.6.23.1 such that the size of the database
file increased, then written again by SQLite version 3.7.0, the database
file could go corrupt.

This problem was fixed on 2010-08-04 for SQLite version 3.7.1.

8.6.  Race condition in recovery on Windows systems

SQLite version 3.7.16.2 fixes a subtle race condition in the locking
logic on Windows systems.  When a database file is in need
of recovery because the previous process writing to it crashed in the
middle of a transaction and two or more processes try to open the 
that database at the same time, then the race condition might cause
one of those processes to get a false indication that the recovery 
has already completed, allowing that process to continue using the
database file without running recovery first.  If that process writes
to the file, then the file might go corrupt.  This race condition
had apparently existed in all prior versions of SQLite for Windows going
back to 2004.  But the race was very tight.  Practically speaking, you
need a fast multi-core machine in which you launch two processes to run
recovery at the same moment on two separate cores.  This defect was
on Windows systems only and did not affect the posix OS interface.



8.7.  Boundary value error in the secondary journals used by nested transactions

When a nested transaction is started using SAVEPOINT, SQLite uses
a secondary rollback journal to track the changes for the nested
transaction, in case the inner transaction needs to be rolled back.  Secondary
journals are not involved in protecting the database from corruption due
to program crashes or power outages.  The secondary journals only come into
play when rolling back an inner transaction of a nested transaction.

These secondary journals can be held either in memory or as temporary
files on disk.  The default behavior is to store them on disk.  But that 
can be changed using the -DSQLITE_TEMP_STORE compile-time option,
or at run-time using the PRAGMA temp_store statement.  The bug
only arises when secondary journals are held in memory.

In SQLite version 3.35.0 (2021-03-12), a new optimization was
added so that when SQLite is holding secondary journals in memory,
less memory will be used. Unfortunately, an boundary check in
the new logic was coded incorrectly.
What should have been a "&lt;" operator was coded as "&lt;=".  This
error might cause the secondary journal to enter an inconsistent state
if it is ever rolled back.  If additional changes are made and the
outer transaction eventually commits, the database might be left in
an inconsistent state.

This problem was discovered by an 
independent researcher
who was attempting to find bugs in SQLite using a fuzzer.  The fuzzer found a
failure in an assert() statement that is used
to help verify the internal state of the secondary journal.  The bug was a
sufficiently obscure corner-case that it might have gone unnoticed for many
years, had it not been for the intensive use of assert() statements in SQLite,
the persistence and tenacity of the security researchers, and their
customized state-of-the-art fuzzer.

This problem was fixed
in version 3.37.2 (2022-01-06).


c2: The Carray() Table-Valued Function
c5: carray.html
c4:
1. Overview

Carray() is a table-valued function with a single column (named
"value") and zero or more rows.
The "value" of each row in the carray() is taken from a C-language array
supplied by the application via parameter binding.
In this way, the carray() function provides a convenient mechanism to
bind C-language arrays to SQL queries.

2. Availability

The carray() function is not compiled into SQLite by default.
It is available as a loadable extension in the
ext/misc/carray.c
source file.

The carray() function was first added to SQLite in version 3.14
(2016-08-08).  The sqlite3_carray_bind() interface and the
single-argument variant of carray() was added in SQLite version 3.34.0
(2020-12-01).  The ability to bind an array of struct iovec
objects that are interpreted as BLOBs was added in SQLite version 3.41.0
(2023-02-21).

3. Details

The carray() function takes one, two, or three arguments.

For the two- and three-argument versions of carray(),
the first argument is a pointer to an array.  Since pointer values cannot
be specified directly in SQL, the first argument must be a parameter that
is bound to a pointer value using the sqlite3_bind_pointer() interface
using a pointer-type of "carray".
The second argument is the number of elements in the array.  The optional
third argument is a string that determines the datatype of the elements
in the C-language array.  Allowed values for the third argument are:


 'int32'
 'int64'
 'double'
 'char*'
 'struct iovec'


The default datatype is 'int32'.

The 'struct iovec' type used for BLOB data is a standard Posix data
structure, normally declared using "#include &lt;sys/uio.h&gt;".
The format is:


struct iovec {
  void  *iov_base; /* Starting address */
  size_t iov_len;  /* Number of bytes to transfer */
};




3.1. Single-Argument CARRAY

The single-argument form of carray() requires a special C-language
interface named "sqlite3_carray_bind()" in order to attach values:


  int sqlite3_carray_bind(
    sqlite3_stmt *pStmt,         /* Statement containing the CARRAY */
    int idx,                     /* Parameter number for CARRAY argument */
    void *aData,                 /* Data array */
    int nData,                   /* Number of entries in the array */
    int mFlags,                  /* Datatype flag */
    void (*xDestroy)(void*)      /* Destructor for aData */
  );


The mFlags parameter to sqlite3_carray_bind() must be one of:


  #define CARRAY_INT32   0
  #define CARRAY_INT64   1
  #define CARRAY_DOUBLE  2
  #define CARRAY_TEXT    3
  #define CARRAY_BLOB    4


Higher order bits of the mFlags parameter must all be zero for now,
though they may be used in future enhancements.  The definitions for the
constants that specify the datatype and a prototype for the
sqlite3_carray_bind() function are both available in the auxiliary
header file
ext/misc/carray.h.

The xDestroy argument to sqlite3_carray_bind() routine is a pointer
to a function that frees the input array.  SQLite will invoke this
function after it has finished with the data.  The xDestroy argument
may optionally be one of the following constants defined in
"sqlite3.h":



     SQLITE_STATIC &rarr;  This means that the application that invokes
     sqlite3_carray_bind() maintains ownership of the data array and that
     the application promises SQLite that it will not change or deallocate
     the data until after the prepared statement is finialized.


     SQLITE_TRANSIENT &rarr;  This special value instructs SQLite to make
     its own private copy of the data before the 
     sqlite3_carray_bind() interface returns.



4. Usage

The carray() function can be used in the FROM clause of a query.
For example, to query two entries from the OBJ table using rowids
taken from a C-language array at address $PTR.

SELECT obj.* FROM obj, carray($PTR, 10) AS x
 WHERE obj.rowid=x.value;


This query gives the same result:

SELECT * FROM obj WHERE rowid IN carray($PTR, 10);

This page last modified on  2023-02-17 13:24:09 UTC


c2: Long Term Support
c5: lts.html
c4:
The intent of the developers is to support SQLite through
the year 2050.


At this writing, 2050 is still 34 years in the future.
Nobody knows what will happen in that time, and we cannot
absolutely promise that SQLite will be viable or useful that
far out.
But we can promise this: we plan as if we will be
supporting SQLite until 2050.
That long-term outlook affects our
decisions in important ways.



Cross-platform Code &rarr;
SQLite runs on any platform with an 8-bit byte,
two's complement 32-bit and 64-bit integers, 
and a C compiler.  It is actively
tested on all currently popular CPUs and operating
systems.
The extreme portability of the SQLite code and file
format will help it remain viable on future platforms.


Stable, Cross-platform Database Files &rarr;
SQLite database files are bit-for-bit identical on
32-bit, 64-bit, big-endian, and little-endian platforms.  You
can copy an SQLite database file from one system to another
without having to translate or convert the database.
Furthermore, the file format is well documented and stable.
Database files created today will be readable and writable by
future versions of SQLite decades in the future.


Aviation-grade testing &rarr;
Every machine-code branch instruction is tested in both
directions.  Multiple times.  On multiple platforms and with
multiple compilers.  This helps make the code robust for
future migrations.  The intense testing also means that new
developers can make experimental enhancements to SQLite and,
assuming legacy tests all pass, be reasonably sure that the
enhancement does not break legacy.


Extensive, detailed documentation &rarr;
SQLite has candid, developer-friendly,
and open-source documentation.  Docs are written by and
for programmers.
(A few examples:
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8])
The extensive documentation helps new developers
come up to speed on SQLite very quickly.


Heavily commented source code &rarr;
The SQLite source code is over 35% comment.  Not boiler-plate
comments, but useful comments that explain the meaning of variables
and objects and the intent of methods and procedures.  
The code is designed
to be accessible to new programmers and maintainable over a span
of decades.


Disaster planning &rarr;
Every byte of source-code history for SQLite is cryptographically
protected and is automatically replicated to multiple
geographically separated servers, in datacenters 
owned by different companies.
Thousands of additional clones exist on private servers around the
world.
The primary developers of SQLite live in different regions of the world.
SQLite can survive a continental catastrophe.


Old school &rarr;
Nobody is completely immune to trends and fads, but the SQLite
developers work hard to avoid being sucked into the latest programming
fashion.  Our aim is to produce timeless code that will be
readable, understandable, and maintainable by programmers 
who have not yet been born.



In addition to "supporting" SQLite through the year 2050, the developers
also promise to keep the SQLite 
C-language API and on-disk format 
fully backwards compatible.
This means that application written to use SQLite today should be able to
link against and use future versions of SQLite released decades in the
future.


Our goal is to make the content you store in SQLite today as 
easily accessible to your grandchildren as it is to you.


Update on 2018-05-31:
Our goal of supporting SQLite long-term have apparently come to the
notice of the preservationist at the 
US Library Of Congress who have identified
SQLite as a recommended storage format for the preservation of
digital content.
This page last modified on  2022-01-08 05:02:57 UTC


c2: SQLite Autoincrement
c5: autoinc.html
c4:
1. Summary



  The AUTOINCREMENT keyword imposes extra CPU, memory, disk space,
  and disk I/O overhead and should be avoided if not strictly needed.
  It is usually not needed.

  In SQLite, a column with type INTEGER PRIMARY KEY is an alias for the ROWID
  (except in WITHOUT ROWID tables) which is always a 64-bit signed integer.

  On an INSERT, if the ROWID or INTEGER PRIMARY KEY column is not 
  explicitly given a value, then it
  will be filled automatically with an unused integer, usually
  one more than the largest ROWID currently in use.
  This is true regardless of whether or not the AUTOINCREMENT keyword is used.

  If the AUTOINCREMENT keyword appears after INTEGER PRIMARY KEY, that
  changes the automatic ROWID assignment algorithm to prevent
  the reuse of ROWIDs over the lifetime of the database.  In other words,
  the purpose of AUTOINCREMENT is to prevent the reuse of ROWIDs from
  previously deleted rows.


2. Background


In SQLite, table rows normally have a 64-bit signed integer ROWID
which is unique among all rows in the same table.
(WITHOUT ROWID tables are the exception.)



You can access the ROWID of an SQLite table using one of the special column
names ROWID, _ROWID_, or OID.
Except if you declare an ordinary table column to use one of those special
names, then the use of that name will refer to the declared column not
to the internal ROWID.



If a table contains a column of type INTEGER PRIMARY KEY, then that
column becomes an alias for the ROWID.  You can then access the ROWID
using any of four different names, the original three names described above
or the name given to the INTEGER PRIMARY KEY column.  All these names are
aliases for one another and work equally well in any context.



When a new row is inserted into an SQLite table, the ROWID can either
be specified as part of the INSERT statement or it can be assigned
automatically by the database engine.  To specify a ROWID manually,
just include it in the list of values to be inserted.  For example:


CREATE TABLE test1(a INT, b TEXT);
INSERT INTO test1(rowid, a, b) VALUES(123, 5, 'hello');



If no ROWID is specified on the insert, or if the specified ROWID has a value
of NULL, then an appropriate ROWID is created
automatically.  The usual algorithm is to give the newly created row
a ROWID that is one larger than the largest ROWID in the table prior
to the insert.  If the table is initially empty, then a ROWID of 1 is
used.  If the largest ROWID is equal to the largest possible integer
(9223372036854775807) then the database
engine starts picking positive candidate ROWIDs at random until it finds one
that is not previously used.
If no unused ROWID can be found after a reasonable number of attempts,
the insert operation fails with an SQLITE_FULL error.
If no negative ROWID values are inserted explicitly, then automatically 
generated ROWID values will always be greater than zero.



The normal ROWID selection algorithm described above
will generate monotonically increasing
unique ROWIDs as long as you never use the maximum ROWID value and you never
delete the entry in the table with the largest ROWID. 
If you ever delete rows or if you ever create a row with the maximum possible
ROWID, then ROWIDs from previously deleted rows might be reused when creating
new rows and newly created ROWIDs might not be in strictly ascending order.



3. The AUTOINCREMENT Keyword


If a column has the type INTEGER PRIMARY KEY AUTOINCREMENT then a slightly
different ROWID selection algorithm is used.  
The ROWID chosen for the new row is at least one larger than the largest ROWID
that has ever before existed in that same table.  If the table has never
before contained any data, then a ROWID of 1 is used.  If the largest possible
ROWID has previously been inserted, then
new INSERTs are not allowed and any attempt to insert a new row will
fail with an SQLITE_FULL error.  
Only ROWID values from previous transactions that
were committed are considered.  ROWID values that were rolled back
are ignored and can be reused.



SQLite keeps track of the largest ROWID
using an internal table named "sqlite_sequence".
The sqlite_sequence table is created
and initialized automatically whenever a normal table that contains an
AUTOINCREMENT column is created.  The content of the sqlite_sequence table
can be modified using ordinary UPDATE, INSERT, and DELETE statements.
But making modifications to this table will likely perturb the AUTOINCREMENT
key generation algorithm.  Make sure you know what you are doing before
you undertake such changes.
The sqlite_sequence table does not track ROWID changes associated with
UPDATE statement, only INSERT statements.



The behavior implemented by the AUTOINCREMENT keyword is subtly different
from the default behavior.  With AUTOINCREMENT, rows with automatically
selected ROWIDs are guaranteed to have ROWIDs that have never been used
before by the same table in the same database.  And the automatically generated
ROWIDs are guaranteed to be monotonically increasing.  These are important
properties in certain applications.  But if your application does not
need these properties, you should probably stay with the default behavior
since the use of AUTOINCREMENT requires additional work to be done
as each row is inserted and thus causes INSERTs to run a little slower.


Note that "monotonically increasing" does not imply that the ROWID always
increases by exactly one.  One is the usual increment.  However, if an
insert fails due to (for example) a uniqueness constraint, the ROWID of
the failed insertion attempt might not be reused on subsequent inserts,
resulting in gaps in the ROWID sequence.  AUTOINCREMENT guarantees that
automatically chosen ROWIDs will be increasing but not that they will be
sequential.

Because AUTOINCREMENT keyword changes the behavior of the ROWID selection
algorithm, AUTOINCREMENT is not allowed on WITHOUT ROWID tables or on any
table column other than INTEGER PRIMARY KEY.  Any attempt to use 
AUTOINCREMENT on a WITHOUT ROWID table or on a column other than the
INTEGER PRIMARY KEY column results in an error.
This page last modified on  2022-01-08 05:02:57 UTC


c2: SQLite Download Page
c5: download.html
c4:
SQLite Download Page











Build Product Names and Info


Build products are named using one of the following templates:


 sqlite-product-version.zip
 sqlite-product-version.tar.gz
 sqlite-product-os-cpu-version.zip
 sqlite-product-date.zip


Templates (1) and (2) are used for source-code products.  Template (1) is
used for generic source-code products and templates (2) is used for source-code
products that are generally only useful on unix-like platforms.  Template (3)
is used for precompiled binaries products.  Template (4) is used for
unofficial pre-release "snapshots" of source code.

The version is encoded so that filenames sort in order of
increasing version number when viewed using "ls".  For version 3.X.Y the
filename encoding is 3XXYY00.  For branch version 3.X.Y.Z, the encoding is
3XXYYZZ.

The date in template (4) is of the form: YYYYMMDDHHMM

For convenient, script-driven extraction of the downloadable
file URLs and associated information, an HTML comment is embedded
in this page's source. Its first line (sans leading tag) reads:
Download product data for scripts to read
Its subsequent lines comprise a CSV table with this column header:
PRODUCT,VERSION,RELATIVE-URL,SIZE-IN-BYTES,SHA3-HASH
The column header and following data lines have no leading space.
The PRODUCT column is a constant value ("PRODUCT") for convenient
regular expression matching. Other columns are self-explanatory.
This format will remain stable except for possible new columns
appended to the right of older columns.





Source Code Repositories


The SQLite source code is maintained in three geographically-dispersed
self-synchronizing
Fossil repositories that are
available for anonymous read-only access.  Anyone can
view the repository contents and download historical versions
of individual files or ZIP archives of historical check-ins.
You can also clone the entire repository.

See the How To Compile SQLite page for additional information
on how to use the raw SQLite source code.
Note that a recent version of Tcl
is required in order to build from the repository sources.
The amalgamation source code files
(the "sqlite3.c" and "sqlite3.h" files) build products and are
not contained in raw source code tree.


https://www.sqlite.org/cgi/src (Dallas)
https://www2.sqlite.org/cgi/src (Newark)
https://www3.sqlite.org/cgi/src (San Francisco)


There is a GitHub mirror at


https://github.com/sqlite/sqlite/


The documentation is maintained in separate
Fossil repositories located
at:


https://www.sqlite.org/cgi/docsrc (Dallas)
https://www2.sqlite.org/cgi/docsrc (Newark)
https://www3.sqlite.org/cgi/docsrc (San Francisco)


c2: List Of Virtual Tables
c5: vtablist.html
c4:
1. Introduction

A virtual table is an object that presents an SQL table
interface but which is not stored in the database file, at least
not directly.  The virtual table mechanism is a feature
of SQLite that allows SQLite to access and manipulate resources
other than bits in the database file using the powerful SQL
query language.


The table below lists a few of the virtual tables implementations
available for SQLite.  Developers can deploy these virtual tables
in their own applications, or use the implementations shown below
as templates for writing their own virtual tables.


The list below is not exhaustive.
Other virtual table implementation exist in the SQLite source tree
and elsewhere.
The list below tries to capture the more interesting virtual table
implementations.



2. Virtual Tables


NameDescription
approximate_match
A demonstration of how to use a virtual table to implement
    approximate string matching.
bytecode
A table-valued function that shows the bytecodes of a prepared
    statement.
carray
A table-valued function that allows a C-language array of integers,
    doubles, or strings to be used as table in a query.
closure
Compute the transitive closure of a set.
completion
Suggests completions for partially-entered words during interactive
    SQL input.  Used by the CLI to help implement tab-completion.
csv
A virtual table that represents a comma-separated-value or CSV file
    (RFC 4180) as a read-only
    table so that it can be used as part of a larger query.
dbstat
Provides information about the purpose and use of each page in a
    database file.  Used in the implementation of the
    sqlite3_analyzer utility program.
files_of_checkin
Provides information about all files in a single check-in in the
    Fossil version control system.  This
    virtual table is not part of the SQLite project but is included because
    it provides an example of how to use virtual tables and because it is
    used to help version control the SQLite sources.
fsdir
A table-valued function returning one row for each file in a
    selected file hierarchy of the host computer.  Used by the
    CLI to help implement the .archive command.
FTS3
A high-performance full-text search index.
FTS5
A higher-performance full-text search index
generate_series
A table-valued function returning a sequence of
    integers, modeled after the table-valued function by the same
    name in PostgreSQL.
json_each
A table-valued function for decomposing a JSON string.
json_tree
A table-valued function for decomposing a JSON string.
OsQuery
Hundreds of virtual tables that publish various aspects of the
    host computer, such as the process table, user lists, active network
    connections, and so forth.
    OsQuery is a separate project, started by Facebook, hosted on
    GitHub, and intended for
    security analysis and intrusion detection
    OsQuery is not a part of the SQLite project, but is included in this
    list because it demonstrates how the SQL language and the
    SQLite virtual table mechanism
    can be leveraged to provide elegant solutions
    to important real-world problems.
pragma
Built-in table-valued functions that return the results of PRAGMA
    statements for use within ordinary SQL queries.
RTree
An implementation of the Guttmann R*Tree spatial index idea.
spellfix1
A virtual table that implements a spelling correction engine.
sqlite_btreeinfo
This experimental table-valued function provides information about
    a single B-tree in a database file, such as the depth, and estimated
    number of pages and number of entries, and so forth.
sqlite_dbpage
Key/value store for the raw database file content.  The key is the
    page number and the value is binary page content.
sqlite_memstat
Provides SQL access to the sqlite3_status64() and
    sqlite3_db_status() interfaces.
sqlite_stmt
A table-valued function containing one row for each
    prepared statement associated with an open database connection.
swarmvtab
An experimental module providing on-demand read-only access to
    multiple tables spread across multiple databases, via a single
    virtual table abstraction.
tables_used
A table-valued function that shows the tables and indexes that
    are accessed by a prepared statement.
tclvar
Represents the global variables of a 
    TCL Interpreter as an SQL
    table. Used as part of the SQLite test suite.
templatevtab
A template virtual table implementation useful as a starting point
    for developers who want to write their own virtual tables
unionvtab
An experimental module providing on-demand read-only access to
    multiple tables spread across multiple databases, via a single
    virtual table abstraction.
vfsstat
A table-valued function which, in combination with a
    co-packaged VFS shim provides information on the number of
    system calls performed by SQLite.
vtablog
A virtual table that prints diagnostic information
    on stdout when its key methods are invoked.  Intended for
    interactive analysis and debugging of virtual table interfaces.
wholenumber
A virtual table returns all integers between 1 and 4294967295.
zipfile
Represent a
    ZIP Archive
    as an SQL table.  Works for both reading and writing.  Used by
    the CLI to implement the ability to read and write ZIP Archives.


This page last modified on  2023-05-01 21:49:55 UTC


c2: SQLite Consortium
c5: consortium.html
c4:
The SQLite Consortium


The SQLite Consortium is a membership association dedicated to
insuring the continuing vitality and independence of SQLite.
SQLite is high-quality, public domain software.  The goal of
the SQLite Consortium is to make sure it stays that way.



Consortium members contribute funding to support the ongoing
development and maintenance of SQLite and in return receive
enterprise-level technical support and legal assurances that
SQLite will remain independent and open-source.  Consortium 
members have direct access to
the SQLite developers and receive priority consideration for
any bug fixes or feature requests.  However, technical control
and direction of SQLite remains with the SQLite architect and
developers.  An important goal of the SQLite Consortium, after
all, is to make sure that SQLite does not fall under the governance
of any single company but remains independent and fully under the
control of the SQLite developers.



Companies that use SQLite as a critical component in their
products or infrastructure may want to consider becoming
SQLite Consortium Members as inexpensive insurance
in the future vitality and independence of SQLite.  The SQLite Consortium 
membership fee is a fraction of the cost of devoting internal
staff to supporting the SQLite code base.  Benefits of
membership include:




Consortium members have the guaranteed, undivided attention of the SQLite
developers for 23 staff-days per year and for as much additional time 
above and beyond that amount that the core developers have available.
There are no arbitrary limits on contact time.
The consortium will never be over-subscribed.  New SQLite developers
will be recruited and trained as necessary to cover the 23 day/year
support commitment.



Consortium members can call any developer at any time, day or
night, and expect to get their full and immediate attention.
Consortium members have access to the home telephone numbers
and cellphone numbers and work schedules of the core developers
so that the developers can be tracked down quickly in a crisis.



The SQLite developers are eager and happy to write new code
or debug existing code for consortium members.
Consortium members can request and expect to receive significant
new enhancements to the public SQLite code base.  Consortium
members can also request their own private, proprietary extensions
and enhancements.



Consortium members can receive support for any
version of SQLite no matter how old.
Bug fixes will be back ported to older versions of SQLite upon
request.



Consortium members can receive private
email and/or telephone notifications and briefings covering
new bugs or recently added features.
The core developers are constantly watching out for the
interests of consortium members and will attempt to protect
those interests in every way possible and alert members
when their interests are threatened in any way.



The SQLite developers will set up customized regression tests
to fully and completely test SQLite as configured by consortium
members.  Normally, only the standard SQLite configuration is fully
tested at each release.  Consortium members that use a non-standard 
configuration can request that their configuration be tested to the 
same level of detail.



Support requests are accepted from any employee or contractor working
for a consortium member.
Support is not limited to a single development
team as it is with a maintenance subscription or technical support
agreement.



Support requests, comments, and suggestions for future
changes to SQLite coming from consortium members take priority 
over requests from all other sources.
Consortium members go to the front of the line.



We will be happy to 
recognize consortium members
 on the SQLite website
with a logo and/or a brief acknowledgement of their contribution to the
project.  This is an opportunity for companies to build good will by
demonstrating that they are giving back to the community.  Or, members 
can remain anonymous.





How To Become A Consortium Member


Membership in the SQLite Consortium is via
contract with Hipp, Wyrick &amp; Company, Inc..
Send email to drh@sqlite.org or phone
+1.704.948.4565 during US east-coast business hours
for additional information.


c2: Automatic Undo/Redo With SQLite
c5: undoredo.html
c4:
Automatic Undo/Redo Using SQLite



This page demonstrates how to use triggers to implement undo/redo 
logic for an application that uses SQLite as its 
application file format.

Object-Oriented Design


This design note considers the database to be a collection of objects.
Each SQL table is a class.
Each row is an instance of that class.
There are, of course, other ways to interpret an SQL database schema,
and the techniques described here work equally well under alternative
interpretations, but an object-oriented view seems be more natural
to most contemporary programmers.

Capture Changes Using Triggers


The core idea is to create a special table (named "UNDOLOG" in the example)
that holds information needed to undo/redo changes to the database. 
For each class (table) in the database that wants to participate in 
the undo/redo, triggers are created that cause entries to be made in 
the UNDOLOG table for each DELETE, INSERT, and UPDATE of the participating
class.
The UNDOLOG entries consist of ordinary SQL statements the can be
played back to reverse the changes.


For example, suppose you wanted undo/redo on a class (table)
that looks like this:


CREATE TABLE ex1(a,b,c);



Triggers to record changes to table EX1 might look like this:


CREATE TEMP TRIGGER ex1_it AFTER INSERT ON ex1 BEGIN
  INSERT INTO undolog VALUES(NULL,'DELETE FROM ex1 WHERE rowid='||new.rowid);
END;
CREATE TEMP TRIGGER ex1_ut AFTER UPDATE ON ex1 BEGIN
  INSERT INTO undolog VALUES(NULL,'UPDATE ex1
     SET a='||quote(old.a)||',b='||quote(old.b)||',c='||quote(old.c)||'
   WHERE rowid='||old.rowid);
END;
CREATE TEMP TRIGGER ex1_dt BEFORE DELETE ON ex1 BEGIN
  INSERT INTO undolog VALUES(NULL,'INSERT INTO ex1(rowid,a,b,c)
    VALUES('||old.rowid||','||quote(old.a)||','||quote(old.b)||
           ','||quote(old.c)||')');
END;



After each INSERT on ex1, the ex1_it trigger constructs text of a 
DELETE statement that will undo the INSERT. The ex1_ut trigger constructs 
an UPDATE statement that will undo the effects of an UPDATE. 
And the ex1_dt trigger constructs a statement that will undo the 
effects of a DELETE.


Note the use of the quote() SQL function in these triggers.
The quote() function converts its argument into a form 
that is appropriate for inclusion in an SQL statement. Numeric values 
come through unchanged. Single quotes are added before and after 
strings and any internal single quotes are escaped.  BLOB values
are rendered using SQL-standard hexadecimal BLOB notation.  The
use of the quote() function ensures that the SQL statements used to
undo and redo are always safe from SQL injection.

Automatic Creation Of Triggers


Triggers such as the above could be entered manually, but that is tedious.
An important feature of the technique demonstrated below is 
that the triggers are generated automatically.


The implementation language for the example code is 
TCL, though you can easily do the same thing 
in another programming language.
Remember that the code here is a demonstration of the technique,
not a drop-in module that will automatically do everything for you.
The demonstration code shown below is derived from actual code 
in production use. But you will need to make changes to tailor it 
to your application.


To activate the undo/redo logic, invoke the undo::activate command 
with all classes (tables) that are to participate in the undo/redo 
as arguments. Use undo::deactivate, undo::freeze, and undo::unfreeze 
to control the state of the undo/redo mechanism.


The undo::activate command creates temporary triggers in the database
that record all changes made to the tables named in the arguments.

Application Interface


After a sequence of changes that define a single undo/redo step, 
invoke the undo::barrier command to define the limit of that step. 
In an interactive program, you can call undo::event after any change 
and undo::barrier will be called automatically as an idle callback.


When the user presses the Undo button, invoke undo::undo. 
Invoke undo::redo when the user presses the Redo button.


On each call to undo::undo or undo::redo, the undo/redo module 
automatically invokes methods status_refresh and reload_all in 
all toplevel namespaces. These methods should be defined to 
reconstruct the display or otherwise update the state of the 
program based on the undone/redone changes to the database.


The demonstration code below includes a status_refresh method 
that grays-out or activates the Undo and Redo buttons and menu 
entries depending on whether or not there is anything to be 
undone or redone. You will need to redefine this method to 
control the Undo and Redo buttons in your application.


The demonstration code assumes that the SQLite database is 
opened used as a database object named "db".

Example Code



# Everything goes in a private namespace
namespace eval ::undo {

# proc:  ::undo::activate TABLE ...
# title: Start up the undo/redo system
#
# Arguments should be one or more database tables (in the database associated
# with the handle "db") whose changes are to be recorded for undo/redo
# purposes.
#
proc activate {args} {
  variable _undo
  if {$_undo(active)} return
  eval _create_triggers db $args
  set _undo(undostack) {}
  set _undo(redostack) {}
  set _undo(active) 1
  set _undo(freeze) -1
  _start_interval
}

# proc:  ::undo::deactivate
# title: Halt the undo/redo system and delete the undo/redo stacks
#
proc deactivate {} {
  variable _undo
  if {!$_undo(active)} return
  _drop_triggers db
  set _undo(undostack) {}
  set _undo(redostack) {}
  set _undo(active) 0
  set _undo(freeze) -1
}

# proc:  ::undo::freeze
# title: Stop accepting database changes into the undo stack
#
# From the point when this routine is called up until the next unfreeze,
# new database changes are rejected from the undo stack.
#
proc freeze {} {
  variable _undo
  if {!&#91;info exists _undo(freeze)&#93;} return
  if {$_undo(freeze)>=0} {error "recursive call to ::undo::freeze"}
  set _undo(freeze) &#91;db one {SELECT coalesce(max(seq),0) FROM undolog}&#93;
}

# proc:  ::undo::unfreeze
# title: Begin accepting undo actions again.
#
proc unfreeze {} {
  variable _undo
  if {!&#91;info exists _undo(freeze)&#93;} return
  if {$_undo(freeze)$_undo(freeze)"
  set _undo(freeze) -1
}

# proc:  ::undo::event
# title: Something undoable has happened
#
# This routine is called whenever an undoable action occurs.  Arrangements
# are made to invoke ::undo::barrier no later than the next idle moment.
#
proc event {} {
  variable _undo
  if {$_undo(pending)==""} {
    set _undo(pending) &#91;after idle ::undo::barrier&#93;
  }
}

# proc:  ::undo::barrier
# title: Create an undo barrier right now.
#
proc barrier {} {
  variable _undo
  catch {after cancel $_undo(pending)}
  set _undo(pending) {}
  if {!$_undo(active)} {
    refresh
    return
  }
  set end &#91;db one {SELECT coalesce(max(seq),0) FROM undolog}&#93;
  if {$_undo(freeze)>=0 && $end>$_undo(freeze)} {set end $_undo(freeze)}
  set begin $_undo(firstlog)
  _start_interval
  if {$begin==$_undo(firstlog)} {
    refresh
    return
  }
  lappend _undo(undostack) &#91;list $begin $end&#93;
  set _undo(redostack) {}
  refresh
}

# proc:  ::undo::undo
# title: Do a single step of undo
#
proc undo {} {
  _step undostack redostack
}

# proc:  ::undo::redo
# title: Redo a single step
#
proc redo {} {
  _step redostack undostack
}

# proc:   ::undo::refresh
# title:  Update the status of controls after a database change
#
# The undo module calls this routine after any undo/redo in order to
# cause controls gray out appropriately depending on the current state
# of the database.  This routine works by invoking the status_refresh
# module in all top-level namespaces.
#
proc refresh {} {
  set body {}
  foreach ns &#91;namespace children ::&#93; {
    if {&#91;info proc ${ns}::status_refresh&#93;==""} continue
    append body ${ns}::status_refresh\n
  }
  proc ::undo::refresh {} $body
  refresh
}

# proc:   ::undo::reload_all
# title:  Redraw everything based on the current database
#
# The undo module calls this routine after any undo/redo in order to
# cause the screen to be completely redrawn based on the current database
# contents.  This is accomplished by calling the "reload" module in
# every top-level namespace other than ::undo.
#
proc reload_all {} {
  set body {}
  foreach ns &#91;namespace children ::&#93; {
    if {&#91;info proc ${ns}::reload&#93;==""} continue
    append body ${ns}::reload\n
  }
  proc ::undo::reload_all {} $body
  reload_all
}

##############################################################################
# The public interface to this module is above.  Routines and variables that
# follow (and whose names begin with "_") are private to this module.
##############################################################################

# state information
#
set _undo(active) 0
set _undo(undostack) {}
set _undo(redostack) {}
set _undo(pending) {}
set _undo(firstlog) 1
set _undo(startstate) {}


# proc:  ::undo::status_refresh
# title: Enable and/or disable menu options a buttons
#
proc status_refresh {} {
  variable _undo
  if {!$_undo(active) || &#91;llength $_undo(undostack)&#93;==0} {
    .mb.edit entryconfig Undo -state disabled
    .bb.undo config -state disabled
  } else {
    .mb.edit entryconfig Undo -state normal
    .bb.undo config -state normal
  }
  if {!$_undo(active) || &#91;llength $_undo(redostack)&#93;==0} {
    .mb.edit entryconfig Redo -state disabled
    .bb.redo config -state disabled
  } else {
    .mb.edit entryconfig Redo -state normal
    .bb.redo config -state normal
  }
}

# xproc:  ::undo::_create_triggers DB TABLE1 TABLE2 ...
# title:  Create change recording triggers for all tables listed
#
# Create a temporary table in the database named "undolog".  Create
# triggers that fire on any insert, delete, or update of TABLE1, TABLE2, ....
# When those triggers fire, insert records in undolog that contain
# SQL text for statements that will undo the insert, delete, or update.
#
proc _create_triggers {db args} {
  catch {$db eval {DROP TABLE undolog}}
  $db eval {CREATE TEMP TABLE undolog(seq integer primary key, sql text)}
  foreach tbl $args {
    set collist &#91;$db eval "pragma table_info($tbl)"&#93;
    set sql "CREATE TEMP TRIGGER _${tbl}_it AFTER INSERT ON $tbl BEGIN\n"
    append sql "  INSERT INTO undolog VALUES(NULL,"
    append sql "'DELETE FROM $tbl WHERE rowid='||new.rowid);\nEND;\n"

    append sql "CREATE TEMP TRIGGER _${tbl}_ut AFTER UPDATE ON $tbl BEGIN\n"
    append sql "  INSERT INTO undolog VALUES(NULL,"
    append sql "'UPDATE $tbl "
    set sep "SET "
    foreach {x1 name x2 x3 x4 x5} $collist {
      append sql "$sep$name='||quote(old.$name)||'"
      set sep ","
    }
    append sql " WHERE rowid='||old.rowid);\nEND;\n"

    append sql "CREATE TEMP TRIGGER _${tbl}_dt BEFORE DELETE ON $tbl BEGIN\n"
    append sql "  INSERT INTO undolog VALUES(NULL,"
    append sql "'INSERT INTO ${tbl}(rowid"
    foreach {x1 name x2 x3 x4 x5} $collist {append sql ,$name}
    append sql ") VALUES('||old.rowid||'"
    foreach {x1 name x2 x3 x4 x5} $collist {append sql ,'||quote(old.$name)||'}
    append sql ")');\nEND;\n"

    $db eval $sql
  }
}

# xproc:  ::undo::_drop_triggers DB
# title:  Drop all of the triggers that _create_triggers created
#
proc _drop_triggers {db} {
  set tlist &#91;$db eval {SELECT name FROM sqlite_temp_schema
                       WHERE type='trigger'}&#93;
  foreach trigger $tlist {
    if {!&#91;regexp {_.*_(i|u|d)t$} $trigger&#93;} continue
    $db eval "DROP TRIGGER $trigger;"
  }
  catch {$db eval {DROP TABLE undolog}}
}

# xproc: ::undo::_start_interval
# title: Record the starting conditions of an undo interval
#
proc _start_interval {} {
  variable _undo
  set _undo(firstlog) &#91;db one {SELECT coalesce(max(seq),0)+1 FROM undolog}&#93;
}

# xproc: ::undo::_step V1 V2
# title: Do a single step of undo or redo
#
# For an undo V1=="undostack" and V2=="redostack".  For a redo,
# V1=="redostack" and V2=="undostack".
#
proc _step {v1 v2} {
  variable _undo
  set op &#91;lindex $_undo($v1) end&#93;
  set _undo($v1) &#91;lrange $_undo($v1) 0 end-1&#93;
  foreach {begin end} $op break
  db eval BEGIN
  set q1 "SELECT sql FROM undolog WHERE seq>=$begin AND seq=$begin AND seq
This page last modified on  2022-04-18 02:55:50 UTC


c2: The UINT Collating Sequence
c5: uintcseq.html
c4:
1. Overview

 The UINT collating sequences is a loadable extension for
SQLite that implements a new collating sequence that compares text
containing unsigned integers in numeric order.

 The UINT collating sequence is not a standard part of SQLite.
It must be loaded as a separate extension.  The source code to
UINT is in the uint.c source file
in the ext/misc/ folder of the
SQLite source tree.

 The UINT collating sequence is not included in standard builds of
the SQLite library, but it is loaded by default in the CLI.  This
is typical of the CLI which loads various extensions above and beyond
what are available in the standard SQLite library.

 The UINT collating sequence works just like the default
BINARY collating sequence for text, except that embedded strings
of digits compare in numeric order.



Leading zeros are handled properly, in the sense that
they do not mess of the maginitude comparison of embedded
strings of digits.  "x00123y" is equal to "x123y".


Only unsigned integers are recognized.  Plus and minus
signs are ignored.  Decimal points and exponential notation
are ignored.


Embedded integers can be of arbitrary length.  Comparison
is not limited to integers that can be expressed as a
64-bit machine integer.


2. Example:



COLLATE binaryCOLLATE uint

'0000123457'
'123456'
'abc0000000010xyz'
'abc0010xyy'
'abc10xzz'
'abc674xyz'
'abc87xyz'
'abc9xyz'

'123456'
'0000123457'
'abc9xyz'
'abc0010xyy'
'abc0000000010xyz'
'abc10xzz'
'abc87xyz'
'abc674xyz'


This page last modified on  2022-01-08 05:02:57 UTC


c2: Datatypes In SQLite version 2
c5: datatypes.html
c4:
Datatypes In SQLite Version 2

1.0 &nbsp; Typelessness

SQLite is "typeless".  This means that you can store any
kind of data you want in any column of any table, regardless of the
declared datatype of that column.  
(See the one exception to this rule in section 2.0 below.)
This behavior is a feature, not
a bug.  A database is supposed to store and retrieve data and it 
should not matter to the database what format that data is in.
The strong typing system found in most other SQL engines and
codified in the SQL language spec is a misfeature -
it is an example of the implementation showing through into the
interface.  SQLite seeks to overcome this misfeature by allowing
you to store any kind of data into any kind of column and by
allowing flexibility in the specification of datatypes.



A datatype to SQLite is any sequence of zero or more names
optionally followed by a parenthesized lists of one or two
signed integers.  Notice in particular that a datatype may
be zero or more names.  That means that an empty
string is a valid datatype as far as SQLite is concerned.
So you can declare tables where the datatype of each column
is left unspecified, like this:



CREATE TABLE ex1(a,b,c);



Even though SQLite allows the datatype to be omitted, it is
still a good idea to include it in your CREATE TABLE statements,
since the data type often serves as a good hint to other
programmers about what you intend to put in the column. And
if you ever port your code to another database engine, that
other engine will probably require a datatype of some kind.
SQLite accepts all the usual datatypes.  For example:



CREATE TABLE ex2(
  a VARCHAR(10),
  b NVARCHAR(15),
  c TEXT,
  d INTEGER,
  e FLOAT,
  f BOOLEAN,
  g CLOB,
  h BLOB,
  i TIMESTAMP,
  j NUMERIC(10,5)
  k VARYING CHARACTER (24),
  l NATIONAL VARYING CHARACTER(16)
);



And so forth.  Basically any sequence of names optionally followed by 
one or two signed integers in parentheses will do.


2.0 &nbsp; The INTEGER PRIMARY KEY


One exception to the typelessness of SQLite is a column whose type
is INTEGER PRIMARY KEY.  (And you must use "INTEGER" not "INT".
A column of type INT PRIMARY KEY is typeless just like any other.)
INTEGER PRIMARY KEY columns must contain a 32-bit signed integer.  Any
attempt to insert non-integer data will result in an error.



INTEGER PRIMARY KEY columns can be used to implement the equivalent
of AUTOINCREMENT.  If you try to insert a NULL into an INTEGER PRIMARY
KEY column, the column will actually be filled with an integer that is
one greater than the largest key already in the table.  Or if the
largest key is 2147483647, then the column will be filled with a
random integer.  Either way, the INTEGER PRIMARY KEY column will be
assigned a unique integer.  You can retrieve this integer using
the sqlite_last_insert_rowid() API function or using the
last_insert_rowid() SQL function in a subsequent SELECT statement.


3.0 &nbsp; Comparison and Sort Order


SQLite is typeless for the purpose of deciding what data is allowed
to be stored in a column.  But some notion of type comes into play
when sorting and comparing data.  For these purposes, a column or
an expression can be one of two types: numeric and text.
The sort or comparison may give different results depending on which
type of data is being sorted or compared.



If data is of type text then the comparison is determined by
the standard C data comparison functions memcmp() or
strcmp().  The comparison looks at bytes from two inputs one
by one and returns the first non-zero difference.
Strings are '\000' terminated so shorter
strings sort before longer strings, as you would expect.



For numeric data, this situation is more complex.  If both inputs
look like well-formed numbers, then they are converted
into floating point values using atof() and compared numerically.
If one input is not a well-formed number but the other is, then the
number is considered to be less than the non-number.  If neither inputs
is a well-formed number, then strcmp() is used to do the
comparison.



Do not be confused by the fact that a column might have a "numeric"
datatype.  This does not mean that the column can contain only numbers.
It merely means that if the column does contain a number, that number
will sort in numerical order.



For both text and numeric values, NULL sorts before any other value.
A comparison of any value against NULL using operators like "&lt;" or
"&gt;=" is always false.


4.0 &nbsp; How SQLite Determines Datatypes


For SQLite version 2.6.3 and earlier, all values used the numeric datatype.
The text datatype appears in version 2.7.0 and later.  In the sequel it
is assumed that you are using version 2.7.0 or later of SQLite.



For an expression, the datatype of the result is often determined by
the outermost operator.  For example, arithmetic operators ("+", "*", "%")
always return a numeric results.  The string concatenation operator
("||") returns a text result.  And so forth.  If you are ever in doubt
about the datatype of an expression you can use the special typeof()
SQL function to determine what the datatype is.  For example:



sqlite&gt; SELECT typeof('abc'+123);
numeric
sqlite&gt; SELECT typeof('abc'||123);
text



For table columns, the datatype is determined by the type declaration
of the CREATE TABLE statement.  The datatype is text if and only if
the type declaration contains one or more of the following strings:



BLOB
CHAR
CLOB
TEXT



The search for these strings in the type declaration is case insensitive,
of course.  If any of the above strings occur anywhere in the type
declaration, then the datatype of the column is text.  Notice that
the type "VARCHAR" contains "CHAR" as a substring so it is considered
text.

If none of the strings above occur anywhere in the type declaration,
then the datatype is numeric.  Note in particular that the datatype for columns
with an empty type declaration is numeric.


5.0 &nbsp; Examples


Consider the following two command sequences:



CREATE TABLE t1(a INTEGER UNIQUE);        CREATE TABLE t2(b TEXT UNIQUE);
INSERT INTO t1 VALUES('0');               INSERT INTO t2 VALUES(0);
INSERT INTO t1 VALUES('0.0');             INSERT INTO t2 VALUES(0.0);


In the sequence on the left, the second insert will fail.  In this case,
the strings '0' and '0.0' are treated as numbers since they are being 
inserted into a numeric column but 0==0.0 which violates the uniqueness
constraint.  However, the second insert in the right-hand sequence works.  In
this case, the constants 0 and 0.0 are treated a strings which means that
they are distinct.

SQLite always converts numbers into double-precision (64-bit) floats
for comparison purposes.  This means that a long sequence of digits that
differ only in insignificant digits will compare equal if they
are in a numeric column but will compare unequal if they are in a text
column.  We have:


INSERT INTO t1                            INSERT INTO t2
   VALUES('12345678901234567890');           VALUES(12345678901234567890);
INSERT INTO t1                            INSERT INTO t2
   VALUES('12345678901234567891');           VALUES(12345678901234567891);


As before, the second insert on the left will fail because the comparison
will convert both strings into floating-point number first and the only
difference in the strings is in the 20-th digit which exceeds the resolution
of a 64-bit float.  In contrast, the second insert on the right will work
because in that case, the numbers being inserted are strings and are
compared using memcmp().


Numeric and text types make a difference for the DISTINCT keyword too:



CREATE TABLE t3(a INTEGER);               CREATE TABLE t4(b TEXT);
INSERT INTO t3 VALUES('0');               INSERT INTO t4 VALUES(0);
INSERT INTO t3 VALUES('0.0');             INSERT INTO t4 VALUES(0.0);
SELECT DISTINCT * FROM t3;                SELECT DISTINCT * FROM t4;



The SELECT statement on the left returns a single row since '0' and '0.0'
are treated as numbers and are therefore indistinct.  But the SELECT 
statement on the right returns two rows since 0 and 0.0 are treated
a strings which are different.
This page last modified on  2022-01-08 05:02:57 UTC


c2: Write-Ahead Logging
c3: 1. Overview
c5: wal.html#overview
c4:
The default method by which SQLite implements
atomic commit and rollback is a rollback journal.
Beginning with version 3.7.0 (2010-07-21), a new "Write-Ahead Log" option
(hereafter referred to as "WAL") is available.

There are advantages and disadvantages to using WAL instead of
a rollback journal.  Advantages include:




WAL is significantly faster in most scenarios.
WAL provides more concurrency as readers do not block writers and 
    a writer does not block readers.  Reading and writing can proceed 
    concurrently.
Disk I/O operations tends to be more sequential using WAL.
WAL uses many fewer fsync() operations and is thus less vulnerable to
    problems on systems where the fsync() system call is broken.


But there are also disadvantages:


WAL normally requires that the VFS 
    support shared-memory primitives.
    (Exception: WAL without shared memory)
    The built-in unix and windows VFSes
    support this but third-party extension VFSes for custom operating
    systems might not.
All processes using a database must be on the same host computer;
    WAL does not work over a network filesystem.
Transactions that involve changes against multiple ATTACHed
    databases are atomic for each individual database, but are not
    atomic across all databases as a set.
It is not possible to change the page_size after entering WAL
    mode, either on an empty database or by using VACUUM or by restoring
    from a backup using the backup API.  You must be in a rollback journal
    mode to change the page size.
It is not possible to open read-only WAL databases.
    The opening process must have write privileges for "-shm"
    wal-index shared memory file associated with the database, if that
    file exists, or else write access on the directory containing
    the database file if the "-shm" file does not exist.
    Beginning with version 3.22.0 (2018-01-22), a read-only 
    WAL-mode database file can be opened if
    the -shm and -wal files
    already exists or those files can be created or the
    database is immutable.
WAL might be very slightly slower (perhaps 1% or 2% slower)
    than the traditional rollback-journal approach
    in applications that do mostly reads and seldom write.
There is an additional quasi-persistent "-wal" file and
    "-shm" shared memory file associated with each
    database, which can make SQLite less appealing for use as an 
    application file-format.
There is the extra operation of checkpointing which, though automatic
    by default, is still something that application developers need to
    be mindful of.
WAL works best with smaller transactions.  WAL does
    not work well for very large transactions.  For transactions larger than
    about 100 megabytes, traditional rollback journal modes will likely
    be faster.  For transactions in excess of a gigabyte, WAL mode may 
    fail with an I/O or disk-full error.
    It is recommended that one of the rollback journal modes be used for
    transactions larger than a few dozen megabytes.
    Beginning with version 3.11.0 (2016-02-15), 
    WAL mode works as efficiently with
    large transactions as does rollback mode.


c2: Write-Ahead Logging
c3: 2. How WAL Works
c5: wal.html#how_wal_works
c4:
The traditional rollback journal works by writing a copy of the
original unchanged database content into a separate rollback journal file
and then writing changes directly into the database file.  In the
event of a crash or ROLLBACK, the original content contained in the
rollback journal is played back into the database file to
revert the database file to its original state.  The COMMIT occurs
when the rollback journal is deleted.

The WAL approach inverts this.  The original content is preserved
in the database file and the changes are appended into a separate
WAL file.  A COMMIT occurs when a special record indicating a commit
is appended to the WAL.  Thus a COMMIT can happen without ever writing
to the original database, which allows readers to continue operating
from the original unaltered database while changes are simultaneously being
committed into the WAL.  Multiple transactions can be appended to the
end of a single WAL file.


c2: Write-Ahead Logging
c3: 2.1. Checkpointing
c5: wal.html#checkpointing
c4:
Of course, one wants to eventually transfer all the transactions that
are appended in the WAL file back into the original database.  Moving
the WAL file transactions back into the database is called a
"checkpoint".

Another way to think about the difference between rollback and 
write-ahead log is that in the rollback-journal
approach, there are two primitive operations, reading and writing,
whereas with a write-ahead log
there are now three primitive operations:  reading, writing, and
checkpointing.

By default, SQLite does a checkpoint automatically when the WAL file
reaches a threshold size of 1000 pages.  (The
SQLITE_DEFAULT_WAL_AUTOCHECKPOINT compile-time option can be used to
specify a different default.) Applications using WAL do
not have to do anything in order to for these checkpoints to occur.  
But if they want to, applications can adjust the automatic checkpoint
threshold.  Or they can turn off the automatic checkpoints and run 
checkpoints during idle moments or in a separate thread or process.


c2: Write-Ahead Logging
c3: 2.2. Concurrency
c5: wal.html#concurrency
c4:
When a read operation begins on a WAL-mode database, it first
remembers the location of the last valid commit record in the WAL.
Call this point the "end mark".  Because the WAL can be growing and
adding new commit records while various readers connect to the database,
each reader can potentially have its own end mark.  But for any
particular reader, the end mark is unchanged for the duration of the
transaction, thus ensuring that a single read transaction only sees
the database content as it existed at a single point in time.

When a reader needs a page of content, it first checks the WAL to
see if that page appears there, and if so it pulls in the last copy
of the page that occurs in the WAL prior to the reader's end mark.
If no copy of the page exists in the WAL prior to the reader's end mark,
then the page is read from the original database file.  Readers can
exist in separate processes, so to avoid forcing every reader to scan
the entire WAL looking for pages (the WAL file can grow to
multiple megabytes, depending on how often checkpoints are run), a
data structure called the "wal-index" is maintained in shared memory
which helps readers locate pages in the WAL quickly and with a minimum
of I/O.  The wal-index greatly improves the performance of readers,
but the use of shared memory means that all readers must exist on the
same machine.  This is why the write-ahead log implementation will not
work on a network filesystem.

Writers merely append new content to the end of the WAL file.
Because writers do nothing that would interfere with the actions of
readers, writers and readers can run at the same time.  However,
since there is only one WAL file, there can only be one writer at
a time.

A checkpoint operation takes content from the WAL file
and transfers it back into the original database file.
A checkpoint can run concurrently with readers, however the checkpoint
must stop when it reaches a page in the WAL that is past the end mark
of any current reader.  The checkpoint has to stop at that point because
otherwise it might overwrite part of the database file that the reader
is actively using.  The checkpoint remembers (in the wal-index) how far
it got and will resume transferring content from the WAL to the database
from where it left off on the next invocation.

Thus a long-running read transaction can prevent a checkpointer from
making progress.  But presumably every read transaction will eventually
end and the checkpointer will be able to continue.

Whenever a write operation occurs, the writer checks how much progress
the checkpointer has made, and if the entire WAL has been transferred into
the database and synced and if no readers are making use of the WAL, then
the writer will rewind the WAL back to the beginning and start putting new
transactions at the beginning of the WAL.  This mechanism prevents a WAL
file from growing without bound.


c2: Write-Ahead Logging
c3: 2.3. Performance Considerations
c5: wal.html#performance_considerations
c4:
Write transactions are very fast since they only involve writing
the content once (versus twice for rollback-journal transactions)
and because the writes are all sequential.  Further, syncing the
content to the disk is not required, as long as the application is
willing to sacrifice durability following a power loss or hard reboot.
(Writers sync the WAL on every transaction commit if
PRAGMA synchronous is set to FULL but omit this sync if
PRAGMA synchronous is set to NORMAL.)

On the other hand, read performance deteriorates as the WAL file
grows in size since each reader must check the WAL file for the content
and the time needed to check the WAL file is proportional
to the size of the WAL file.  The wal-index helps find content
in the WAL file much faster, but performance still falls off with
increasing WAL file size.  Hence, to maintain good read performance 
it is important to keep the WAL file size down by
running checkpoints at regular intervals.

Checkpointing does require sync operations in order to avoid
the possibility of database corruption following a power loss
or hard reboot.  The WAL must be synced to persistent storage
prior to moving content from the WAL into the database and the
database file must by synced prior to resetting the WAL.
Checkpoint also requires more seeking.
The checkpointer makes an effort to
do as many sequential page writes to the database as it can (the pages
are transferred from WAL to database in ascending order) but even
then there will typically be many seek operations interspersed among
the page writes.  These factors combine to make checkpoints slower than
write transactions.

The default strategy is to allow successive write transactions to
grow the WAL until the WAL becomes about 1000 pages in size, then to
run a checkpoint operation for each subsequent COMMIT until the WAL
is reset to be smaller than 1000 pages.  By default, the checkpoint will be
run automatically by the same thread that does the COMMIT that pushes
the WAL over its size limit.  This has the effect of causing most
COMMIT operations to be very fast but an occasional COMMIT (those that trigger
a checkpoint) to be much slower.  If that effect is undesirable, then
the application can disable automatic checkpointing and run the
periodic checkpoints in a separate thread, or separate process.
(Links to commands and interfaces to accomplish this are
shown below.)


Note that with PRAGMA synchronous set to NORMAL, the checkpoint
is the only operation to issue an I/O barrier or sync operation
(fsync() on unix or FlushFileBuffers() on windows).  If an application
therefore runs checkpoint in a separate thread or process, the main
thread or process that is doing database queries and updates will never
block on a sync operation.  This helps to prevent "latch-up" in applications
running on a busy disk drive.  The downside to
this configuration is that transactions are no longer durable and
might rollback following a power failure or hard reset.


Notice too that there is a tradeoff between average read performance
and average write performance.  To maximize the read performance,
one wants to keep the WAL as small as possible and hence run checkpoints
frequently, perhaps as often as every COMMIT.  To maximize
write performance, one wants to amortize the cost of each checkpoint
over as many writes as possible, meaning that one wants to run checkpoints
infrequently and let the WAL grow as large as possible before each 
checkpoint.  The decision of how often to run checkpoints may therefore
vary from one application to another depending on the relative read
and write performance requirements of the application.
The default strategy is to run a checkpoint once the WAL
reaches 1000 pages and this strategy seems to work well in test applications on 
workstations, but other strategies might work better on different 
platforms or for different workloads.


c2: Write-Ahead Logging
c3: 3. Activating And Configuring WAL Mode
c5: wal.html#activating_and_configuring_wal_mode
c4:
An SQLite database connection defaults to 
journal_mode=DELETE.  To convert to WAL mode, use the
following pragma:


PRAGMA journal_mode=WAL;


The journal_mode pragma returns a string which is the new journal mode.
On success, the pragma will return the string "wal".  If 
the conversion to WAL could not be completed (for example, if the VFS
does not support the necessary shared-memory primitives) then the
journaling mode will be unchanged and the string returned from the
primitive will be the prior journaling mode (for example "delete").


c2: Write-Ahead Logging
c3: 3.1. Automatic Checkpoint
c5: wal.html#automatic_checkpoint
c4:
By default, SQLite will automatically checkpoint whenever a COMMIT
occurs that causes the WAL file to be 1000 pages or more in size, or when the 
last database connection on a database file closes.  The default 
configuration is intended to work well for most applications.
But programs that want more control can force a checkpoint
using the wal_checkpoint pragma or by calling the
sqlite3_wal_checkpoint() C interface.  The automatic checkpoint
threshold can be changed or automatic checkpointing can be completely
disabled using the wal_autocheckpoint pragma or by calling the
sqlite3_wal_autocheckpoint() C interface.  A program can also 
use sqlite3_wal_hook() to register a callback to be invoked whenever
any transaction commits to the WAL.  This callback can then invoke
sqlite3_wal_checkpoint() or sqlite3_wal_checkpoint_v2() based on whatever
criteria it thinks is appropriate.  (The automatic checkpoint mechanism
is implemented as a simple wrapper around sqlite3_wal_hook().)


c2: Write-Ahead Logging
c3: 3.2. Application-Initiated Checkpoints
c5: wal.html#application_initiated_checkpoints
c4:
An application can initiate a checkpoint using any writable database
connection on the database simply by invoking
sqlite3_wal_checkpoint() or sqlite3_wal_checkpoint_v2().
There are three subtypes of checkpoints that vary in their aggressiveness:
PASSIVE, FULL, and RESTART.  The default checkpoint style is PASSIVE, which
does as much work as it can without interfering with other database
connections, and which might not run to completion if there are
concurrent readers or writers.
All checkpoints initiated by sqlite3_wal_checkpoint() and
by the automatic checkpoint mechanism are PASSIVE.  FULL and RESTART
checkpoints try harder to run the checkpoint to completion and can only
be initiated by a call to sqlite3_wal_checkpoint_v2().  See the
sqlite3_wal_checkpoint_v2() documentation for additional information
on FULL and RESET checkpoints.


