# page_content

c2: Generated Columns
c3: 1. Introduction
c5: gencol.html#introduction
c4:
Generated columns (also sometimes called "computed columns")
are columns of a table whose values are a function of other columns
in the same row.
Generated columns can be read, but their values can not be directly
written.  The only way to change the value of a generated column is to
modify the values of the other columns used to calculate
the generated column.


c2: Generated Columns
c3: 2. Syntax
c5: gencol.html#syntax
c4:
Syntactically, generated columns are designated using a
"GENERATED ALWAYS" column-constraint.  For example:

CREATE TABLE t1(
   a INTEGER PRIMARY KEY,
   b INT,
   c TEXT,
   d INT GENERATED ALWAYS AS (a*abs(b)) VIRTUAL,
   e TEXT GENERATED ALWAYS AS (substr(c,b,b+1)) STORED
);


The statement above has three ordinary columns, "a" (the PRIMARY KEY),
"b", and "c", and two generated columns "d" and "e".

The "GENERATED ALWAYS" keywords at the beginning of the constraint
and the "VIRTUAL" or "STORED" keyword at the end are all optional.
Only the "AS" keyword and the parenthesized expression are required.
If the trailing "VIRTUAL" or "STORED" keyword is omitted, then
VIRTUAL is the default.  Hence, the example statement above could
be simplified to just:

CREATE TABLE t1(
   a INTEGER PRIMARY KEY,
   b INT,
   c TEXT,
   d INT AS (a*abs(b)),
   e TEXT AS (substr(c,b,b+1)) STORED
);


c2: Generated Columns
c3: 2.1. VIRTUAL versus STORED columns
c5: gencol.html#virtual_versus_stored_columns
c4:
Generated columns can be either VIRTUAL or STORED.  The value of
a VIRTUAL column is computed when read, whereas the value of a STORED
column is computed when the row is written.  STORED columns take up space
in the database file, whereas VIRTUAL columns use more CPU cycles when
being read.

From the point of view of SQL, STORED and VIRTUAL columns are almost
exactly the same.  Queries against either class of generated column
produce the same results.  The only functional difference is that
one cannot add new STORED columns using the
ALTER TABLE ADD COLUMN command.  Only VIRTUAL columns can be added
using ALTER TABLE.


c2: Generated Columns
c3: 2.2. Capabilities
c5: gencol.html#capabilities
c4:
Generated columns can have a datatype.  SQLite attempts to transform
the result of the generating expression into that datatype using the
same affinity rules as for ordinary columns.


Generated columns may have NOT NULL, CHECK, and UNIQUE constraints,
and foreign key constraints, just like ordinary columns.


Generated columns can participate in indexes, just like ordinary
columns.


The expression of a generated column can refer to any of the
other declared columns in the table, including other generated columns,
as long as the expression does not directly or indirectly refer back
to itself.


Generated columns can occur anywhere in the table definition.  Generated
columns can be interspersed among ordinary columns.  It is not necessary
to put generated columns at the end of the list of columns in the
table definition, as is shown in the examples above.


c2: Generated Columns
c3: 2.3. Limitations
c5: gencol.html#limitations
c4:
Generated columns may not have a default value (they may not use the
"DEFAULT" clause).  The value of a generated column is always the value
specified by the expression that follows the "AS" keyword.


Generated columns may not be used as part of the PRIMARY KEY.
(Future versions of SQLite might relax this constraint for STORED columns.)


The expression of a generated column may only reference
constant literals and columns within the same row, and may only use
scalar deterministic functions.  The expression may not use subqueries,
aggregate functions, window functions, or table-valued functions.


The expression of a generated column may refer to other generated columns
in the same row, but no generated column can depend upon itself, either
directly or indirectly.  

The expression of a generated column may not directly reference
the ROWID, though it can reference the INTEGER PRIMARY KEY column,
which is often the same thing.


Every table must have at least one non-generated column.


It is not possible to ALTER TABLE ADD COLUMN a STORED column.
One can add a VIRTUAL column, however.


The datatype and collating sequence of the generated column are determined
only by the datatype and COLLATE clause on the column definition.
The datatype and collating sequence of the GENERATED ALWAYS AS expression
have no affect on the datatype and collating sequence of the column itself.


Generated columns are not included in the list of columns provided by
the PRAGMA table_info statement.  But they are included in the output of
the newer PRAGMA table_xinfo statement.


c2: Generated Columns
c3: 3. Compatibility
c5: gencol.html#compatibility
c4:
Generated column support was added with SQLite version 3.31.0
(2020-01-22).  If an earlier version of SQLite attempts to read
a database file that contains a generated column in its schema, then
that earlier version will perceive the generated column syntax as an
error and will report that the database schema is corrupt.

To clarify:  SQLite version 3.31.0 can read and write any database
created by any prior version of SQLite going back to 
SQLite 3.0.0 (2004-06-18).  And, earlier versions of SQLite,
prior to 3.31.0, can read and write databases created by SQLite
version 3.31.0 and later as long
as the database schema does not contain features, such as
generated columns, that are not understood by the earlier version.
Problems only arise if you create a new database that contains
generated columns, using SQLite version 3.31.0 or later, and then
try to read or write that database file using an earlier version of
SQLite that does not understand generated columns.
This page last modified on  2022-11-09 20:11:26 UTC


c2: The SQLite Zipfile Module
c3: 1. Overview
c5: zipfile.html#overview
c4:
The zipfile module provides read/write access to simple 
ZIP archives.
The current implementation has the following restrictions:


   Does not support encryption.
   Does not support ZIP archives that span multiple files.
   Does not support zip64 extensions.
   The only compression algorithm supported is
       "deflate".


 Some or all of these restrictions may be removed in the future.


c2: The SQLite Zipfile Module
c3: 2. Obtaining and Compiling Zipfile
c5: zipfile.html#obtaining_and_compiling_zipfile
c4:
The code for the zipfile module is found in the 
ext/misc/zipfile.c
file of the
main SQLite source tree.
It may be compiled into an SQLite 
loadable extension using a command like:

gcc -g -fPIC -shared zipfile.c -o zipfile.so


Alternatively, the zipfile.c file may be compiled into the application. 
In this case, the following function should be invoked to register the
extension with each new database connection:

int sqlite3_zipfile_init(sqlite3 *db, void*, void*);


 The first argument passed should be the database handle to register the
extension with. The second and third arguments should both be passed 0.

 Zipfile is included in most builds of the command-line shell.


c2: The SQLite Zipfile Module
c3: 3. Using Zipfile
c5: zipfile.html#using_zipfile
c4:
The zipfile module provides three similar interfaces for accessing, updating
and creating zip file archives:


   A table-valued function, which provides read-only access to existing
       archives, either from the file-system or in-memory.
   A virtual table, which provides read and write access to archives
       stored in the file-system.
   An SQL aggregate function, which can be used to create new archives
       in memory.


The zipfile module provides two similar interfaces for accessing zip
archives. A table-valued function, which provides read-only access to
existing archives, and a virtual table interface, which provides both
read and write access.


c2: The SQLite Zipfile Module
c3: 3.1. Table-Valued Function (read-only access)
c5: zipfile.html#table_valued_function_read_only_access_
c4:
For reading existing zip archives, the Zipfile module provides a
table-valued function that accepts a single argument. If the argument
is a text value, then it is a path to a zip archive to read from the
file-system. Or, if the argument is an SQL blob, then it is the zip
archive data itself.

For example, to inspect the contents of zip archive "test.zip" from 
the current directory:

SELECT * FROM zipfile('test.zip');


Or, from the SQLite shell tool (the readfile() 
function reads the contents of a file from the file-system and returns it as a
blob):

SELECT * FROM zipfile( readfile('test.zip') );


The table-valued function returns one row for each record (file, 
directory or symbolic link) in the zip archive. Each row has the 
following columns:


Column&nbsp;NameContents
name  File name/path for the zip file record.
mode  UNIX mode, as returned by stat(2) for the zip file record (an
                  integer). This identifies the type of record (file, directory
                  or symbolic link), and the associated user/group/all 
                  permissions.
mtime  UTC timestamp, in seconds since the UNIX epoch (an integer).
sz     Size of associated data in bytes after it has been 
                   uncompressed (an integer).
rawdata  Raw (possibly compressed) data associated with zip file
                     entry (a blob).
data  If the compression method for the record is either 0 or 8
                  (see below), then the uncompressed data associated with the
                  zip file entry. Or, if the compression method is not 0 or 8, 
                  this column contains a NULL value.
method  The compression method used to compress the data (an
                    integer). The value 0 indicates that the data is stored
                    in the zip archive without compression. 8 means the
                    raw deflate algorithm.


c2: The SQLite Zipfile Module
c3: 3.2. Virtual Table Interface (read/write access)
c5: zipfile.html#virtual_table_interface_read_write_access_
c4:
In order to create or modify an existing zip file, a "zipfile" virtual 
table must be created in the database schema. The CREATE VIRTUAL TABLE
statement expects a path to the zip file as its only argument. For example, to
write to zip file "test.zip" in the current directory, a zipfile table may be
created using:

CREATE VIRTUAL TABLE temp.zip USING zipfile('test.zip');


Such a virtual table has the same columns as the table-valued function
described in the previous section. It may be read from using a SELECT 
statement in the same way as the table-valued function can. 

Using the virtual table interface, new entries may be added to a zip
archive by inserting new rows into the virtual table. Entries may be
removed by deleting rows or modified by updating them.


c2: The SQLite Zipfile Module
c3: 3.2.1. Adding Entries to a Zip Archive
c5: zipfile.html#adding_entries_to_a_zip_archive
c4:
Entries may be added to a zip archive by inserting new rows. The easiest
way to do this is to specify values for the "name" and "data" columns only and
have zipfile fill in sensible defaults for other fields.  To insert a directory
into the archive, set the "data" column to NULL.  For example, to add the
directory "dir1" and the file "m.txt" containing the text "abcdefghi" to zip
archive "test.zip":

INSERT INTO temp.zip(name, data) VALUES('dir1', NULL);           -- Add directory 
INSERT INTO temp.zip(name, data) VALUES('m.txt', 'abcdefghi');   -- Add regular file 


When a directory is inserted, if the "name" value does not end with
a '/' character, the zipfile module appends one. This is necessary for
compatibility with other programs (most notably "info-zip") that 
manipulate zip archives.

To insert a symbolic link, the user must also supply a "mode" value.
For example, to add a symbolic link from "link.txt" to "m.txt":

INSERT INTO temp.zip(name, mode, data) VALUES('link.txt', 'lrwxrw-rw-', 'm.txt');


The following rules and caveats apply to the values specified as part of
each INSERT statement:


Columns  Notes
 name
   A non-NULL text value must be specified for the name column. 
       It is an error if the specified name already exists in the
       archive. 

 mode
   If NULL is inserted into the mode column, then the mode of the
       new archive entry is automatically set to either 33188 (-rw-r--r--)
       or 16877 (drwxr-xr-x), depending on whether or not the values 
       specified for columns "sz", "data" and "rawdata" indicate that
       the new entry is a directory.
       
       If the specified value is an integer (or text that looks like
       an integer), it is inserted verbatim. If the value is not a valid UNIX
       mode, some programs may behave unexpectedly when extracting files
       from the archive.

       Finally, if the value specified for this column is not an integer
       or a NULL, then it is assumed to be a UNIX permissions string similar
       to those output by the "ls -l" command (e.g. "-rw-r--r--", "drwxr-xr-x"
       etc.). In this case, if the string cannot be parsed it is an error.

 mtime
   If NULL is inserted into the mtime column, then the timestamp
       of the new entry is set to the current time. Otherwise, the specified
       value is interpreted as an integer and used as is.

 sz
   This column must be set to NULL. If a non-NULL value is inserted into
       this column, or if a new non-NULL value is provided using an UPDATE
       statement, it is an error.

 rawdata
   This column must be set to NULL. If a non-NULL value is inserted into
       this column, or if a new non-NULL value is provided using an UPDATE
       statement, it is an error.

 data

       To insert a directory into the archive, this field must be set to 
       NULL. In this case if a value was explicitly specified for the "mode"
       column, then it must be consistent with a directory (i.e. it must be
       true that (mode & 0040000)=0040000). 

       Otherwise, the value inserted into this field is the file contents
       for a regular file, or the target of a symbolic link.
 method

       This field must be set one of integer values 0 and 8, or else to
       NULL. 

       For a directory entry, any value inserted into this field is ignored.
       Otherwise, if it is set to 0, then the file data or symbolic link
       target is stored as is in the zip archive and the compression method
       set to 0. If it is set to 8, then the file data or link target is
       compressed using deflate compression before it is stored and the
       compression method set to 8. Finally, if a NULL value is written
       to this field, the zipfile module automatically decides whether
       or not to compress the data before storing it.


 Specifying an explicit value for the rowid field as part of an INSERT
statement is not supported. Any value supplied is ignored.


c2: The SQLite Zipfile Module
c3: 3.2.2. Deleting Zip Archive Entries
c5: zipfile.html#_deleting_zip_archive_entries_
c4:
Records may be removed from an existing zip archive by deleting the
corresponding rows. For example, to remove file "m.txt" from zip archive
"test.zip" using the virtual table created above:

DELETE FROM temp.zip WHERE name = 'm.txt';


Note that deleting records from a zip archive does not reclaim the 
space used within the archive - it merely removes an entry from the
archives "Central Directory Structure", making the entry inaccessible.
One way to work around this inefficiency is to create a new zip 
archive based on the contents of the edited archive. For example, after
editing the archive accessed via virtual table temp.zzz:

-- Create a new, empty, archive: 
CREATE VIRTUAL TABLE temp.newzip USING zipfile('new.zip');

-- Copy the contents of the existing archive into the new archive
INSERT INTO temp.newzip(name, mode, mtime, data, method)
    SELECT name, mode, mtime, data, method FROM temp.zzz;


c2: The SQLite Zipfile Module
c3: 3.2.3. Updating Existing Zip Archive Entries
c5: zipfile.html#_updating_existing_zip_archive_entries_
c4:
Existing zip archive entries may be modified using UPDATE statements.

The three leftmost columns of a zipfile virtual table, "name", "mode" 
and "mtime", may each be set to any value that may be inserted into the same
column (see above). If either "mode" or "mtime" is set to NULL, the final 
value is determined as described for an INSERT of a NULL value - the current
time for "mtime" and either 33188 or 16877 for "mode", depending on whether 
or not the values specified for the next four columns of the zipfile table
indicate that the entry is a directory or a file.

It is an error to attempt to set the sz or rawdata field to any value
other than NULL. 

The data and method columns may also be set as described for an INSERT
above.


c2: The SQLite Zipfile Module
c3: 3.3. The zipfile() Aggregate Function
c5: zipfile.html#_the_zipfile_aggregate_function_
c4:
New zip archives may be constructed entirely within memory using the
zipfile() aggregate function. Each row visited by the aggregate function
adds an entry to the zip archive. The value returned is a blob containing
the entire archive image.

 The zipfile() aggregate function may be called with 2, 4 or 5 
arguments. If it is called with 5 arguments, then the entry added to
the archive is equivalent to inserting the same values into the "name", 
"mode", "mtime", "data" and "method" columns of a zipfile virtual table.

 If zipfile() is invoked with 2 arguments, then the entry added to
the archive is equivalent to that added by inserting the same two values into
the "name" and "data" columns of a zipfile virtual table, with all
other values set to NULL. If invoked with 4 arguments, it is equivalent
to inserting the 4 values into the "name", "mode", "mtime" and "data"
columns. In other words, the following pairs of queries are equivalent:

SELECT zipfile(name, data) ...
SELECT zipfile(name, NULL, NULL, data, NULL) ...

SELECT zipfile(name, mode, mtime, data) ...
SELECT zipfile(name, mode, mtime, data, NULL) ...


 For example, to create an archive containing two text files, "a.txt" and
"b.txt", containing the text "abc" and "123" respectively:

WITH contents(name, data) AS (
  VALUES('a.txt', 'abc') UNION ALL
  VALUES('b.txt', '123')
)
SELECT zipfile(name, data) FROM contents;

This page last modified on  2023-06-07 13:17:51 UTC


c2: Pragma statements supported by SQLite
c5: pragma.html#
c4:
PRAGMA Statements



The PRAGMA statement is an SQL extension specific to SQLite and used to 
modify the operation of the SQLite library or to query the SQLite library for 
internal (non-table) data. The PRAGMA statement is issued using the same
interface as other SQLite commands (e.g. SELECT, INSERT) but is
different in the following important respects:


The pragma command is specific to SQLite and is
    not compatible with any other SQL database engine.
Specific pragma statements may be removed and others added in future
    releases of SQLite. There is no guarantee of backwards compatibility.
No error messages are generated if an unknown pragma is issued.
    Unknown pragmas are simply ignored. This means if there is a typo in 
    a pragma statement the library does not inform the user of the fact.
Some pragmas take effect during the SQL compilation stage, not the
    execution stage. This means if using the C-language sqlite3_prepare(), 
    sqlite3_step(), sqlite3_finalize() API (or similar in a wrapper 
    interface), the pragma may run during the sqlite3_prepare() call,
    not during the sqlite3_step() call as normal SQL statements do.
    Or the pragma might run during sqlite3_step() just like normal
    SQL statements.  Whether or not the pragma runs during sqlite3_prepare()
    or sqlite3_step() depends on the pragma and on the specific release
    of SQLite.
The EXPLAIN and EXPLAIN QUERY PLAN prefixes to SQL statements
    only affect the behavior of the statement during sqlite3_step().
    That means that PRAGMA statements that take effect during
    sqlite3_prepare() will behave the same way regardless of whether or
    not they are prefaced by "EXPLAIN".


The C-language API for SQLite provides the SQLITE_FCNTL_PRAGMA
file control which gives VFS implementations the
opportunity to add new PRAGMA statements or to override the meaning of
built-in PRAGMA statements.



PRAGMA command syntax
pragma-stmt:
hide
 
 
pragma-value:
hide
 
 
signed-number:
show
 
 






A pragma can take either zero or one argument.  The argument is may be either
in parentheses or it may be separated from the pragma name by an equal sign.
The two syntaxes yield identical results.
In many pragmas, the argument is a boolean.  The boolean can be one of:



1 yes true on0 no false off


Keyword arguments can optionally appear in quotes.  
(Example:  'yes' &#91;FALSE&#93;.) Some pragmas
takes a string literal as their argument.  When pragma takes a keyword
argument, it will usually also take a numeric equivalent as well.
For example, "0" and "no" mean the same thing, as does "1" and "yes".
When querying the value of a setting, many pragmas return the number
rather than the keyword.

A pragma may have an optional schema-name
before the pragma name.
The schema-name is the name of an ATTACH-ed database 
or "main" or "temp" for the main and the TEMP databases.  If the optional
schema name is omitted, "main" is assumed.  In some pragmas, the schema
name is meaningless and is simply ignored.  In the documentation below,
pragmas for which the schema name is meaningful are shown with a
"schema." prefix.



PRAGMA functions



PRAGMAs that return results and that have no side-effects can be 
accessed from ordinary SELECT statements as table-valued functions.
For each participating PRAGMA, the corresponding table-valued function
has the same name as the PRAGMA with a 7-character "pragma_" prefix.
The PRAGMA argument and schema, if any, are passed as arguments to the
table-valued function, with the schema as an optional, last argument.

For example, information about the columns in an index can be
read using the index_info pragma as follows:


PRAGMA index_info('idx52');


Or, the same content can be read using:


SELECT * FROM pragma_index_info('idx52');


The advantage of the table-valued function format is that the query
can return just a subset of the PRAGMA columns, can include a WHERE clause,
can use aggregate functions, and the table-valued function can be just
one of several data sources in a join.
For example, to get a list of all indexed columns in a schema, one
could query:


SELECT DISTINCT m.name || '.' || ii.name AS 'indexed-columns'
  FROM sqlite_schema AS m,
       pragma_index_list(m.name) AS il,
       pragma_index_info(il.name) AS ii
 WHERE m.type='table'
 ORDER BY 1;



Additional notes:


Table-valued functions exist only for built-in PRAGMAs, not for PRAGMAs
defined using the SQLITE_FCNTL_PRAGMA file control.

Table-valued functions exist only for PRAGMAs that return results and
that have no side-effects.

This feature could be used to implement
information schema
by first creating a separate schema using

ATTACH ':memory:' AS 'information_schema';

Then creating
VIEWs in that schema that implement the official information schema
tables using table-valued PRAGMA functions.

The table-valued functions for PRAGMA feature was added
in SQLite version 3.16.0 (2017-01-02).  Prior versions of SQLite
cannot use this feature.



List Of PRAGMAs


analysis_limit
application_id
auto_vacuum
automatic_index
busy_timeout
cache_size
cache_spill
case_sensitive_like&sup1;
cell_size_check
checkpoint_fullfsync
collation_list
compile_options
count_changes&sup1;
data_store_directory&sup1;
data_version
database_list
default_cache_size&sup1;
defer_foreign_keys
empty_result_callbacks&sup1;
encoding
foreign_key_check
foreign_key_list
foreign_keys
freelist_count
full_column_names&sup1;
fullfsync
function_list
hard_heap_limit
ignore_check_constraints
incremental_vacuum
index_info
index_list
index_xinfo
integrity_check
journal_mode
journal_size_limit
legacy_alter_table
legacy_file_format
locking_mode
max_page_count
mmap_size
module_list
optimize
page_count
page_size
parser_trace&sup2;
pragma_list
query_only
quick_check
read_uncommitted
recursive_triggers
reverse_unordered_selects
schema_version&sup3;
secure_delete
short_column_names&sup1;
shrink_memory
soft_heap_limit
stats&sup3;
synchronous
table_info
table_list
table_xinfo
temp_store
temp_store_directory&sup1;
threads
trusted_schema
user_version
vdbe_addoptrace&sup2;
vdbe_debug&sup2;
vdbe_listing&sup2;
vdbe_trace&sup2;
wal_autocheckpoint
wal_checkpoint
writable_schema&sup3;



Notes:

Pragmas whose names are struck through
are deprecated. Do not use them. They exist
for historical compatibility.
These pragmas are only available in builds using non-standard
compile-time options.
These pragmas are used for testing SQLite and are not recommended
for use in application programs.


c2: Pragma statements supported by SQLite
c3: PRAGMA analysis_limit
c5: pragma.html#pragma_analysis_limit
c4:
PRAGMA analysis_limit;
        PRAGMA analysis_limit = N;
   Query or change a limit on the approximate ANALYZE setting.
      This is the approximate number of
      rows examined in each index by the ANALYZE command.
      If the argument N is omitted, then the analysis limit
      is unchanged.
      If the limit is zero, then the analysis limit is disabled and
      the ANALYZE command will examine all rows of each index.
      If N is greater than zero, then the analysis limit is set to N
      and subsequent ANALYZE commands will stop analyzing
      each index after it has examined approximately N rows.
      If N is a negative number or something other than an integer value,
      then the pragma behaves as if the N argument was omitted.
      In all cases, the value returned is the new analysis limit used
      for subsequent ANALYZE commands.
   This pragma can be used to help the ANALYZE command run faster
      on large databases.  The results of analysis are not as good
      when only part of each index is examined, but the results are
      usually good enough.  Setting N to 100 or 1000 allows the
      ANALYZE command to run very quickly, even on multi-gigabyte
      database files.  This pragma is particularly useful in combination
      with PRAGMA optimize.
   This pragma was added in SQLite version 3.32.0 (2020-05-22).
      The current implementation only uses the lower 31 bits of the
      N value - higher order bits are silently ignored.  Future versions
      of SQLite might begin using higher order bits.


c2: Pragma statements supported by SQLite
c3: PRAGMA application_id
c5: pragma.html#pragma_application_id
c4:
PRAGMA schema.application_id;
     PRAGMA schema.application_id = integer ;
  
    The application_id PRAGMA is used to query or set the 32-bit
       signed big-endian "Application ID" integer located at offset
       68 into the database header.  Applications that use SQLite as their
       application file-format should set the Application ID integer to
       a unique integer so that utilities such as 
       file(1) can determine the specific
       file type rather than just reporting "SQLite3 Database".  A list of
       assigned application IDs can be seen by consulting the
       magic.txt file in the SQLite source repository.

   See also the user_version pragma.


c2: Pragma statements supported by SQLite
c3: PRAGMA auto_vacuum
c5: pragma.html#pragma_auto_vacuum
c4:
PRAGMA schema.auto_vacuum;
          PRAGMA schema.auto_vacuum = 
           0 | NONE | 1 | FULL | 2 | INCREMENTAL;

    Query or set the auto-vacuum status in the database.

    The default setting for auto-vacuum is 0 or "none",
    unless the SQLITE_DEFAULT_AUTOVACUUM compile-time option is used.
    The "none" setting means that auto-vacuum is disabled.
    When auto-vacuum is disabled and data is deleted data from a database,
    the database file remains the same size.  Unused database file 
    pages are added to a "freelist" and reused for subsequent inserts.  So
    no database file space is lost.  However, the database file does not
    shrink.  In this mode the VACUUM
    command can be used to rebuild the entire database file and
    thus reclaim unused disk space.

    When the auto-vacuum mode is 1  or "full", the freelist pages are
    moved to the end of the database file and the database file is truncated
    to remove the freelist pages at every transaction commit.
    Note, however, that auto-vacuum only truncates the freelist pages
    from the file.  Auto-vacuum does not defragment the database nor
    repack individual database pages the way that the
    VACUUM command does.  In fact, because
    it moves pages around within the file, auto-vacuum can actually
    make fragmentation worse.

    Auto-vacuuming is only possible if the database stores some
    additional information that allows each database page to be
    traced backwards to its referrer.  Therefore, auto-vacuuming must
    be turned on before any tables are created.  It is not possible
    to enable or disable auto-vacuum after a table has been created.

    When the value of auto-vacuum is 2 or "incremental" then the additional
    information needed to do auto-vacuuming is stored in the database file
    but auto-vacuuming does not occur automatically at each commit as it
    does with auto_vacuum=full.  In incremental mode, the separate
    incremental_vacuum pragma must
    be invoked to cause the auto-vacuum to occur.

    The database connection can be changed between full and incremental
    autovacuum mode at any time.  However, changing from
    "none" to "full" or "incremental" can only occur when the database 
    is new (no tables
    have yet been created) or by running the VACUUM command.  To
    change auto-vacuum modes, first use the auto_vacuum pragma to set
    the new desired mode, then invoke the VACUUM command to 
    reorganize the entire database file.  To change from "full" or
    "incremental" back to "none" always requires running VACUUM even
    on an empty database.
    

    When the auto_vacuum pragma is invoked with no arguments, it
    returns the current auto_vacuum mode.


c2: Pragma statements supported by SQLite
c3: PRAGMA automatic_index
c5: pragma.html#pragma_automatic_index
c4:
PRAGMA automatic_index;
     PRAGMA automatic_index = boolean;

    Query, set, or clear the automatic indexing capability.
    Automatic indexing is enabled by default as of 
    version 3.7.17 (2013-05-20),
    but this might change in future releases of SQLite.


c2: Pragma statements supported by SQLite
c3: PRAGMA busy_timeout
c5: pragma.html#pragma_busy_timeout
c4:
PRAGMA busy_timeout;
         PRAGMA busy_timeout = milliseconds;
    Query or change the setting of the
    busy timeout.
    This pragma is an alternative to the sqlite3_busy_timeout() C-language
    interface which is made available as a pragma for use with language
    bindings that do not provide direct access to sqlite3_busy_timeout().

    Each database connection can only have a single
    busy handler.  This PRAGMA sets the busy handler
    for the process, possibly overwriting any previously set busy handler.


c2: Pragma statements supported by SQLite
c3: PRAGMA cache_size
c5: pragma.html#pragma_cache_size
c4:
PRAGMA schema.cache_size;
       PRAGMA schema.cache_size = pages;
       PRAGMA schema.cache_size = -kibibytes;
    Query or change the suggested maximum number of database disk pages
    that SQLite will hold in memory at once per open database file.  Whether
    or not this suggestion is honored is at the discretion of the
    Application Defined Page Cache.
    The default page cache that is built into SQLite honors the request,
    however alternative application-defined page cache implementations
    may choose to interpret the suggested cache size in different ways
    or to ignore it all together.
    The default suggested cache size is -2000, which means the cache size
    is limited to 2048000 bytes of memory.
    The default suggested cache size can be altered using the
    SQLITE_DEFAULT_CACHE_SIZE compile-time options.
    The TEMP database has a default suggested cache size of 0 pages.

    If the argument N is positive then the suggested cache size is set 
    to N. If the argument N is negative, then the
    number of cache pages is adjusted to be a number of pages that would
    use approximately abs(N*1024) bytes of memory based on the current
    page size.  SQLite remembers the number of pages in the page cache,
    not the amount of memory used.  So if you set the cache size using
    a negative number and subsequently change the page size (using the
    PRAGMA page_size command) then the maximum amount of cache
    memory will go up or down in proportion to the change in page size.

    Backwards compatibility note:
    The behavior of cache_size with a negative N
    was different prior to version 3.7.10 (2012-01-16).  In
    earlier versions, the number of pages in the cache was set
    to the absolute value of N.

    When you change the cache size using the cache_size pragma, the
    change only endures for the current session.  The cache size reverts
    to the default value when the database is closed and reopened.

    The default page cache implemention does not allocate
    the full amount of cache memory all at once.  Cache memory
    is allocated in smaller chunks on an as-needed basis.  The page_cache
    setting is a (suggested) upper bound on the amount of memory that the
    cache can use, not the amount of memory it will use all of the time.
    This is the behavior of the default page cache implementation, but an
    application defined page cache is free
    to behave differently if it wants.


c2: Pragma statements supported by SQLite
c3: PRAGMA cache_spill
c5: pragma.html#pragma_cache_spill
c4:
PRAGMA cache_spill;
         PRAGMA cache_spill=boolean;
         PRAGMA schema.cache_spill=N;

    The cache_spill pragma enables or disables the ability of the pager
    to spill dirty cache pages to the database file in the middle of a 
    transaction.  Cache_spill is enabled by default and most applications
    should leave it that way as cache spilling is usually advantageous.
    However, a cache spill has the side-effect of acquiring an
    EXCLUSIVE lock on the database file.  Hence, some applications that
    have large long-running transactions may want to disable cache spilling
    in order to prevent the application from acquiring an exclusive lock
    on the database until the moment that the transaction COMMITs.
    The "PRAGMA cache_spill=N" form of this pragma sets a minimum
    cache size threshold required for spilling to occur. The number of pages
    in cache must exceed both the cache_spill threshold and the maximum cache
    size set by the PRAGMA cache_size statement in order for spilling to
    occur.
    The "PRAGMA cache_spill=boolean" form of this pragma applies
    across all databases attached to the database connection.  But the
    "PRAGMA cache_spill=N" form of this statement only applies to
    the "main" schema or whatever other schema is specified as part of the
    statement.


c2: Pragma statements supported by SQLite
c3: PRAGMA case_sensitive_like
c5: pragma.html#pragma_case_sensitive_like
c4:
PRAGMA case_sensitive_like = boolean;
    The default behavior of the LIKE operator is to ignore case
    for ASCII characters. Hence, by default 'a' LIKE 'A' is
    true.  The case_sensitive_like pragma installs a new application-defined
    LIKE function that is either case sensitive or insensitive depending
    on the value of the case_sensitive_like pragma.
    When case_sensitive_like is disabled, the default LIKE behavior is
    expressed.  When case_sensitive_like is enabled, case becomes
    significant.  So, for example,
    'a' LIKE 'A' is false but 'a' LIKE 'a' is still true.

    This pragma uses sqlite3_create_function() to overload the
    LIKE and GLOB functions, which may override previous implementations
    of LIKE and GLOB registered by the application.  This pragma
    only changes the behavior of the SQL LIKE operator.  It does not
    change the behavior of the sqlite3_strlike() C-language interface,
    which is always case insensitive.

    WARNING: If a database uses the LIKE operator anywhere in
    the schema, such as in a CHECK constraint or in an
    expression index or in the WHERE clause of a partial index, then
    changing the definition of the LIKE operator using this PRAGMA can
    cause the database to appear to be corrupt.  PRAGMA integrity_check
    will report errors.  The database is not really corrupt in that
    changing the behavior of LIKE back to the way
    it was when the schema was defined and the database was populated
    will clear the problem.   If the use of LIKE occurs only in indexes,
    then the problem can be cleared by running REINDEX.  Nevertheless, 
    the use of the case_sensitive_like pragma is discouraged.

    
    
    This pragma is deprecated and exists
    for backwards compatibility only.  New applications
    should avoid using this pragma.  Older applications should discontinue
    use of this pragma at the earliest opportunity.  This pragma may be omitted
    from the build when SQLite is compiled using SQLITE_OMIT_DEPRECATED.


c2: Pragma statements supported by SQLite
c3: PRAGMA cell_size_check
c5: pragma.html#pragma_cell_size_check
c4:
PRAGMA cell_size_check
       PRAGMA cell_size_check = boolean;
    The cell_size_check pragma enables or disables additional sanity
    checking on database b-tree pages as they are initially read from disk.
    With cell size checking enabled, database corruption is detected earlier
    and is less likely to "spread".  However, there is a small performance
    hit for doing the extra checks and so cell size checking is turned off
    by default.


c2: Pragma statements supported by SQLite
c3: PRAGMA checkpoint_fullfsync
c5: pragma.html#pragma_checkpoint_fullfsync
c4:
PRAGMA checkpoint_fullfsync
       PRAGMA checkpoint_fullfsync = boolean;
    Query or change the fullfsync flag for checkpoint operations.
    If this flag is set, then the F_FULLFSYNC syncing method is used
    during checkpoint operations on systems that support F_FULLFSYNC. 
    The default value of the checkpoint_fullfsync flag
    is off.  Only Mac OS-X supports F_FULLFSYNC.

    If the fullfsync flag is set, then the F_FULLFSYNC syncing
    method is used for all sync operations and the checkpoint_fullfsync
    setting is irrelevant.


c2: Pragma statements supported by SQLite
c3: PRAGMA collation_list
c5: pragma.html#pragma_collation_list
c4:
PRAGMA collation_list;
    Return a list of the collating sequences defined for the current
    database connection.


c2: Pragma statements supported by SQLite
c3: PRAGMA compile_options
c5: pragma.html#pragma_compile_options
c4:
PRAGMA compile_options;
    This pragma returns the names of compile-time options used when
    building SQLite, one option per row.  The "SQLITE_" prefix is omitted
    from the returned option names.  See also the
    sqlite3_compileoption_get() C/C++ interface and the
    sqlite_compileoption_get() SQL functions.


c2: Pragma statements supported by SQLite
c3: PRAGMA count_changes
c5: pragma.html#pragma_count_changes
c4:
PRAGMA count_changes;
       PRAGMA count_changes = boolean;

    Query or change the count-changes flag. Normally, when the
    count-changes flag is not set, INSERT, UPDATE and DELETE statements
    return no data. When count-changes is set, each of these commands 
    returns a single row of data consisting of one integer value - the
    number of rows inserted, modified or deleted by the command. The 
    returned change count does not include any insertions, modifications
    or deletions performed by triggers, any changes made automatically
    by foreign key actions, or updates caused by an upsert.

    Another way to get the row change counts is to use the
    sqlite3_changes() or sqlite3_total_changes() interfaces.
    There is a subtle different, though.  When an INSERT, UPDATE, or
    DELETE is run against a view using an INSTEAD OF trigger,
    the count_changes pragma reports the number of rows in the view
    that fired the trigger, whereas sqlite3_changes() and
    sqlite3_total_changes() do not.

    
    
    This pragma is deprecated and exists
    for backwards compatibility only.  New applications
    should avoid using this pragma.  Older applications should discontinue
    use of this pragma at the earliest opportunity.  This pragma may be omitted
    from the build when SQLite is compiled using SQLITE_OMIT_DEPRECATED.


c2: Pragma statements supported by SQLite
c3: PRAGMA data_store_directory
c5: pragma.html#pragma_data_store_directory
c4:
PRAGMA data_store_directory;
      PRAGMA data_store_directory = 'directory-name';
    Query or change the value of the sqlite3_data_directory global
    variable, which windows operating-system interface backends use to
    determine where to store database files specified using a relative
    pathname.

    Changing the data_store_directory setting is not threadsafe.
    Never change the data_store_directory setting if another thread
    within the application is running any SQLite interface at the same time.
    Doing so results in undefined behavior.  Changing the data_store_directory
    setting writes to the sqlite3_data_directory global
    variable and that global variable is not protected by a mutex.

    This facility is provided for WinRT which does not have an OS
    mechanism for reading or changing the current working directory.
    The use of this pragma in any other context is discouraged and may
    be disallowed in future releases.

    
    
    This pragma is deprecated and exists
    for backwards compatibility only.  New applications
    should avoid using this pragma.  Older applications should discontinue
    use of this pragma at the earliest opportunity.  This pragma may be omitted
    from the build when SQLite is compiled using SQLITE_OMIT_DEPRECATED.


c2: Pragma statements supported by SQLite
c3: PRAGMA data_version
c5: pragma.html#pragma_data_version
c4:
PRAGMA schema.data_version;
    The "PRAGMA data_version" command provides an indication that the
    database file has been modified.
    Interactive programs that hold database content in memory or that
    display database content on-screen can use the PRAGMA data_version
    command to determine if they need to flush and reload their memory
    or update the screen display.

    The integer values returned by two
    invocations of "PRAGMA data_version" from the same connection
    will be different if changes were committed to the database 
    by any other connection in the interim.
    The "PRAGMA data_version" value is unchanged for commits made
    on the same database connection.
    The behavior of "PRAGMA data_version" is the same for all database
    connections, including database connections in separate processes
    and shared cache database connections.

    The "PRAGMA data_version" value is a local property of each
    database connection and so values returned by two concurrent invocations
    of "PRAGMA data_version" on separate database connections are 
    often different even though the underlying database is identical.
    It is only meaningful to compare the "PRAGMA data_version" values
    returned by the same database connection at two different points in
    time.


c2: Pragma statements supported by SQLite
c3: PRAGMA database_list
c5: pragma.html#pragma_database_list
c4:
PRAGMA database_list;
    This pragma works like a query to return one row for each database
    attached to the current database connection.
    The second column is "main" for the main database file, "temp"
    for the database file used to store TEMP objects, or the name of the
    ATTACHed database for other database files.
    The third column is the name of the database file itself, or an empty
    string if the database is not associated with a file.


c2: Pragma statements supported by SQLite
c3: PRAGMA default_cache_size
c5: pragma.html#pragma_default_cache_size
c4:
PRAGMA schema.default_cache_size;
       PRAGMA schema.default_cache_size
            = Number-of-pages;

    This pragma queries or sets the suggested maximum number of pages
    of disk cache that will be allocated per open database file.
    The difference between this pragma and cache_size is that the
    value set here persists across database connections.
    The value of the default cache size is stored in the 4-byte
    big-endian integer located at offset 48 in the header of the
    database file.
    

    
    
    This pragma is deprecated and exists
    for backwards compatibility only.  New applications
    should avoid using this pragma.  Older applications should discontinue
    use of this pragma at the earliest opportunity.  This pragma may be omitted
    from the build when SQLite is compiled using SQLITE_OMIT_DEPRECATED.


c2: Pragma statements supported by SQLite
c3: PRAGMA defer_foreign_keys
c5: pragma.html#pragma_defer_foreign_keys
c4:
PRAGMA defer_foreign_keys
       PRAGMA defer_foreign_keys = boolean;
    When the defer_foreign_keys PRAGMA is on,
    enforcement of all foreign key constraints is delayed until the
    outermost transaction is committed.  The defer_foreign_keys pragma
    defaults to OFF so that foreign key constraints are only deferred if
    they are created as "DEFERRABLE INITIALLY DEFERRED".  The 
    defer_foreign_keys pragma is automatically switched off at each
    COMMIT or ROLLBACK.  Hence, the defer_foreign_keys pragma must be
    separately enabled for each transaction.  This pragma is
    only meaningful if foreign key constraints are enabled, of course.

    The sqlite3_db_status(db,SQLITE_DBSTATUS_DEFERRED_FKS,...)
    C-language interface can be used during a transaction to determine 
    if there are deferred and unresolved foreign key constraints.


c2: Pragma statements supported by SQLite
c3: PRAGMA empty_result_callbacks
c5: pragma.html#pragma_empty_result_callbacks
c4:
PRAGMA empty_result_callbacks;
       PRAGMA empty_result_callbacks = boolean;

    Query or change the empty-result-callbacks flag.

    The empty-result-callbacks flag affects the sqlite3_exec() API only.
    Normally, when the empty-result-callbacks flag is cleared, the
    callback function supplied to the sqlite3_exec() is not invoked
    for commands that return zero rows of data.  When empty-result-callbacks
    is set in this situation, the callback function is invoked exactly once,
    with the third parameter set to 0 (NULL). This is to enable programs  
    that use the sqlite3_exec() API to retrieve column-names even when
    a query returns no data.

    
    
    This pragma is deprecated and exists
    for backwards compatibility only.  New applications
    should avoid using this pragma.  Older applications should discontinue
    use of this pragma at the earliest opportunity.  This pragma may be omitted
    from the build when SQLite is compiled using SQLITE_OMIT_DEPRECATED.


c2: Pragma statements supported by SQLite
c3: PRAGMA encoding
c5: pragma.html#pragma_encoding
c4:
PRAGMA encoding;
       PRAGMA encoding = 'UTF-8';
       PRAGMA encoding = 'UTF-16';
       PRAGMA encoding = 'UTF-16le';
       PRAGMA encoding = 'UTF-16be';
    In first form, if the main database has already been
    created, then this pragma returns the text encoding used by the
    main database, one of 'UTF-8', 'UTF-16le' (little-endian UTF-16
    encoding) or 'UTF-16be' (big-endian UTF-16 encoding).  If the main
    database has not already been created, then the value returned is the
    text encoding that will be used to create the main database, if 
    it is created by this session.

    The second through fifth forms of this pragma
    set the encoding that the main database will be created with if
    it is created by this session. The string 'UTF-16' is interpreted
    as "UTF-16 encoding using native machine byte-ordering".  It is not
    possible to change the text encoding of a database after it has been
    created and any attempt to do so will be silently ignored.

    If no encoding is first set with this pragma,
    then the encoding with which the main database will be created
    defaults to one determined by the
    API used to open the connection.

    Once an encoding has been set for a database, it cannot be changed.

    Databases created by the ATTACH command always use the same encoding
    as the main database.  An attempt to ATTACH a database with a different
    text encoding from the "main" database will fail.


c2: Pragma statements supported by SQLite
c3: PRAGMA foreign_key_check
c5: pragma.html#pragma_foreign_key_check
c4:
PRAGMA schema.foreign_key_check;
        PRAGMA schema.foreign_key_check(table-name);

    The foreign_key_check pragma checks the database, or the table
    called "table-name", for 
    foreign key constraints that are violated.  The foreign_key_check
    pragma returns one row output for each foreign key violation.
    There are four columns in each result row.
    The first column is the name of the table that contains the REFERENCES
    clause.  The second column is the rowid of the row that
    contains the invalid REFERENCES clause, or NULL if the child table is a
    WITHOUT ROWID table.  The third column is the name
    of the table that is referred to. The fourth column is the index of
    the specific foreign key constraint that failed.  The fourth column
    in the output of the foreign_key_check pragma is the same integer as
    the first column in the output of the foreign_key_list pragma.
    When a "table-name" is specified, the only foreign key constraints
    checked are those created by REFERENCES clauses in the
    CREATE TABLE statement for table-name.


c2: Pragma statements supported by SQLite
c3: PRAGMA foreign_key_list
c5: pragma.html#pragma_foreign_key_list
c4:
PRAGMA foreign_key_list(table-name);

    This pragma returns one row for each foreign key constraint
    created by a REFERENCES clause in the CREATE TABLE statement of
    table "table-name".


c2: Pragma statements supported by SQLite
c3: PRAGMA foreign_keys
c5: pragma.html#pragma_foreign_keys
c4:
PRAGMA foreign_keys;
       PRAGMA foreign_keys = boolean;
    Query, set, or clear the enforcement of foreign key constraints.

    This pragma is a no-op within a transaction; foreign key constraint
       enforcement may only be enabled or disabled when there is no pending
       BEGIN or SAVEPOINT.

    Changing the foreign_keys setting affects the execution of
       all statements prepared
       using the database connection, including those prepared before the
       setting was changed. Any existing statements prepared using the legacy 
       sqlite3_prepare() interface may fail with an SQLITE_SCHEMA error
       after the foreign_keys setting is changed.

    As of SQLite version 3.6.19, the default setting for foreign
       key enforcement is OFF.  However, that might change in a future
       release of SQLite.  The default setting for foreign key enforcement
       can be specified at compile-time using the SQLITE_DEFAULT_FOREIGN_KEYS
       preprocessor macro.  To minimize future problems, applications should
       set the foreign key enforcement flag as required by the application
       and not depend on the default setting.


c2: Pragma statements supported by SQLite
c3: PRAGMA full_column_names
c5: pragma.html#pragma_full_column_names
c4:
PRAGMA full_column_names;
       PRAGMA full_column_names = boolean;

    Query or change the full_column_names flag. This flag together 
    with the short_column_names flag determine
    the way SQLite assigns names to result columns of SELECT statements.
    Result columns are named by applying the following rules in order:
    
    If there is an AS clause on the result, then the name of
        the column is the right-hand side of the AS clause.
    If the result is a general expression, not a just the name of
        a source table column,
        then the name of the result is a copy of the expression text.
    If the short_column_names pragma is ON, then the name of the
        result is the name of the source table column without the 
        source table name prefix:  COLUMN.
    If both pragmas short_column_names and full_column_names
        are OFF then case (2) applies.
        
    The name of the result column is a combination of the source table
        and source column name:  TABLE.COLUMN
    

    
    
    This pragma is deprecated and exists
    for backwards compatibility only.  New applications
    should avoid using this pragma.  Older applications should discontinue
    use of this pragma at the earliest opportunity.  This pragma may be omitted
    from the build when SQLite is compiled using SQLITE_OMIT_DEPRECATED.


c2: Pragma statements supported by SQLite
c3: PRAGMA fullfsync
c5: pragma.html#pragma_fullfsync
c4:
PRAGMA fullfsync
       PRAGMA fullfsync = boolean;
    Query or change the fullfsync flag. This flag
    determines whether or not the F_FULLFSYNC syncing method is used
    on systems that support it.  The default value of the fullfsync flag
    is off.  Only Mac OS X supports F_FULLFSYNC.

    See also checkpoint_fullfsync.


c2: Pragma statements supported by SQLite
c3: PRAGMA function_list
c5: pragma.html#pragma_function_list
c4:
PRAGMA function_list;
    This pragma returns a list of SQL functions
    known to the database connection.  Each row of the result
    describes a single calling signature for a single SQL function.
    Some SQL functions will have multiple rows in the result set
    if they can (for example) be invoked with a varying number of
    arguments or can accept text in various encodings.


c2: Pragma statements supported by SQLite
c3: PRAGMA hard_heap_limit
c5: pragma.html#pragma_hard_heap_limit
c4:
PRAGMA hard_heap_limit
          PRAGMA hard_heap_limit=N

    This pragma invokes the sqlite3_hard_heap_limit64() interface with
    the argument N, if N is specified and N is a positive integer that
    is less than the current hard heap limit.
    The hard_heap_limit pragma always returns the same integer
    that would be returned by the sqlite3_hard_heap_limit64(-1) C-language
    function.  That is to say, it always returns the value of the hard
    heap limit that is set after any changes imposed by this PRAGMA.
    

    This pragma can only lower the heap limit, never raise it.
    The C-language interface sqlite3_hard_heap_limit64() must be used
    to raise the heap limit.

    See also the soft_heap_limit pragma.


c2: Pragma statements supported by SQLite
c3: PRAGMA ignore_check_constraints
c5: pragma.html#pragma_ignore_check_constraints
c4:
PRAGMA ignore_check_constraints  = boolean;

    This pragma enables or disables the enforcement of CHECK constraints.
    The default setting is off, meaning that CHECK constraints are
    enforced by default.


c2: Pragma statements supported by SQLite
c3: PRAGMA incremental_vacuum
c5: pragma.html#pragma_incremental_vacuum
c4:
PRAGMA schema.incremental_vacuum(N);
      PRAGMA schema.incremental_vacuum;
    The incremental_vacuum pragma causes up to N pages to
    be removed from the freelist.  The database file is truncated by
    the same amount.  The incremental_vacuum pragma has no effect if
    the database is not in
    auto_vacuum=incremental mode
    or if there are no pages on the freelist.  If there are fewer than
    N pages on the freelist, or if N is less than 1, or
    if the "(N)" argument is omitted, then the entire
    freelist is cleared.


c2: Pragma statements supported by SQLite
c3: PRAGMA index_info
c5: pragma.html#pragma_index_info
c4:
PRAGMA schema.index_info(index-name);
    This pragma returns one row for each key column in the named index.
    A key column is a column that is actually named in the CREATE INDEX
    index statement or UNIQUE constraint or PRIMARY KEY constraint that
    created the index.  Index entries also usually contain auxiliary
    columns that point back to the table row being indexed.  The auxiliary
    index-columns are not shown by the index_info pragma, but they are
    listed by the index_xinfo pragma.

    Output columns from the index_info pragma are as follows:
    
    The rank of the column within the index.  (0 means left-most.)
    The rank of the column within the table being indexed.
        A value of -1 means rowid and a value of -2 means that an
        expression is being used.
    The name of the column being indexed.  This columns is NULL
        if the column is the rowid or an expression.
    

    If there is no index named index-name but there is a
    WITHOUT ROWID table with that name, then (as of
    SQLite version 3.30.0 on 2019-10-04) this pragma returns the
    PRIMARY KEY columns of the WITHOUT ROWID table as they are used
    in the records of the underlying b-tree, which is to say with
    duplicate columns removed.


c2: Pragma statements supported by SQLite
c3: PRAGMA index_list
c5: pragma.html#pragma_index_list
c4:
PRAGMA schema.index_list(table-name);
    This pragma returns one row for each index associated with the
    given table.
    Output columns from the index_list pragma are as follows:
    
    A sequence number assigned to each index for internal tracking
          purposes.
    The name of the index.
    "1" if the index is UNIQUE and "0" if not.
    "c" if the index was created by a CREATE INDEX statement,
          "u" if the index was created by a UNIQUE constraint, or
          "pk" if the index was created by a PRIMARY KEY constraint.
    "1" if the index is a partial index and "0" if not.


c2: Pragma statements supported by SQLite
c3: PRAGMA index_xinfo
c5: pragma.html#pragma_index_xinfo
c4:
PRAGMA schema.index_xinfo(index-name);
    This pragma returns information about every column in an index.
    Unlike this index_info pragma, this pragma returns information about
    every column in the index, not just the key columns.
    (A key column is a column that is actually named in the CREATE INDEX
    index statement or UNIQUE constraint or PRIMARY KEY constraint that
    created the index.  Auxiliary columns are additional columns needed to
    locate the table entry that corresponds to each index entry.)

    Output columns from the index_xinfo pragma are as follows:
    
    The rank of the column within the index. (0 means left-most.
         Key columns come before auxiliary columns.)
    The rank of the column within the table being indexed, or -1 if
         the index-column is the rowid of the table being indexed and -2
         if the index is on an expression.
    The name of the column being indexed, or NULL if the index-column
         is the rowid of the table being indexed or an
         expression.
    1 if the index-column is sorted in reverse (DESC) order by the
         index and 0 otherwise.
    The name for the collating sequence
         used to compare values in the index-column.
    1 if the index-column is a key column and 0 if the index-column
         is an auxiliary column.
    

    If there is no index named index-name but there is a
    WITHOUT ROWID table with that name, then (as of
    SQLite version 3.30.0 on 2019-10-04) this pragma returns the
    columns of the WITHOUT ROWID table as they are used
    in the records of the underlying b-tree, which is to say with
    de-duplicated PRIMARY KEY columns first followed by data columns.


c2: Pragma statements supported by SQLite
c3: PRAGMA integrity_check
c5: pragma.html#pragma_integrity_check
c4:
PRAGMA schema.integrity_check;
      PRAGMA schema.integrity_check(N)
      PRAGMA schema.integrity_check(TABLENAME)
    This pragma does a low-level formatting and consistency check
    of the database.  The integrity_check pragma look for:
    
     Table or index entries that are out of sequence
     Misformatted records
     Missing pages
     Missing or surplus index entries
     UNIQUE, CHECK, and NOT NULL constraint errors
     Integrity of the freelist
     Sections of the database that are used more than once, or not at all
    
    If the integrity_check pragma finds problems, strings are returned
    (as multiple rows with a single column per row) which describe
    the problems.  Pragma integrity_check will return at most N
    errors before the analysis quits, with N defaulting
    to 100.  If pragma integrity_check finds no errors, a
    single row with the value 'ok' is returned.

    The usual case is that the entire database file is checked.  However,
    if the argument is TABLENAME, then checking is only performed for the
    the table named and its associated indexes.
    This is called a "partial integrity check".  Because only a subset of the
    database is checked, errors such as unused sections of the file or duplication
    use of the same section of the file by two or more tables cannot be detected.
    The freelist is only verified on a
    partial integrity check if TABLENAME is sqlite_schema or one of its
    aliases.  Support for partial integrity checks was added with
    version 3.33.0 (2020-08-14).

    PRAGMA integrity_check does not find
    FOREIGN KEY errors.  
    Use the PRAGMA foreign_key_check command to find errors in
    FOREIGN KEY constraints.

    See also the PRAGMA quick_check command which does most of the
    checking of PRAGMA integrity_check but runs much faster.


c2: Pragma statements supported by SQLite
c3: PRAGMA journal_mode
c5: pragma.html#pragma_journal_mode
c4:
PRAGMA schema.journal_mode;
        PRAGMA schema.journal_mode
              = DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF

    This pragma queries or sets the journal mode for databases
    associated with the current database connection.

    The first form of this pragma queries the current journaling
    mode for database.  When database is omitted, the
    "main" database is queried.

    The second form changes the journaling mode for "database"
    or for all attached databases if "database" is omitted.
    The new journal mode is returned.  If the journal mode
    could not be changed, the original journal mode is returned.

    The DELETE journaling mode is the normal behavior.  In the DELETE
    mode, the rollback journal is deleted at the conclusion of each
    transaction.  Indeed, the delete operation is the action that causes
    the transaction to commit.
    (See the document titled 
    Atomic Commit In SQLite for additional detail.)

    The TRUNCATE journaling mode commits transactions by truncating
    the rollback journal to zero-length instead of deleting it.  On many
    systems, truncating a file is much faster than deleting the file since
    the containing directory does not need to be changed.

    The PERSIST journaling mode prevents the rollback journal from
    being deleted at the end of each transaction.  Instead, the header
    of the journal is overwritten with zeros.  This will prevent other
    database connections from rolling the journal back.  The PERSIST
    journaling mode is useful as an optimization on platforms where
    deleting or truncating a file is much more expensive than overwriting
    the first block of a file with zeros.  See also:
    PRAGMA journal_size_limit and SQLITE_DEFAULT_JOURNAL_SIZE_LIMIT.

    The MEMORY journaling mode stores the rollback journal in 
    volatile RAM.  This saves disk I/O but at the expense of database
    safety and integrity.  If the application using SQLite crashes in
    the middle of a transaction when the MEMORY journaling mode is set,
    then the database file will very likely 
    go corrupt.

    The WAL journaling mode uses a write-ahead log instead of a
    rollback journal to implement transactions.  The WAL journaling mode
    is persistent; after being set it stays in effect
    across multiple database connections and after closing and
    reopening the database.  A database in WAL journaling mode
    can only be accessed by SQLite version 3.7.0 (2010-07-21)
    or later.

    The OFF journaling mode disables the rollback journal completely.
    No rollback journal is ever created and hence there is never a rollback
    journal to delete.  The OFF journaling mode disables the atomic
    commit and rollback capabilities of SQLite. The ROLLBACK command
    no longer works; it behaves in an undefined way.  Applications must
    avoid using the ROLLBACK command when the journal mode is OFF.
    If the application crashes
    in the middle of a transaction when the OFF journaling mode is
    set, then the database file will very likely
    go corrupt. Without a journal, there is no way for
    a statement to unwind partially completed operations following
    a constraint error.  This might also leave the database in a corrupted
    state.  For example, if a duplicate entry causes a 
    CREATE UNIQUE INDEX statement to fail half-way through,
    it will leave behind a partially created, and hence corrupt, index.
    Because OFF journaling
    mode allows the database file to be corrupted using ordinary SQL,
    it is disabled when SQLITE_DBCONFIG_DEFENSIVE is enabled.

    Note that the journal_mode for an in-memory database
    is either MEMORY or OFF and can not be changed to a different value.
    An attempt to change the journal_mode of an in-memory database to
    any setting other than MEMORY or OFF is ignored.  Note also that
    the journal_mode cannot be changed while a transaction is active.


c2: Pragma statements supported by SQLite
c3: PRAGMA journal_size_limit
c5: pragma.html#pragma_journal_size_limit
c4:
PRAGMA schema.journal_size_limit
    PRAGMA schema.journal_size_limit = N ;

  If a database connection is operating in
  exclusive locking mode or in
  persistent journal mode 
  (PRAGMA journal_mode=persist) then
  after committing a transaction the rollback journal file may remain in
  the file-system. This increases performance for subsequent transactions
  since overwriting an existing file is faster than append to a file,
  but it also consumes
  file-system space. After a large transaction (e.g. a VACUUM),
  the rollback journal file may consume a very large amount of space.

  Similarly, in WAL mode, the write-ahead log file is not truncated
  following a checkpoint.  Instead, SQLite reuses the existing file
  for subsequent WAL entries since overwriting is faster than appending.

  The journal_size_limit pragma may be used to limit the size of 
  rollback-journal and WAL files left
  in the file-system after transactions or checkpoints.
  Each time a transaction is committed or a WAL file resets, SQLite 
  compares the size of the rollback journal file or WAL file left in 
  the file-system to the size limit
  set by this pragma and if the journal or WAL file is larger 
  it is truncated to the limit.

  The second form of the pragma listed above is used to set a new limit
  in bytes for the specified database.  A negative number implies no limit.
  To always truncate rollback journals and WAL files to their minimum size, 
  set the journal_size_limit to zero.
  Both the first and second forms of the pragma listed above return a single
  result row containing a single integer column - the value of the journal
  size limit in bytes. The default journal size limit is -1 (no limit).  The
  SQLITE_DEFAULT_JOURNAL_SIZE_LIMIT preprocessor macro can be used to change
  the default journal size limit at compile-time.

  This pragma only operates on the single database specified prior
  to the pragma name (or on the "main" database if no database is specified.)
  There is no way to change the journal size limit on all attached databases
  using a single PRAGMA statement.  The size limit must be set separately for
  each attached database.


c2: Pragma statements supported by SQLite
c3: PRAGMA legacy_alter_table
c5: pragma.html#pragma_legacy_alter_table
c4:
PRAGMA legacy_alter_table;
       PRAGMA legacy_alter_table = boolean
    This pragma sets or queries the value of the legacy_alter_table
    flag.  When this flag is on, the ALTER TABLE RENAME
    command (for changing the name of a table) works as it did
    in SQLite 3.24.0 (2018-06-04) and earlier.  More specifically,
    when this flag is on
    the ALTER TABLE RENAME command only rewrites the initial occurrence
    of the table name in its CREATE TABLE statement and in any associated
    CREATE INDEX and CREATE TRIGGER statements.  Other references to the
    table are unmodified, including:
    
     References to the table within the bodies of triggers and views.
     References to the table within CHECK constraints in the original
         CREATE TABLE statement.
     References to the table within the WHERE clauses of partial indexes.
    
    The default setting for this pragma is OFF, which means that all
    references to the table anywhere in the schema are converted to the new name.
    This pragma is provided as a work-around for older programs that
    contain code that expect the incomplete behavior
    of ALTER TABLE RENAME found in older versions of SQLite.
    New applications should leave this flag turned off.
    For compatibility with older virtual table implementations,
    this flag is turned on temporarily while the sqlite3_module.xRename
    method is being run.  The value of this flag is restored after the 
    sqlite3_module.xRename method finishes.
    The legacy alter table behavior can also be toggled on and off
    using the SQLITE_DBCONFIG_LEGACY_ALTER_TABLE option to the
    sqlite3_db_config() interface.
    The legacy alter table behavior is a per-connection setting.  Turning
    this features on or off affects all attached database files within the
    database connection.
    The setting does not persist.  Changing this setting in one connection
    does not affect any other connections.


c2: Pragma statements supported by SQLite
c3: PRAGMA legacy_file_format
c5: pragma.html#pragma_legacy_file_format
c4:
PRAGMA legacy_file_format;
    This pragma no longer functions.  It has become a no-op.
    The capabilities formerly provided by PRAGMA legacy_file_format
    are now available using the SQLITE_DBCONFIG_LEGACY_FILE_FORMAT
    option to the sqlite3_db_config() C-language interface.


c2: Pragma statements supported by SQLite
c3: PRAGMA locking_mode
c5: pragma.html#pragma_locking_mode
c4:
PRAGMA schema.locking_mode;
    PRAGMA schema.locking_mode
                = NORMAL | EXCLUSIVE
    This pragma sets or queries the database connection locking-mode. 
    The locking-mode is either NORMAL or EXCLUSIVE.

    In NORMAL locking-mode (the default unless overridden at compile-time
    using SQLITE_DEFAULT_LOCKING_MODE), a database connection
    unlocks the database file at the conclusion of each read or
    write transaction. When the locking-mode is set to EXCLUSIVE, the
    database connection never releases file-locks. The first time the
    database is read in EXCLUSIVE mode, a shared lock is obtained and 
    held. The first time the database is written, an exclusive lock is
    obtained and held.

    Database locks obtained by a connection in EXCLUSIVE mode may be
    released either by closing the database connection, or by setting the
    locking-mode back to NORMAL using this pragma and then accessing the
    database file (for read or write). Simply setting the locking-mode to
    NORMAL is not enough - locks are not released until the next time
    the database file is accessed.

    There are three reasons to set the locking-mode to EXCLUSIVE.
    
    The application wants to prevent other processes from
        accessing the database file.
    The number of system calls for filesystem operations is reduced,
        possibly resulting in a small performance increase.
    WAL databases can be accessed in EXCLUSIVE mode without the
        use of shared memory. 
        (Additional information)
    
    

    When the locking_mode pragma specifies a particular database,
    for example:

    
PRAGMA main.locking_mode=EXCLUSIVE;
    

    then the locking mode applies only to the named database.  If no
    database name qualifier precedes the "locking_mode" keyword then
    the locking mode is applied to all databases, including any new
    databases added by subsequent ATTACH commands.

   The "temp" database (in which TEMP tables and indices are stored)
   and in-memory databases
   always uses exclusive locking mode.  The locking mode of temp and
   in-memory databases cannot
   be changed.  All other databases use the normal locking mode by default
   and are affected by this pragma.

   If the locking mode is EXCLUSIVE when first entering
   WAL journal mode, then the locking mode cannot be changed to
   NORMAL until after exiting WAL journal mode. 
   If the locking mode is NORMAL when first entering WAL
   journal mode, then the locking mode can be changed between NORMAL and
   EXCLUSIVE and back again at any time and without needing to exit
   WAL journal mode.


c2: Pragma statements supported by SQLite
c3: PRAGMA max_page_count
c5: pragma.html#pragma_max_page_count
c4:
PRAGMA schema.max_page_count;
        PRAGMA schema.max_page_count = N;
    Query or set the maximum number of pages in the database file.
    Both forms of the pragma return the maximum page count.  The second
    form attempts to modify the maximum page count.  The maximum page
    count cannot be reduced below the current database size.


c2: Pragma statements supported by SQLite
c3: PRAGMA mmap_size
c5: pragma.html#pragma_mmap_size
c4:
PRAGMA schema.mmap_size;
         PRAGMA schema.mmap_size=N

    Query or change the maximum number of bytes that are set
       aside for memory-mapped I/O on a single database.  The first form
       (without an argument) queries the current limit.  The second
       form (with a numeric argument) sets the limit for the specified
       database, or for all databases if the optional database name is
       omitted.  In the second form, if the database name is omitted, the
       limit that is set becomes the default limit for all databases that
       are added to the database connection by subsequent ATTACH
       statements.

    The argument N is the maximum number of bytes of the database file
       that will be accessed using memory-mapped I/O.  If N is zero then
       memory mapped I/O is disabled.  If N is negative, then the limit
       reverts to the default value determined by the most recent
       sqlite3_config(SQLITE_CONFIG_MMAP_SIZE), or to the compile
       time default determined by SQLITE_DEFAULT_MMAP_SIZE if not
       start-time limit has been set.

    The PRAGMA mmap_size statement will never increase the amount
       of address space used for memory-mapped I/O above the
       hard limit set by the SQLITE_MAX_MMAP_SIZE compile-time option,
       nor the hard limit set at startup-time by the second argument to
       sqlite3_config(SQLITE_CONFIG_MMAP_SIZE)

    The size of the memory-mapped I/O region cannot be changed while
       the memory-mapped I/O region is in active use, to avoid unmapping
       memory out from under running SQL statements.  For this reason,
       the mmap_size pragma may be a no-op if the prior mmap_size is non-zero
       and there are other SQL statements running concurrently on the same
       database connection.


c2: Pragma statements supported by SQLite
c3: PRAGMA module_list
c5: pragma.html#pragma_module_list
c4:
PRAGMA module_list;
    This pragma returns a list of 
    virtual table modules registered with the database connection.


c2: Pragma statements supported by SQLite
c3: PRAGMA optimize
c5: pragma.html#pragma_optimize
c4:
PRAGMA optimize;
       PRAGMA optimize(MASK);
       PRAGMA schema.optimize;
       PRAGMA schema.optimize(MASK);
  Attempt to optimize the database.  All schemas are optimized in the 
  first two forms, and only the specified schema is optimized in the latter
  two.
  To achieve the best long-term query performance without the need to
  do a detailed engineering analysis of the application schema and SQL,
  it is recommended that applications run "PRAGMA optimize" (with no arguments)
  just before closing each database connection.  Long-running applications
  might also benefit from setting a timer to run "PRAGMA optimize" every
  few hours.
  
  This pragma is usually a no-op or nearly so and is very fast.
  However if SQLite feels
  that performing database optimizations (such as running ANALYZE
  or creating new indexes) will improve the performance of future queries, then
  some database I/O may be done.  Applications that want to limit the amount
  of work performed can set a timer that will invoke
  sqlite3_interrupt() if the pragma goes on for too long.
  Or, since SQLite 3.32.0, the application can use 
  PRAGMA analysis_limit=N for some small
  value of N (a few hundred or a few thousand) to limit the depth
  of analyze.
  
  The details of optimizations performed by this pragma are expected
  to change and improve over time.  Applications should anticipate that
  this pragma will perform new optimizations in future releases.

  The optional MASK argument is a bitmask of optimizations to perform:
  
  
         Debugging mode.  Do not actually perform any optimizations
         but instead return one line of text for each optimization
         that would have been done.  Off by default.
  
         Run ANALYZE on tables that might benefit.  On by default.
         See below for additional information.
  
         (Not yet implemented)
         Record usage and performance 
         information from the current session in the
         database file so that it will be available to "optimize"
         pragmas run by future database connections.
  
         (Not yet implemented)
         Create indexes that might have been helpful to recent queries.
  
  The default MASK is and always shall be 0xfffe.  The 0xfffe mask means
  perform all of the optimizations listed above except Debug Mode.  If new
  optimizations are added in the future that should be off by default, those
  new optimizations will be given a mask of 0x10000 or larger.

  To see all optimizations that would have been done without actually
  doing them, run "PRAGMA optimize(-1)".  To use only the ANALYZE
  optimization, run "PRAGMA optimize(0x02)".
  
  Determination Of When To Run Analyze
   In the current implementation, a table is analyzed if and only if 
      all of the following are true:
  
  
    MASK bit 0x02 is set.
  
    The query planner used sqlite_stat1-style statistics for one or
    more indexes of the table at some point during the lifetime of
    the current connection.
  
    One or more indexes of the table are currently unanalyzed or
    the number of rows in the table has increased by 25 times or more
    since the last time ANALYZE was run.
  
   The rules for when tables are analyzed are likely to change in
      future releases.


c2: Pragma statements supported by SQLite
c3: PRAGMA page_size
c5: pragma.html#pragma_page_size
c4:
PRAGMA schema.page_size;
       PRAGMA schema.page_size = bytes;
    Query or set the page size of the database. The page
    size must be a power of two between 512 and 65536 inclusive.
    

    When a new database is created, SQLite assigns a page size to
    the database based on platform and filesystem.  For many years,
    the default page size was almost always 1024 bytes, but beginning
    with SQLite version 3.12.0 (2016-03-29), 
    the default page size increased to 4096.
    The default page size is recommended for most applications.

    Specifying a new page size does not change the page size
    immediately.  Instead, the new page size is remembered and is used
    to set the page size when the database is first created, if it does
    not already exist when the page_size pragma is issued, or at the
    next VACUUM command that is run on the same database connection
    while not in WAL mode.

    The SQLITE_DEFAULT_PAGE_SIZE compile-time option can be used
    to change the default page size assigned to new databases.


c2: Pragma statements supported by SQLite
c3: PRAGMA parser_trace
c5: pragma.html#pragma_parser_trace
c4:
PRAGMA parser_trace = boolean; 

    If SQLite has been compiled with the SQLITE_DEBUG compile-time
    option, then the parser_trace pragma can be used to turn on tracing
    for the SQL parser used internally by SQLite.
    This feature is used for debugging SQLite itself.

    
    
    This pragma is intended for use when debugging SQLite itself.  It
    is only available when the SQLITE_DEBUG compile-time option
    is used.


c2: Pragma statements supported by SQLite
c3: PRAGMA pragma_list
c5: pragma.html#pragma_pragma_list
c4:
PRAGMA pragma_list;
    This pragma returns a list of PRAGMA commands
    known to the database connection.


c2: Pragma statements supported by SQLite
c3: PRAGMA query_only
c5: pragma.html#pragma_query_only
c4:
PRAGMA query_only;
      PRAGMA query_only = boolean;

    The query_only pragma prevents data changes on database files when
    enabled.  When this pragma is enabled, any attempt to CREATE, DELETE,
    DROP, INSERT, or UPDATE will result in an SQLITE_READONLY error.
    However, the database is not truly read-only.  You can still run
    a checkpoint or a COMMIT and the return value of the
    sqlite3_db_readonly() routine is not affected.


c2: Pragma statements supported by SQLite
c3: PRAGMA quick_check
c5: pragma.html#pragma_quick_check
c4:
PRAGMA schema.quick_check;
      PRAGMA schema.quick_check(N)
      PRAGMA schema.quick_check(TABLENAME)
    The pragma is like integrity_check except that it does not verify
    UNIQUE constraints and does not verify
    that index content matches table content.  By skipping UNIQUE
    and index consistency checks, quick_check is able to run faster.
    PRAGMA quick_check runs in O(N) time whereas PRAGMA integrity_check
    requires O(NlogN) time where N is the total number of rows in the 
    database. Otherwise the two pragmas are the same.


c2: Pragma statements supported by SQLite
c3: PRAGMA read_uncommitted
c5: pragma.html#pragma_read_uncommitted
c4:
PRAGMA read_uncommitted;
       PRAGMA read_uncommitted = boolean;
    Query, set, or clear READ UNCOMMITTED isolation. The default isolation
    level for SQLite is SERIALIZABLE.  Any process or thread can select
    READ UNCOMMITTED isolation, but SERIALIZABLE will still be used except
    between connections that share a common page and schema cache.
    Cache sharing is enabled using the sqlite3_enable_shared_cache() API.
    Cache sharing is disabled by default.
    

    See SQLite Shared-Cache Mode for additional information.


c2: Pragma statements supported by SQLite
c3: PRAGMA recursive_triggers
c5: pragma.html#pragma_recursive_triggers
c4:
PRAGMA recursive_triggers;
       PRAGMA recursive_triggers = boolean;
    Query, set, or clear the recursive trigger capability.

    Changing the recursive_triggers setting affects the execution of
       all statements prepared
       using the database connection, including those prepared before the
       setting was changed. Any existing statements prepared using the legacy 
       sqlite3_prepare() interface may fail with an SQLITE_SCHEMA error
       after the recursive_triggers setting is changed.

    Prior to SQLite version 3.6.18 (2009-09-11), 
    recursive triggers were not supported.
    The behavior of SQLite was always as if this pragma was
    set to OFF.  Support for recursive triggers was added in version 3.6.18
    but was initially turned OFF by default, for compatibility.  Recursive
    triggers may be turned on by default in future versions of SQLite.
    

    The depth of recursion for triggers has a hard upper limit set by
    the SQLITE_MAX_TRIGGER_DEPTH compile-time option and a run-time
    limit set by sqlite3_limit(db,SQLITE_LIMIT_TRIGGER_DEPTH,...).


c2: Pragma statements supported by SQLite
c3: PRAGMA reverse_unordered_selects
c5: pragma.html#pragma_reverse_unordered_selects
c4:
PRAGMA reverse_unordered_selects;
       PRAGMA reverse_unordered_selects = boolean;
    When enabled, this PRAGMA causes many SELECT statements without
    an ORDER BY clause to emit their results in the reverse order from what
    they normally would.  This can help debug applications that are
    making invalid assumptions about the result order.  
    The reverse_unordered_selects pragma works for most SELECT statements,
    however the query planner may sometimes choose an algorithm that is
    not easily reversed, in which case the output will appear in the same
    order regardless of the reverse_unordered_selects setting.
    SQLite makes no
    guarantees about the order of results if a SELECT omits the ORDER BY
    clause.  Even so, the order of results does not change from one
    run to the next, and so many applications mistakenly come to depend
    on the arbitrary output order whatever that order happens to be.  However, 
    sometimes new versions of SQLite will contain optimizer enhancements
    that will cause the output order of queries without ORDER BY clauses
    to shift.  When that happens, applications that depend on a certain
    output order might malfunction.  By running the application multiple
    times with this pragma both disabled and enabled, cases where the
    application makes faulty assumptions about output order can be
    identified and fixed early, reducing problems
    that might be caused by linking against a different version of SQLite.


c2: Pragma statements supported by SQLite
c3: PRAGMA schema_version
c5: pragma.html#pragma_schema_version
c4:
PRAGMA schema.schema_version; 
      PRAGMA schema.schema_version = integer ;

   The schema_version pragma will get or set
       the value of the schema-version integer at offset 40 in the
       database header. 

    SQLite automatically increments the schema-version whenever the
       schema changes. As each SQL statement runs, the schema version is
       checked to ensure that the schema has not changed since the SQL
       statement was prepared.
       Subverting this mechanism by using "PRAGMA schema_version=N"
       to change the value of the schema_version
       may cause SQL statement to run using an obsolete schema,
       which can lead to incorrect answers and/or
       database corruption.
       It is always safe to read the schema_version, but changing the
       schema_version can cause problems.  For this reason, attempts
       to change the value of schema_version are a silent no-op when
       defensive mode is enabled for a
       database connection.

       
    
    Warning:
    Misuse of this pragma can result in database corruption.
    
  

    For the purposes of this pragma, the VACUUM command is considered
       a schema change, since VACUUM will usually alter the "rootpage"
       values for entries in the sqlite_schema table.


    See also the application_id pragma and user_version pragma.


c2: Pragma statements supported by SQLite
c3: PRAGMA secure_delete
c5: pragma.html#pragma_secure_delete
c4:
PRAGMA schema.secure_delete;
     PRAGMA schema.secure_delete = boolean|FAST
    Query or change the secure-delete setting. When secure_delete is
    on, SQLite overwrites deleted content with zeros.  The default
    setting for secure_delete is determined by the SQLITE_SECURE_DELETE
    compile-time option and is normally off.  The off setting for
    secure_delete improves performance by reducing the number of CPU cycles
    and the amount of disk I/O.  Applications that wish to avoid leaving
    forensic traces after content is deleted or updated should enable the
    secure_delete pragma prior to performing the delete or update, or else
    run VACUUM after the delete or update.

    The "fast" setting for secure_delete (added circa 2017-08-01)
    is an intermediate setting in between "on" and "off".
    When secure_delete is set to "fast",
    SQLite will overwrite deleted content with zeros only if doing so
    does not increase the amount of I/O.  In other words, the "fast"
    setting uses more CPU cycles but does not use more I/O.
    This has the effect of purging all old content from b-tree pages,
    but leaving forensic traces on freelist pages.

    
    When there are attached databases and no database
    is specified in the pragma, all databases have their secure-delete
    setting altered.
    The secure-delete setting for newly attached databases is the setting
    of the main database at the time the ATTACH command is evaluated.

    
    When multiple database connections share the same cache, changing
    the secure-delete flag on one database connection changes it for them
    all.
    

    Limitation:
    The secure_delete pragma only causes deleted content to be scrubbed
    from ordinary tables.  If virtual tables store content in
    shadow tables, then deleting content from the virtual table does
    not necessarily remove forensic traces from the shadow tables.
    In particular, the FTS3 and FTS5 virtual tables that come
    bundled with SQLite might leave forensic traces in their shadow tables
    even if the secure_delete pragma is enabled.


c2: Pragma statements supported by SQLite
c3: PRAGMA short_column_names
c5: pragma.html#pragma_short_column_names
c4:
PRAGMA short_column_names;
       PRAGMA short_column_names = boolean;

    Query or change the short-column-names flag. This flag affects
    the way SQLite names columns of data returned by SELECT statements.
    See the full_column_names pragma for full details.
    

    
    
    This pragma is deprecated and exists
    for backwards compatibility only.  New applications
    should avoid using this pragma.  Older applications should discontinue
    use of this pragma at the earliest opportunity.  This pragma may be omitted
    from the build when SQLite is compiled using SQLITE_OMIT_DEPRECATED.


c2: Pragma statements supported by SQLite
c3: PRAGMA shrink_memory
c5: pragma.html#pragma_shrink_memory
c4:
PRAGMA shrink_memory

    This pragma causes the database connection on which it is invoked
    to free up as much memory as it can, by calling
    sqlite3_db_release_memory().


c2: Pragma statements supported by SQLite
c3: PRAGMA soft_heap_limit
c5: pragma.html#pragma_soft_heap_limit
c4:
PRAGMA soft_heap_limit
          PRAGMA soft_heap_limit=N

    This pragma invokes the sqlite3_soft_heap_limit64() interface with
    the argument N, if N is specified and is a non-negative integer.
    The soft_heap_limit pragma always returns the same integer
    that would be returned by the sqlite3_soft_heap_limit64(-1) C-language
    function.
    
    See also the hard_heap_limit pragma.


c2: Pragma statements supported by SQLite
c3: PRAGMA stats
c5: pragma.html#pragma_stats
c4:
PRAGMA stats; 
    This pragma returns auxiliary information about tables and
    indices.  The returned information is used during testing to help
    verify that the query planner is operating correctly.  The format
    and meaning of this pragma will likely change from one release
    to the next. Because of its volatility, the behavior and output
    format of this pragma are deliberately undocumented.

    
    
    The intended use of this pragma is only for testing and validation of
    SQLite.  This pragma is subject to change without notice and is not
    recommended for use by application programs.


c2: Pragma statements supported by SQLite
c3: PRAGMA synchronous
c5: pragma.html#pragma_synchronous
c4:
PRAGMA schema.synchronous;
        PRAGMA schema.synchronous = 
          0 | OFF | 1 | NORMAL | 2 | FULL | 3 | EXTRA;

    Query or change the setting of the "synchronous" flag.
    The first (query) form will return the synchronous setting as an 
    integer.  The second form changes the synchronous setting.
    The meanings of the various synchronous settings are as follows:
    
    EXTRA (3)
    
    EXTRA synchronous is like FULL with the addition that the directory
    containing a rollback journal is synced after that journal is unlinked
    to commit a transaction in DELETE mode.  EXTRA provides additional
    durability if the commit is followed closely by a power loss.
    FULL (2)
    
    When synchronous is FULL (2), the SQLite database engine will
    use the xSync method of the VFS to ensure that all content is safely
    written to the disk surface prior to continuing.
    This ensures that an operating system crash or power failure will
    not corrupt the database.
    FULL synchronous is very safe, but it is also slower.  FULL is the
    most commonly used synchronous setting when not in WAL mode.
    NORMAL (1)
    
    When synchronous is NORMAL (1), the SQLite database
    engine will still sync at the most critical moments, but less often
    than in FULL mode.  There is a very small (though non-zero) chance that
    a power failure at just the wrong time could corrupt the database in
    journal_mode=DELETE on an older filesystem.
    WAL mode is safe from corruption with synchronous=NORMAL, and probably
    DELETE mode is safe too on modern filesystems.  WAL mode is always consistent
    with synchronous=NORMAL, but WAL mode does lose durability.  A transaction
    committed in WAL mode with synchronous=NORMAL might roll back following
    a power loss or system crash.  Transactions are durable across application
    crashes regardless of the synchronous setting or journal mode.
    The synchronous=NORMAL setting is a good choice for most applications
    running in WAL mode.
    OFF (0)
    
    With synchronous OFF (0), SQLite continues without syncing
    as soon as it has handed data off to the operating system.
    If the application running SQLite crashes, the data will be safe, but
    the database might become corrupted if the operating system
    crashes or the computer loses power before that data has been written
    to the disk surface.  On the other hand, commits can be orders of
    magnitude faster with synchronous OFF.
    
    
 
    In WAL mode when synchronous is NORMAL (1), the WAL file is
    synchronized before each checkpoint and the database file is
    synchronized after each completed checkpoint and the WAL file
    header is synchronized when a WAL file begins to be reused after
    a checkpoint, but no sync operations occur during most transactions.
    With synchronous=FULL in WAL mode, an additional
    sync operation of the WAL file happens after each transaction commit.
    The extra WAL sync following each transaction helps ensure that 
    transactions are durable across a power loss.  Transactions are
    consistent with or without the extra syncs provided by
    synchronous=FULL.
    If durability is not a concern, then synchronous=NORMAL is normally
    all one needs in WAL mode.

    The TEMP schema always has synchronous=OFF since the content of
    of TEMP is ephemeral and is not expected to survive a power outage.
    Attempts to change the synchronous setting for TEMP are
    silently ignored.

    See also the fullfsync and checkpoint_fullfsync pragmas.


c2: Pragma statements supported by SQLite
c3: PRAGMA table_info
c5: pragma.html#pragma_table_info
c4:
PRAGMA schema.table_info(table-name);
    This pragma returns one row for each normal column
    in the named table.
    Columns in the result set include: "name" (its name); "type"
    (data type if given, else ''); "notnull" (whether or not the column
    can be NULL); "dflt_value" (the default value for the column);
    and "pk" (either zero for columns that are not part of the primary key,
    or the 1-based index of the column within the primary key).
    The "cid" column should not be taken to mean more than
    "rank within the current result set".
    The table named in the table_info pragma can also be a view.
    This pragma does not show information about generated columns or
    hidden columns.  Use PRAGMA table_xinfo to get a more complete list
    of columns that includes generated and hidden columns.


c2: Pragma statements supported by SQLite
c3: PRAGMA table_list
c5: pragma.html#pragma_table_list
c4:
PRAGMA table_list;
         PRAGMA schema.table_list;
         PRAGMA table_list(table-name);
    This pragma returns information about the tables and views in the schema,
    one table per row of output.  The table_list pragma first appeared
    in SQLite version 3.37.0 (2021-11-27).  As of its initial release
    the columns returned by the table_list pragma include those listed below.
    Future versions of SQLite will probably add additional columns of
    output.
    
    
     schema: the schema in which the table or view appears
    (for example "main" or "temp").
     name: the name of the table or view.
     type: the type of object - one of "table", "view",
           "shadow" (for shadow tables), or "virtual" for
           virtual tables.
     ncol: the number of columns in the table, including
         generated columns and hidden columns.
     wr: 1 if the table is a WITHOUT ROWID table or 0 if is not.
     strict: 1 if the table is a STRICT table or 0 if it is not.
     Additional columns will likely be added in future releases.
    
    
    The default behavior is to show all tables in all schemas.  If the
    schema. name appears before the pragma, then only tables in that
    one schema are shown.  If a table-name argument is supplied, then
    only information about that one table is returned.


c2: Pragma statements supported by SQLite
c3: PRAGMA table_xinfo
c5: pragma.html#pragma_table_xinfo
c4:
PRAGMA schema.table_xinfo(table-name);
    This pragma returns one row for each column in the named table,
    including generated columns and hidden columns.
    The output has the same columns as for PRAGMA table_info plus
    a column, "hidden", whose value signifies a normal column (0),
    a dynamic or stored generated column (2 or 3),
    or a hidden column in a virtual table (1). The rows for which
    this field is non-zero are those omitted for PRAGMA table_info.


c2: Pragma statements supported by SQLite
c3: PRAGMA temp_store
c5: pragma.html#pragma_temp_store
c4:
PRAGMA temp_store;
        PRAGMA temp_store = 
            0 | DEFAULT | 1 | FILE | 2 | MEMORY;

    Query or change the setting of the "temp_store" parameter.
    When temp_store is DEFAULT (0), the compile-time C preprocessor macro
    SQLITE_TEMP_STORE is used to determine where temporary tables and indices
    are stored.  When
    temp_store is MEMORY (2) temporary tables and indices are kept
    as if they were in pure in-memory databases.
    When temp_store is FILE (1) temporary tables and indices are stored
    in a file.  The temp_store_directory pragma can be used to specify
    the directory containing temporary files when
    FILE is specified.  When the temp_store setting is changed,
    all existing temporary tables, indices, triggers, and views are
    immediately deleted.

    It is possible for the library compile-time C preprocessor symbol
    SQLITE_TEMP_STORE to override this pragma setting.
    The following table summarizes
    the interaction of the SQLITE_TEMP_STORE preprocessor macro and the
    temp_store pragma:

    
    
    SQLITE_TEMP_STORE
        PRAGMAtemp_store
        Storage used forTEMP tables and indices
    0
        any
        file
    1
        0
        file
    1
        1
        file
    1
        2
        memory
    2
        0
        memory
    2
        1
        file
    2
        2
        memory
    3
        any
        memory


c2: Pragma statements supported by SQLite
c3: PRAGMA temp_store_directory
c5: pragma.html#pragma_temp_store_directory
c4:
PRAGMA temp_store_directory;
      PRAGMA temp_store_directory = 'directory-name';
    Query or change the value of the sqlite3_temp_directory global
    variable, which many operating-system interface backends use to
    determine where to store temporary tables and indices.

    When the temp_store_directory setting is changed, all existing temporary
    tables, indices, triggers, and viewers in the database connection that
    issued the pragma are immediately deleted.  In
    practice, temp_store_directory should be set immediately after the first
    database connection for a process is opened.  If the temp_store_directory
    is changed for one database connection while other database connections
    are open in the same process, then the behavior is undefined and
    probably undesirable.

    Changing the temp_store_directory setting is not threadsafe.
    Never change the temp_store_directory setting if another thread
    within the application is running any SQLite interface at the same time.
    Doing so results in undefined behavior.  Changing the temp_store_directory
    setting writes to the sqlite3_temp_directory global
    variable and that global variable is not protected by a mutex.

    The value directory-name should be enclosed in single quotes.
    To revert the directory to the default, set the directory-name to
    an empty string, e.g., PRAGMA temp_store_directory = ''.  An
    error is raised if directory-name is not found or is not
    writable. 

    The default directory for temporary files depends on the OS.  Some
    OS interfaces may choose to ignore this variable and place temporary
    files in some other directory different from the directory specified
    here.  In that sense, this pragma is only advisory.

    
    
    This pragma is deprecated and exists
    for backwards compatibility only.  New applications
    should avoid using this pragma.  Older applications should discontinue
    use of this pragma at the earliest opportunity.  This pragma may be omitted
    from the build when SQLite is compiled using SQLITE_OMIT_DEPRECATED.


c2: Pragma statements supported by SQLite
c3: PRAGMA threads
c5: pragma.html#pragma_threads
c4:
PRAGMA threads;
      PRAGMA threads = N;
    Query or change the value of the 
    sqlite3_limit(db,SQLITE_LIMIT_WORKER_THREADS,...) limit for
    the current database connection.  This limit sets an upper bound
    on the number of auxiliary threads that a prepared statement is
    allowed to launch to assist with a query.  The default limit is 0
    unless it is changed using the SQLITE_DEFAULT_WORKER_THREADS
    compile-time option.  When the limit is zero, that means no
    auxiliary threads will be launched.

    This pragma is a thin wrapper around the
    sqlite3_limit(db,SQLITE_LIMIT_WORKER_THREADS,...) interface.


c2: Pragma statements supported by SQLite
c3: PRAGMA trusted_schema
c5: pragma.html#pragma_trusted_schema
c4:
PRAGMA trusted_schema;
      PRAGMA trusted_schema = boolean;
    The trusted_schema setting is a per-connection boolean that
    determines whether or not SQL functions and virtual tables that
    have not been security audited are allowed to be run by views,
    triggers, or in expressions of the schema such as CHECK constraints,
    DEFAULT clauses, generated columns, expression indexes, and/or
    partial indexes.  This setting can also be controlled using
    the sqlite3_db_config(db,SQLITE_DBCONFIG_TRUSTED_SCHEMA,...)
    C-language interface.
    In order to maintain backwards compatibility, this setting is
    ON by default.  There are advantages to turning it off, and most
    applications will be unaffected if it is turned off.  For that reason,
    all applications are encouraged to switch this setting off on every
    database connection as soon as that connection is opened.
    The -DSQLITE_TRUSTED_SCHEMA=0 compile-time option will cause
    this setting to default to OFF.


c2: Pragma statements supported by SQLite
c3: PRAGMA user_version
c5: pragma.html#pragma_user_version
c4:
PRAGMA schema.user_version;
      PRAGMA schema.user_version = integer ;

  
    The user_version pragma will get or set
       the value of the user-version integer at offset 60 in the
       database header.  The user-version is an integer that is
       available to applications to use however they want.  SQLite
       makes no use of the user-version itself.

    See also the application_id pragma and schema_version pragma.


c2: Pragma statements supported by SQLite
c3: PRAGMA vdbe_addoptrace
c5: pragma.html#pragma_vdbe_addoptrace
c4:
PRAGMA vdbe_addoptrace = boolean;

    If SQLite has been compiled with the SQLITE_DEBUG compile-time
    option, then the vdbe_addoptrace pragma can be used to cause a complete
    VDBE opcodes to be displayed as they are created during code generation.
    This feature is used for debugging SQLite itself.  See the 
    VDBE documentation for more 
    information.

    
    
    This pragma is intended for use when debugging SQLite itself.  It
    is only available when the SQLITE_DEBUG compile-time option
    is used.


c2: Pragma statements supported by SQLite
c3: PRAGMA vdbe_debug
c5: pragma.html#pragma_vdbe_debug
c4:
PRAGMA vdbe_debug = boolean;

    If SQLite has been compiled with the SQLITE_DEBUG compile-time
    option, then the vdbe_debug pragma is a shorthand for three other
    debug-only pragmas: vdbe_addoptrace, vdbe_listing, and vdbe_trace.
    This feature is used for debugging SQLite itself.  See the 
    VDBE documentation for more 
    information.

    
    
    This pragma is intended for use when debugging SQLite itself.  It
    is only available when the SQLITE_DEBUG compile-time option
    is used.


c2: Pragma statements supported by SQLite
c3: PRAGMA vdbe_listing
c5: pragma.html#pragma_vdbe_listing
c4:
PRAGMA vdbe_listing = boolean;

    If SQLite has been compiled with the SQLITE_DEBUG compile-time
    option, then the vdbe_listing pragma can be used to cause a complete
    listing of the virtual machine opcodes to appear on standard output
    as each statement is evaluated.
    With listing is on, the entire content of a program is printed
    just prior to beginning execution.  The statement
    executes normally after the listing is printed.
    This feature is used for debugging SQLite itself.  See the 
    VDBE documentation for more 
    information.

    
    
    This pragma is intended for use when debugging SQLite itself.  It
    is only available when the SQLITE_DEBUG compile-time option
    is used.


c2: Pragma statements supported by SQLite
c3: PRAGMA vdbe_trace
c5: pragma.html#pragma_vdbe_trace
c4:
PRAGMA vdbe_trace = boolean;

    If SQLite has been compiled with the SQLITE_DEBUG compile-time
    option, then the vdbe_trace pragma can be used to cause virtual machine
    opcodes to be printed on standard output as they are evaluated.
    This feature is used for debugging SQLite.  See the 
    VDBE documentation for more 
    information.

    
    
    This pragma is intended for use when debugging SQLite itself.  It
    is only available when the SQLITE_DEBUG compile-time option
    is used.


c2: Pragma statements supported by SQLite
c3: PRAGMA wal_autocheckpoint
c5: pragma.html#pragma_wal_autocheckpoint
c4:
PRAGMA wal_autocheckpoint;
          PRAGMA wal_autocheckpoint=N;

    This pragma queries or sets the write-ahead log 
    auto-checkpoint interval.
    When the write-ahead log is enabled (via the
    journal_mode pragma) a checkpoint will be run automatically whenever
    the write-ahead log equals or exceeds N pages in length.
    Setting the auto-checkpoint size to zero or a negative value
    turns auto-checkpointing off.
    
    This pragma is a wrapper around the
    sqlite3_wal_autocheckpoint() C interface.
    All automatic checkpoints are PASSIVE.

    Autocheckpointing is enabled by default with an interval
    of 1000 or SQLITE_DEFAULT_WAL_AUTOCHECKPOINT.


c2: Pragma statements supported by SQLite
c3: PRAGMA wal_checkpoint
c5: pragma.html#pragma_wal_checkpoint
c4:
PRAGMA schema.wal_checkpoint;
       PRAGMA schema.wal_checkpoint(PASSIVE);
       PRAGMA schema.wal_checkpoint(FULL);
       PRAGMA schema.wal_checkpoint(RESTART);
       PRAGMA schema.wal_checkpoint(TRUNCATE);
   

    If the write-ahead log is enabled (via the journal_mode pragma),
    this pragma causes a checkpoint operation to run on database
    database, or on all attached databases if database
    is omitted.  If write-ahead log mode is disabled, this pragma is a
    harmless no-op.

    Invoking this
    pragma without an argument is equivalent to calling the
    sqlite3_wal_checkpoint() C interface.
    Invoking this pragma with an argument is equivalent to calling the
    sqlite3_wal_checkpoint_v2() C interface with a 
    3rd parameter
    corresponding to the argument:

    
    PASSIVE
      Checkpoint as many frames as possible without waiting for any database 
      readers or writers to finish. Sync the db file if all frames in the log
      are checkpointed. This mode is the same as calling the
      sqlite3_wal_checkpoint() C interface. The
      busy-handler callback is never invoked in
      this mode.
   
    FULL
      This mode blocks 
      (invokes the busy-handler callback)
      until there is no
      database writer and all readers are reading from the most recent database
      snapshot. It then checkpoints all frames in the log file and syncs the
      database file. FULL blocks concurrent writers while it is
      running, but readers can proceed.
   
    RESTART
      This mode works the same way as FULL with the addition that after 
      checkpointing the log file it blocks (calls the 
      busy-handler callback)
      until all readers are finished with the log file. This ensures 
      that the next client to write to the database file restarts the log file 
      from the beginning. RESTART blocks concurrent writers while it is
      running, but allowed readers to proceed.
   
    TRUNCATE
      This mode works the same way as RESTART with the 
      addition that the WAL file is truncated to zero bytes upon successful
      completion.
    


    The wal_checkpoint pragma returns a single row with three
    integer columns.  The first column is usually 0 but will be
    1 if a RESTART or FULL or TRUNCATE checkpoint was blocked from completing,
    for example because another thread or process was actively
    using the database.  In other words, the first column is 0 if the
    equivalent call to sqlite3_wal_checkpoint_v2() would have returned
    SQLITE_OK or 1 if the equivalent call would have returned SQLITE_BUSY.
    The second column is the number of modified pages that have been
    written to the write-ahead log file.
    The third column is the number of pages in the write-ahead log file
    that have been successfully moved back into the database file at
    the conclusion of the checkpoint.
   The second and third column are -1 if there is no
    write-ahead log, for example if this pragma is invoked on a database
    connection that is not in WAL mode.


c2: Pragma statements supported by SQLite
c3: PRAGMA writable_schema
c5: pragma.html#pragma_writable_schema
c4:
PRAGMA writable_schema  = boolean;
         PRAGMA writable_schema = RESET

    When this pragma is on, and the SQLITE_DBCONFIG_DEFENSIVE flag
    is off, then the sqlite_schema table
    can be changed using ordinary UPDATE, INSERT, and DELETE
    statements.  If the argument is "RESET" then schema writing is
    disabled (as with "PRAGMA writable_schema=OFF") and, in addition, the
    schema is reloaded.  Warning:
    misuse of this pragma can easily result in
    a corrupt database file.


This page last modified on  2023-10-16 15:24:42 UTC


c2: The Bytecode() And Tables_Used() Table-Valued Functions
c3: 1. Overview
c5: bytecodevtab.html#overview
c4:
Bytecode and tables_used are virtual tables built into SQLite that
access information about prepared statements.
Both bytecode and tables_used operate as table-valued functions.
They take a single required argument which is either the text of
an SQL statement, or a pointer to an existing prepared statement.
The bytecode function returns one row of result for each bytecode
operation in the prepared statement.  The tables_used function returns
one row for each persistent btree (either a table or an index) accessed
by the prepared statement.


c2: The Bytecode() And Tables_Used() Table-Valued Functions
c3: 2. Usage
c5: bytecodevtab.html#usage
c4:
The bytecode and tables_used tables are only available if SQLite has
been compiled with the -DSQLITE_ENABLE_BYTECODE_VTAB compile-time option.
The CLI has been compiled that way, and so you can use the standard
CLI as a test platform to experiement.


Both virtual tables are read-only eponymous-only virtual tables.  You use them
by mentioning them directly in the FROM clause of a SELECT statement.
They both require a single argument which is the SQL statement to be
analyzed.  For example:

SELECT * FROM bytecode('SELECT * FROM bytecode(?1)');



The argument can be either the text of an SQL statement, in which case
the bytecode (or tables_used) for that statement is returned, or the
argument can be a parameter such as ?1 or $stmt that is later bound
to a prepared statement object using the
sqlite3_bind_pointer() interface.  Use a pointer type of
"stmt-pointer" for the sqlite3_bind_pointer() interface.


c2: The Bytecode() And Tables_Used() Table-Valued Functions
c3: 2.1. Schema
c5: bytecodevtab.html#schema
c4:
The schema of the bytecode table is:

CREATE TABLE bytecode(
  addr INT,
  opcode TEXT,
  p1 INT,
  p2 INT,
  p3 INT,
  p4 TEXT,
  p5 INT,
  comment TEXT,
  subprog TEXT,
  stmt HIDDEN
);



The first eight columns are the address, opcode, and operands for a
single bytecode in the virtual machine that implements the statement.
These columns are the same columns output when using EXPLAIN.  The
bytecode virtual tables shows all operations in the prepared statement,
both the main body of the prepared statement and in subprograms used
to implement triggers or foreign key actions.  The "subprog" field
is NULL for the main body of the prepared statement, or is the trigger
name or the string "(FK)" for triggers and foreign key actions.


The schema for the tables_used table is:

CREATE TABLE tables_used(
  type TEXT,
  schema TEXT,
  name TEXT,
  wr INT,
  subprog TEXT,
  stmt HIDDEN
);



The tables_used table is intended to show which btrees of the database file
are read or written by a prepared statement, both by the main statement
itself but also by related triggers and foreign key actions.  The columns
are as follows:


type &rarr; Either "table" or "index", depending on what role
the btree is serving.

schema &rarr; Which database file the btree is located in.
This will be "main" for the main database (the usual case), or "temp" for
TEMP tables and indexes, or the name assigned to attached databases by
the ATTACH statement.

name &rarr; The name of the table or index

wr &rarr; 0 if the object is read, 1 if the object is written

subprog &rarr;  The sub-program in which the object is
accessed.  NULL means the main body of the prepared statement.  Otherwise
this field is the name of a trigger or "(FK)" for a foreign key action.

This page last modified on  2022-01-08 05:02:57 UTC


c2: TH3
c3: 1. Overview
c5: th3.html#overview
c4:
SQLite Test Harness #3 (hereafter "TH3") is one of
three test harnesses used for testing SQLite.
TH3 meets the following objectives:


 TH3 is able to run on embedded platforms that lack the support
     infrastructure of workstations.

 TH3 tests SQLite in an as-deployed configuration using only
     published and documented interfaces.
     In other words, TH3 tests the compiled object code, not
     the source code, thus verifying that no problems were introduced
     by compiler bugs.  "Test what you fly and fly what you test."

 TH3 checks SQLite's response to out-of-memory errors, disk I/O
     errors, and power loss during transaction commit. 

 TH3 exercises SQLite in a variety of run-time configurations
     (UTF8 vs UTF16, different pages sizes, varying journal modes, etc.)
     

 TH3 achieves 100% branch test coverage (and 100% 
    MC/DC)
    over the SQLite core.
    (Test coverage of extensions such as FTS and RTREE is less than 100%).
     


TH3 was originally written for validation testing only, but has
subsequently been used for development testing and debugging
as well, and has proven very helpful in those roles.  A full-coverage
test takes less than five minutes on a workstation and hence
serves as a fast regression test during day-to-day maintenance
of the SQLite code base.


c2: TH3
c3: 1.1. History
c5: th3.html#history
c4:
TH3 originated from an effort to test SQLite on 
SymbianOS.
Prior to TH3, all SQLite tests were run using the
TCL script language, but TCL would not (easily)
compile on SymbianOS which made testing difficult.  The first attempt
to remedy this problem was the "TH1" (Test Harness #1) scripting 
language - a reimplementation of parts of the TCL language in a 
more portable form that would compile and run on SymbianOS, and 
that was sufficient to run the SQLite tests.  TH1
did not survive as a standard testing tool for SQLite,
but it did find continued service as a
scripting language used to customize the 
Fossil version control system.
There was also a "Test Harness #2" which was an attempt to
create a simple scripting language using operator prefix notation
to drive tests. TH3 was the third attempt.

At about that same time, some avionics manufacturers were
expressing interest in SQLite, which prompted the SQLite developers
to design TH3 to support the rigorous testing standards of
DO-178B.

The first code for TH3 was laid down on 2008-09-25.
An intense effort over the next 10 months resulted in TH3 achieving
100% MC/DC on 2009-07-25.  The TH3 code continues to be improved and
expanded.

As of 2018-05-19, the TH3 source tree consists 
and well over 500,000 lines of source code in 1709 separate files.


c2: TH3
c3: 2. Operation
c5: th3.html#operation
c4:
TH3 is a test program generator.  The output of TH3 is a program
implemented in C-code and intended to be
linked against the SQLite library under test.  The generated test
program is compiled and run on the target platform in order to verify
correct operation of SQLite on that platform.

The inputs to TH3 are test modules written in C or SQL and
small configuration
files that determine how to initialize SQLite.  The
TH3 package includes  1,444 test
modules and more than 47 configurations (as of 2018-05-19).
New modules and configurations
can be added to customize TH3 for specialized applications.
Each time TH3 is run, it reads
a subset of the available test modules and configuration files to generate
a custom C program that performs all of the specified tests under all
specified configurations.  A complete test of SQLite normally involves running
TH3 multiple times to generate multiple test programs covering different
aspects of SQLite's operation, then linking all test programs against
a common SQLite library and running them separately on the target platform.


There are no arbitrary limits in TH3.  One could generate a
single test program that contained all test modules and all configuration files.
However, such a test program might be too large to deploy on embedded
platforms.  (As of 2018-05-19, a full-up TH3 test is over 850,000 lines and
58MB of C code.)  TH3 provides the ability to break the library of test
modules up into smaller, more easily digested pieces.

Each individual test module might contain dozens, hundreds, or thousands
of separate tests.  The test modules can be written in C or as scripts of
SQL or a mixture of the two.  About two-thirds of the existing test modules are
written in pure SQL with the remainder either in pure C or a combination 
of C and SQL.


Each test module file contains a header which describes the circumstances
under which the test is valid.  For a particular configuration, only those
modules that are compatible with the configuration are run.


c2: TH3
c3: 3. Generating A Test Program
c5: th3.html#generating_a_test_program
c4:
The TH3 program generator is a TCL script named "mkth3.tcl".
To generate a test program, one has merely to run this script and supply
the names of files containing test modules and configurations on the
command line.  Test modules are files that use the ".test" suffix
and configurations are files that use the ".cfg" suffix.  A
typical invocation of mkth3.tcl might look something like the following:

tclsh mkth3.tcl *.test *.cfg &gt;testprog1.c


The output from the mkth3.tcl script is a C program that contains
everything needed to run the tests - everything that is except for
the SQLite library itself.  The generated test program contains 
implementations for all of the support interfaces used by the test
modules and it contains the main() routine that drives the
tests.  To convert the test program into a working executable, simply
compile it against SQLite:

cc -o testprog1 testprog1.c sqlite3.c


The compilation step shown immediately above is merely representative.
In a working installation, one would normally want
to specify optimization parameters and compile-time switches on the
compiler command line.

For testing on embedded systems, the mkth3.tcl script and the compiler
steps shown above are performed on an ordinary workstation using
a cross-compiler, then the resulting test program is
transferred onto the device to be run.

Once the test program is generated, it is run with no arguments to
perform the tests.  Progress information as well as error diagnostics
appear on standard output.  (Alternative output arrangements can be made
using a compile-time option for embedded devices that lack a standard
output channel.) The program returns zero if there are no
errors and non-zero if any problems were detected.

Typical output from a single TH3 test program run looks like this:

With SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e
-DSQLITE_COVERAGE_TEST
-DSQLITE_NO_SYNC
-DSQLITE_SYSTEM_MALLOC
-DSQLITE_THREADSAFE=1
Config-begin c1.
Begin c1.pager08
End c1.pager08
Begin c1.build33
End c1.build33
Begin c1.orderby01
End c1.orderby01
... 15014 lines of output omitted ....
Begin 64k.syscall01
End 64k.syscall01
Begin 64k.build01
End 64k.build01
Begin 64k.auth01
End 64k.auth01
Config-end 64k. TH3 memory used: 6373738
Config-begin wal1.
Begin wal1.wal37
End wal1.wal37
Config-end wal1. TH3 memory used: 100961
All 226 VDBE coverage points reached
th3: 0 errors out of 1442264 tests in 213.741 seconds. 64-bit little-endian
th3: SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e


The output begins with a report of the SQLITE_SOURCE_ID
(cross-checked again sqlite3_sourceid()) for the
SQLite under test and the compile-time options used as reported
by sqlite3_compileoption_get().  The output concludes with a summary
of the test results and a repeat of the SQLITE_SOURCE_ID.  If any
errors are detected, additional lines detail the problem.  The error
reporting lines always begin with a single space character so that they
can be quickly extracted from large output files using:

grep "&#94; "


The default output shows the beginning and end of each configuration
and test module combination.  In the example above "c1" and "64k" are
configurations and "pager08", "build33", "orderby01", etc. are test modules.
Compile-time and run-time options are available to increase or decrease
the amount of output.
The output can be increased by showing each test case within each
test module.  The output can be decreased
by degrees: omitting test modules starts and stops,
omitting configuration starts and stops, and finally by omitting all output.


c2: TH3
c3: 3.1. Test Automation Scripts
c5: th3.html#test_automation_scripts
c4:
TH3 comes with additional TCL scripts that help automate the testing
process on workstations.  The "th3make" script automatically runs "mkth3.tcl"
and "gcc" and then runs the resulting test program and checks the results.
Arguments to th3make include all of the "*.test" test modules and 
"*.cfg" configurations that are to be included in the test.  Additional
options to th3make can cause the test program to be compiled using different
compilers (GCC, Clang, MSVC), to use different output verbosity levels, to
run the test program under valgrind, to check the output for coverage using
gcov, and so forth.  The th3make script also accepts "*.rc" filenames as
arguments.  These *.rc files are just collections of other arguments that
are commonly used together for a single purpose.  For example, the "quick.rc"
file contains a set of eight arguments to th3make that run a fast (3-minute)
full-coverage test.  This allows the operator to type "./th3make quick.rc" as
a short-cut to typing out all of the required command-line options.  The
following are a few of the more than 40 available *.rc files:


alignmentN.rc - 
    These files contain -D options to the compiler that are used by
    various notable downstreams.
cov.rc - Options for measuring test coverage
extensions.rc - Options to enable FTS4, R-Trees,
    and STAT4.
fast.rc - Run most tests, including those not needed for
    coverage, skipping only soak tests, using delivery compiler options
    (ex: -O3)
memdebug.rc - like test.rc but also enable
    -DSQLITE_MEMDEBUG.
min.rc - Run the minimum set of tests needed for 100% coverage.
quick.rc - Run all tests required for 100% coverage tests
    using -Os and -DSQLITE_DEBUG.
test.rc - Run the same tests as in fast.rc but without
    compiler optimization and enabling options like
    -DSQLITE_DEBUG and
    -DSQLITE_ENABLE_EXPENSIVE_ASSERT.
test-ex.rc - long-running soak tests.


The TH3 repository also includes the "multitest.tcl" script, another
TCL script used to automate TH3 testing on workstations.  Multitest.tcl
automatically compiles SQLite, then
runs ./th3make repeatedly with a variety of alignments, and captures
the output in a succinct summary screen.  A typical multitest.tcl run
generates output that looks like this:

./multitest.tcl -q --jobs 3
start-time: 2018-05-19 03:17:12 UTC
file mkdir sqlite3bld
cd sqlite3bld
exec sh /ramdisk/sqlite/configure
file copy -force config.h ../config.h
exec make clean sqlite3.c
file rename sqlite3.c ../sqlite3.c
file rename sqlite3.h ../sqlite3.h
exec make clean sqlite3.c OPTS=-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1
file rename sqlite3.c ../sqlite3udl.c
exec make clean sqlite3.c OPTS=-DSQLITE_SMALL_STACK=1
file rename sqlite3.c ../sqlite3ss.c
cd ..
*******************************************************************************
t01: cov.rc.................................................... Ok   (00:03:42)
t02: cov.rc ++STAT4 ++DESERIALIZE -D_HAVE_SQLITE_CONFIG_H...... Ok   (00:04:45)
t03: vfs-cov.rc................................................ Ok   (00:03:59)
t04: demo.rc................................................... Ok   (00:00:05)
t07: test.rc ../th3private/*.test.............................. Ok   (00:00:21)
t08: test.rc ../th3private/*.test ++STAT4...................... Ok   (00:01:41)
t05: quick.rc.................................................. Ok   (00:04:26)
t09: quick.rc ~TEST_REALLOC_STRESS -funsigned-char............. Ok   (00:05:39)
t10: quick.rc ~THREADSAFE=0 -DLONGDOUBLE_TYPE=double........... Ok   (00:03:24)
t06: quick.rc extensions.rc -D_HAVE_SQLITE_CONFIG_H............ Ok   (00:09:03)
t11: quick.rc sqlite3ss.c ~MAX_ATTACHED=125.................... Ok   (00:04:39)
t12: quick.rc ~BYTEORDER=0 ++RTREE............................. Ok   (00:07:28)
t13: quick.rc ~DISABLE_INTRINSIC ++RTREE....................... Ok   (00:07:31)
t16: quick.rc ~TRACE_SIZE_LIMIT=15 cov1/main16.test............ Ok   (00:00:22)
t14: quick.rc ~DIRECT_OVERFLOW_READ -fsigned-char.............. Ok   (00:04:35)
t15: quick.rc ~UNTESTABLE ~EXTRA_IFNULLROW..................... Ok   (00:01:44)
t17: quick.rc ~MAX_MMAP_SIZE=0................................. Ok   (00:04:46)
t18: quick.rc ++NULL_TRIM ++OFFSET_SQL_FUNC.................... Ok   (00:04:47)
t19: quick.rc ++BATCH_ATOMIC_WRITE ++DESERIALIZE............... Ok   (00:05:41)
t20: lean1.rc quick.rc......................................... Ok   (00:03:09)
t22: test.rc alignment2.rc sqlite3udl.c........................ Ok   (00:44:22)
t21: test.rc alignment1.rc..................................... Ok   (01:02:32)
t23: memdebug1.rc extensions.rc................................ Ok   (01:49:58)
t25: valgrind1.rc -O3 extensions.rc............................ Ok   (00:56:08)
t24: memdebug2.rc extensions.rc................................ Ok   (01:43:34)
t27: test-ex1.rc............................................... Ok   (00:45:00)
t26: valgrind2.rc -O3 extensions.rc............................ Ok   (01:02:52)
t29: test-ex3.rc............................................... Ok   (00:31:48)
t28: test-ex2.rc............................................... Ok   (01:12:03)
t30: test-ex4.rc............................................... Ok   (01:09:47)
t32: test.rc alignment4.rc -m32 CC=clang....................... Ok   (00:48:31)
t31: test.rc alignment3.rc sqlite3udl.c........................ Ok   (01:22:29)
t34: test.rc alignment6.rc..................................... Ok   (00:35:31)
t33: test.rc alignment5.rc extensions.rc....................... Ok   (00:59:33)
t35: test.rc alignment7.rc..................................... Ok   (00:44:10)
t40: fast.rc alignment2.rc sqlite3udl.c........................ Ok   (00:15:46)
t39: fast.rc alignment1.rc extensions.rc -m32.................. Ok   (00:33:19)
t36: test.rc ~MUTATION_TEST.................................... Ok   (00:35:45)
t42: fast.rc alignment4.rc..................................... Ok   (00:13:03)
t43: fast.rc alignment5.rc..................................... Ok   (00:13:32)
t44: fast.rc alignment6.rc..................................... Ok   (00:11:41)
t41: fast.rc alignment3.rc sqlite3udl.c........................ Ok   (00:26:31)
t45: fast.rc alignment7.rc..................................... Ok   (00:12:57)
t46: fast.rc -fsanitize=undefined.............................. Ok   (00:38:18)
*******************************************************************************
0 failures on 44 th3makes and 198583082 tests in (07:16:01) 3 cores on bella
SQLite 3.24.0 2018-05-18 17:58:33 c6071ac99cfa4b6272ac4d739fc61a85acb544f6c1c2a


As can be seen above, a single run
of multitest.tcl invokes th3make dozens of times and takes between 12 and 24
CPU hours.  The middle section of the output shows the arguments to each
individual th3make run and the result and elapse time for that th3make.
All build products and output for the separate th3make runs are
captures in subdirectories for post-test analysis.
The two-line summary at the bottom shows the total number of errors and tests
over all th3make runs and the total elapse time, together with the 
SQLITE_SOURCE_ID information for the version of SQLite that was
tested.  This summary information is recorded in the
release
checklist during final testing.

Abbreviations are applied in the multitest.tcl output so that
each th3make invocation will fit on a single 80-column output line.
The initial "th3make" verb is omitted.
"~" is shorthand for "-DSQLITE_" and "++" is stands for
"-DSQLITE_ENABLE".  Hence, multitest.tcl output line

quick.rc ~DISABLE_INTRINSIC ++RTREE


Really means

th3make quick.rc -DSQLITE_DISABLE_INTRINSIC -DSQLITE_ENABLE_RTREE


c2: TH3
c3: 4. Test Coverage
c5: th3.html#test_coverage
c4:
Using one particular subset of the available TH3 test modules (the "cov1"
tests) SQLite obtained 
100% branch test coverage and 100% MC/DC as measured
by gcov
on Linux x86 and x86_64 hardware.  All releases of SQLite since
version 3.6.17 (2009-08-10) have been tested to this standard. 
The SQLite developers 
are committed to maintaining 100% branch coverage and MC/DC for all 
future releases of SQLite.

The cov1 test set used to obtain 100% branch test coverage are only a
subset of the tests currently implemented using TH3.  New test modules are
added on a regular basis.


c2: TH3
c3: 5. Mutation Testing
c5: th3.html#mutation_testing
c4:
The TH3 source tree contains a scripted name
"mutation-test.tcl" that automates the process of
mutation testing.

The mutation-test.tcl script takes care of all of the details for
running a mutation test:


 The script compiles the TH3 test harness into machine code ("th3.o") if
     necessary.
 The script compiles the sqlite3.c source file into assembly language
     ("sqlite3.s") if necessary.
 The script loops through instructions in the assembly language file
     to locate branch operations.
     
     The script makes a copy of the original sqlite3.s file.
     The copy is edited to change the branch instruction into either
         a no-op or an unconditional jump.
     The copy of sqlite3.s is assembled into sqlite3.o then linked
         again th3.o to generate the "th3" executable.
     The "th3" binary is run and the output checked for errors.
     
 The script shows progress for each cycle of the previous step then
     displays a summary of "survivors" at the end.  A "survivor" is a
     mutation that was not detected by TH3.


Mutation testing can be slow, since each test can take up to 5
minutes on a fast workstation, and there are two tests for each
branch instructions, and over 20,000 branch instructions.  Efforts are
made to expedite operation.  For example, TH3 is compiled in such a
way that it exits as soon as it finds the first error, and as many
of the mutations are easily detected,  many cycles happen in only
a few seconds.  Nevertheless, the mutation-test.tcl script includes
command-line options to limit the range of code lines tested so that
mutation testing only needs to be performed on blocks of code that
have recently changed.


c2: TH3
c3: 6. TH3 License
c5: th3.html#th3_license
c4:
SQLite itself is in the public domain and
can be used for any purpose.  But TH3 is proprietary and requires a license.


Even though open-source users do not have direct access to TH3, all
users of SQLite benefit from TH3 indirectly since each version of SQLite is
validated running TH3 on multiple platforms (Linux, Windows, WinRT, Mac,
OpenBSD) prior to release.  So anyone using an official release
of SQLite can deploy their application with the confidence of knowing that
it has been tested using TH3.  They simply cannot rerun those tests
themselves without purchasing a TH3 license.
This page last modified on  2022-01-08 05:02:57 UTC


c2: Why SQLite Does Not Use Git
c3: 1. Introduction
c5: whynotgit.html#introduction
c4:
SQLite does not use the
Git version control system.
SQLite uses
Fossil instead, which is a
version control system that was specifically designed
and written to support SQLite.


People often wonder why SQLite does not use the
Git version control system like everybody
else.
This article attempts to answer that question.  Also,
in section 3,
this article provides hints to Git users
about how they can easily access the SQLite source code.


This article is not a comparison between Fossil
and Git.  See
https://fossil-scm.org/fossil/doc/trunk/www/fossil-v-git.wiki
for one comparison of the two systems.  Other third-party
comparisons are available as well - use a search engine to find them.


This article is not advocating that you switch your projects
away from Git.  You can use whatever version control system you want.
If you are perfectly happy with Git, then by all means keep using
Git.  But, if Git is not working well for you or you are wondering
if it can be improved or if there is something better,
then maybe try to understand the perspectives presented below.
Use the insights thus obtained to find or write a different and
better version control system, or to just make
improvements to Git itself.


c2: Why SQLite Does Not Use Git
c3: 1.1. Edits
c5: whynotgit.html#edits
c4:
This article has been revised multiple times in an attempt
to improve clarity, address concerns and misgivings,
and to fix errors.
The complete edit history for this document can be seen at
https://sqlite.org/docsrc/finfo/pages/whynotgit.in.
(Usage hint: Click on any two nodes of the graph for a diff.
BTW, are there any Git web interfaces that offers a similar
capability?)


c2: Why SQLite Does Not Use Git
c3: 2.1. Git does not provide good situational awareness
c5: whynotgit.html#git_does_not_provide_good_situational_awareness
c4:
When I want to see what has been happening on SQLite, I visit the
timeline and in a single
screen I see a summary of the latest changes, on all branches.
In a few clicks, I can drill down as much detail as I
want.  I can even do this from a phone.


GitHub and GitLab offer nothing comparable.  The closest I have
found is the network,
which is slow to render (unless it is already cached), does not
offer nearly as much details, and scarcely works at all on mobile.
The commits view
of GitHub provides more detail, renders quickly,
and works on mobile, but only shows a single branch at a time,
so I cannot easily know if I've seen all of the recent changes.
And even if GitHub/GitLab did offer better interfaces, both are
third-party services.  They are not a core part of Git.  Hence,
using them introduces yet another dependency into the project.


I am told that Git users commonly install third-party graphical
viewers for Git, many of which do a better job of showing recent
activity on the project.  That is great, but these are still
more third-party applications that must be installed and
managed separately.  Many are platform-specific.  (One of the
better ones, GitUp, only works on Mac, for
example.)  All require that you first sync your local repository
then bring up their graphical interface on your desktop.  And
even with all that, I still cannot see what I typically want to
see without multiple clicks.  Checking on project status from
a phone while away from the office is not an option.


c2: Why SQLite Does Not Use Git
c3: 2.2. Git makes it difficult to find successors (descendants) of a check-in
c5: whynotgit.html#git_makes_it_difficult_to_find_successors_descendants_of_a_check_in
c4:
Git lets you look backwards in time, but not forwards.
Given some historical check-in, you can see what came before,
but it is challenging see what came next.


In contrast, Fossil offers helpful displays such as
https://sqlite.org/src/timeline?df=major-release
to show all check-ins that are derived from the most
recent major release.





All SQLite check-ins derived from the most recent major release



It is not impossible to find the descendants of a check-in
in Git.  It is merely difficult.  For example,
there is a
stackoverflow page
showing the command sequence for finding the descendants of a check-in
in unix:

git rev-list --all --parents | grep ".\{40\}.*.*" | awk '{print $1}'




But this is not the same thing.  The command above gives
a list of descendants without showing the branching structure, which
is important for understanding what happened.  And the command only works
if you have a local clone of the repository; finding the descendants of
a check-in is not something you can do with web interfaces such
as GitHub or GitLab.


This is not really about just finding the descendants of a check-in
from time to time.  The fact that descendants are readily available in
Fossil means that the information pervades the web pages provided by
Fossil.  One example: Every Fossil check-in information page
(example) shows
a small "Context" graph of the immediate predecessor and successors
to that check-in.  This helps the user maintain better situational
awareness, and it provides useful capabilities, such as the ability
to click forward to the next check-in within a sequence.  Another example:
Fossil easily shows the context around a specific check-in
(example)
which again helps to promote situational awareness and a deeper
understanding of what is happening in the code.  There is a
whole page of additional examples
in the Fossil documentation.


All of the above is theoretically possible with Git, given the right extensions
and tools and using the right commands.  But it is not easy to do,
and so it rarely gets done.  Consequently, developers have less awareness
of what is happening in the code.


c2: Why SQLite Does Not Use Git
c3: 2.3. The mental model for Git is needlessly complex
c5: whynotgit.html#the_mental_model_for_git_is_needlessly_complex
c4:
The complexity of Git
distracts attention from the software under development.  A user of Git
needs to keep all of the following in mind:

 The working directory
 The "index" or staging area
 The local head
 The local copy of the remote head
 The actual remote head


Git has commands (and/or options on commands) for moving and
comparing content between all of these locations.

In contrast,
Fossil users only need to think about their working directory and
the check-in they are working on.  That is 60% less distraction.
Every developer has a finite number of brain-cycles.  Fossil
requires fewer brain-cycles to operate, thus freeing up
intellectual resources to focus on the software under development.

One user of both Git and Fossil
writes in HN:


Fossil gives me peace of mind that I have everything ... synced to
the server with a single command....
I never get this peace of mind with git.


c2: Why SQLite Does Not Use Git
c3: 2.4. Git does not track historical branch names
c5: whynotgit.html#git_does_not_track_historical_branch_names
c4:
Git keeps the complete DAG of the check-in sequence.  But branch
tags are local information that is not synced and not retained
once a branch closes.
This makes review of historical
branches tedious.


As an example, suppose a customer asks you:
"What ever became of that 'prefer-coroutine-sort-subquery' branch
from two years ago?"
You might try to answer by consulting the history in
your version control system, thusly:


GitHub: https://github.com/sqlite/sqlite/commits/prefer-coroutine-sort-subquery
Fossil: https://sqlite.org/src/timeline?r=prefer-coroutine-sort-subquery



The Fossil view clearly shows that the branch was eventually merged back into
trunk.  It shows where the branch started, and it shows two occasions where changes
on trunk were merged into the branch.  GitHub shows none of this.  In fact, the
GitHub display is mostly useless in trying to figure out what happened.


Many readers have recommended various third-party GUIs for Git that
might do a better job of showing historical development activity.  Maybe
some of them do work better than native Git and/or GitHub, though they
will all be hampered by the fact that Git does not preserve historical
branch names across syncs.  And even if those other tools are better,
the fact that it is necessary to go to a third-party tool to get the information
desired does not speak well of the core system.


c2: Why SQLite Does Not Use Git
c3: 2.5. Git requires more administrative support
c5: whynotgit.html#git_requires_more_administrative_support
c4:
Git is complex software.
One needs an installer of some kind to put Git on a developer
workstation, or to upgrade to a newer version of Git.
Standing up a Git server is non-trivial, and so most developers
use a third-party service such as GitHub or GitLab,
and thus introduce additional dependencies.


In contrast, Fossil is a single standalone binary which is
installed by putting it on $PATH.  That one binary contains all
the functionality of core Git and also GitHub and/or GitLab.  It
manages a community server with wiki, bug tracking, and forums,
provides packaged downloads for consumers, login managements,
and so forth, with no extra software required.  Standing up a
community server for Fossil takes minutes.  And Fossil is efficient.
A Fossil server will run fine on a $5/month VPS or a Raspberry Pi,
whereas GitLab and similar require beefier hardware.


Less administration means that programmers spend more time working
on the software (SQLite in this case) and less time fussing with
the version control system.


c2: Why SQLite Does Not Use Git
c3: 2.6. Git provides a poor user experience
c5: whynotgit.html#git_provides_a_poor_user_experience
c4:
The following https://xkcd.com/1597/ cartoon is an
exaggeration, yet hits close to home:




Let's be real.  Few people dispute that Git provides
a suboptimal user experience.  A lot of
the underlying implementation shows through into the user
interface.  The interface is so bad that there is even a
parody site that generates
fake git man pages.

Designing software is hard.  It takes a lot of focus.
A good version control system should provide the developer with
assistance, not frustration.  Git has gotten better in this
regard over the past decade, but it still has a long way to go.


c2: Why SQLite Does Not Use Git
c3: 3. A Git-User's Guide To Accessing SQLite Source Code
c5: whynotgit.html#a_git_user_s_guide_to_accessing_sqlite_source_code
c4:
If you are a devoted Git user, you can still easily access SQLite.
This section gives some hints on how to do so.


c2: Why SQLite Does Not Use Git
c3: 3.1. The Official GitHub Mirror
c5: whynotgit.html#the_official_github_mirror
c4:
As of 2019-03-20, there is now an
official Git mirror of the
SQLite sources on GitHub.

The mirror is an incremental export of the
canonical Fossil repository for
SQLite.  A cron-job updates the GitHub repository once an hour.
This is a one-way, read-only code mirror.  No pull requests or
changes are accepted via GitHub.  The GitHub repository merely copies
the content from the Fossil repository.  All changes are input via
Fossil.


The hashes that identify check-ins and files on the Git mirror are
different from the hashes in Fossil.  There are many reasons for
this, chief among them that Fossil uses a SHA3-256 hash whereas
Git uses a SHA1 hash.  During export, the original Fossil hash for
each check-in is added as a footer to check-in comments.  To avoid
confusion, always use the original Fossil hash, not the Git hash,
when referring to SQLite check-ins.


c2: Why SQLite Does Not Use Git
c3: 3.2. Web Access
c5: whynotgit.html#web_access
c4:
The SQLite Fossil Repository contains links
for downloading  a Tarball, ZIP Archive, or SQLite Archive for any
historical version of SQLite.  The URLs for these downloads are
simple and can be incorporated easily into automated tools.  The format is:


https://sqlite.org/src/tarball/VERSION/sqlite.tar.gz



Simply replace VERSION with some description of the version to be
downloaded.  The VERSION can be a prefix of the cryptographic hash
name of a specific check-in, or the name of a branch (in which case the
most recent version of the branch is fetched) or a tag for a specific
check-in like "version-3.23.1":


https://sqlite.org/src/tarball/version-3.23.1/sqlite.tar.gz



To get the latest release, use "release"
for VERSION, like this:


https://sqlite.org/src/tarball/release/sqlite.tar.gz



To get the latest trunk check-in, us "trunk" for VERSION:


https://sqlite.org/src/tarball/trunk/sqlite.tar.gz



And so forth.
For ZIP archives and SQLite Archives, simply change the "/tarball/" element
into either "/zip/" or "/sqlar/", and maybe also change the name of the
download file to have a ".zip" or ".sqlar" suffix.


c2: Why SQLite Does Not Use Git
c3: 3.3. Fossil Access
c5: whynotgit.html#fossil_access
c4:
Fossil is easy to install and use.  Here are the steps for unix.
(Windows is similar.)



Download the self-contained Fossil executable from
https://fossil-scm.org/fossil/uv/download.html and put the executable
somewhere on your $PATH.
mkdir ~/fossils
fossil clone https://sqlite.org/src ~/fossils/sqlite.fossil
mkdir ~/sqlite; cd ~/sqlite
fossil open ~/fossils/sqlite.fossil



At this point you are ready to type "./configure; make"
(or on Windows with MSVC, "nmake /f Makefile.msc").


To change your checkout to a different version of Fossil use
the "update" command:


fossil update VERSION



Use "trunk" for VERSION to get the latest trunk version of SQLite.
Or use a prefix of a cryptographic hash name, or the name of some branch
or tag.  See
https://fossil-scm.org/fossil/doc/trunk/www/checkin_names.wiki for more
suggestions on what names can be used for VERSION.


Use the "fossil ui" command from within the ~/sqlite checkout to
bring up a local copy of the website.


Additional documentation on Fossil can be found at
https://fossil-scm.org/fossil/doc/trunk/www/permutedindex.html


Do not be afraid to explore and experiment.
Without a log-in you won't be able to
push back any changes you make, so you cannot damage the project.


c2: Why SQLite Does Not Use Git
c3: 3.4. Verifying Source Code Integrity
c5: whynotgit.html#verifying_source_code_integrity
c4:
If you need to verify that the SQLite source code that you have is
authentic and has not been modified in any way (perhaps by an adversary)
that can be done using a few simple command-line tools.  At the root
of the SQLite source tree is a file named "manifest".  The manifest
file contains the name of every other file in the source tree together
with either a SHA1 or SHA3-256 hash for that file.  (SHA1 is used for
older files and SHA3-256 for newer files.)  You can write a
script to extract these hashes and verify them against the source code
files.  The hash name for the check-in is just the SHA3-256 hash of the
"manifest" file itself, possibly with the last line omitted if the
last line begins with "# Remove this line..."


c2: Why SQLite Does Not Use Git
c3: 4. See Also
c5: whynotgit.html#see_also
c4:
Other pages that talk about Fossil and Git include:

Fossil vs. Git
What others say about Fossil and Git

This page last modified on  2023-02-27 02:07:35 UTC


c2: EXPLAIN QUERY PLAN
c3: 1. The EXPLAIN QUERY PLAN Command
c5: eqp.html#the_explain_query_plan_command
c4:
Warning: The data returned by the EXPLAIN QUERY PLAN command is 
intended for interactive debugging only.  The output format may change
between SQLite releases. Applications should not depend on the output
format of the EXPLAIN QUERY PLAN command.


Alert: As warned above, the EXPLAIN QUERY PLAN output format did
change substantially with the version 3.24.0 release (2018-06-04).
Additional minor changes occurred in version 3.36.0 (2021-06-18).
Further changes are possible in subsequent releases.


The EXPLAIN QUERY PLAN SQL command is used to obtain a high-level
description of the strategy or plan that SQLite uses to implement a specific
SQL query.  Most significantly, EXPLAIN QUERY PLAN reports on the way in 
which the query uses database indices. This document is a guide to
understanding and interpreting the EXPLAIN QUERY PLAN output.  Background
information is available separately:


 A primer on How SQLite Works.
 Notes on the query optimizer.
 How indexing works.
 The next generation query planner.


A query plan is represented as a tree.
In raw form, as returned by sqlite3_step(), each node of the tree
consists of four fields:  An integer node id, an integer parent id,
an auxiliary integer field that is not currently used, and a description
of the node.
The entire tree is therefore a table with four columns and zero or more
rows.
The command-line shell will usually intercept this table and renders
it as an ASCII-art graph for more convenient viewing.  To disable the
shells automatic graph rendering and to display EXPLAIN QUERY PLAN
output in its tabular format, run the command ".explain off" to set
the "EXPLAIN formatting mode" to off.  To restore automatic graph rendering,
run ".explain auto".  You can see the current "EXPLAIN formatting mode" 
setting using the ".show" command.

One can also set the CLI into automatic EXPLAIN QUERY PLAN mode
using the ".eqp on" command:

sqlite&gt; .eqp on


 In automatic EXPLAIN QUERY PLAN mode, the shell automatically runs
a separate EXPLAIN QUERY PLAN query for each statement you enter and
displays the result before actually running the query.  Use the
".eqp off" command to turn automatic EXPLAIN QUERY PLAN mode back off.

EXPLAIN QUERY PLAN is most useful on a SELECT statement,
but may also appear with other statements that read data from database
tables (e.g. UPDATE, DELETE, INSERT INTO ... SELECT).


c2: EXPLAIN QUERY PLAN
c3: 1.1. Table and Index Scans
c5: eqp.html#table_and_index_scans
c4:
When processing a SELECT (or other) statement, SQLite may retrieve data from
  database tables in a variety of ways. It may scan through all the records in
  a table (a full-table scan), scan a contiguous subset of the records in a
  table based on the rowid index, scan a contiguous subset of the entries in a
  database index, or use a combination of the above strategies
  in a single scan. The various ways in which SQLite may retrieve data from a
  table or index are described in detail here.


  For each table read by the query, the output of EXPLAIN QUERY 
  PLAN includes a record for which the value in the "detail" column begins
  with either "SCAN" or "SEARCH". "SCAN" is used for a full-table scan,
  including cases where SQLite iterates through all records in a table
  in an order defined by an index. "SEARCH" indicates that only a subset of 
  the table rows are visited. Each SCAN or SEARCH record includes the
  following information:


   The name of the table, view, or subquery that data is read from.
   Whether or not an index or automatic index is used.
   Whether or not the covering index optimization applies.
   Which terms of the WHERE clause are used for indexing.



  For example, the following EXPLAIN QUERY PLAN command operates on a SELECT
  statement that is implemented by performing a full-table scan on table t1:
sqlite&gt; EXPLAIN QUERY PLAN SELECT a, b FROM t1 WHERE a=1;
QUERY PLAN
`--SCAN t1



  The example above shows
  SQLite picking full-table scan will visit all rows in the table.
  If the query were able to use an index, then the 
  SCAN/SEARCH record would include the name of the index and, for a
  SEARCH record, an indication of how the subset of rows visited is
  identified. For example:
sqlite&gt; CREATE INDEX i1 ON t1(a);
sqlite&gt; EXPLAIN QUERY PLAN SELECT a, b FROM t1 WHERE a=1;
QUERY PLAN
`--SEARCH t1 USING INDEX i1 (a=?)



  The previous example, SQLite uses index "i1" to optimize
  a WHERE clause term of the form (a=?) - in this case "a=1".
  The previous example could not use a covering index, but the following
  example can, and that fact is reflected in the output:
sqlite&gt; CREATE INDEX i2 ON t1(a, b);
sqlite&gt; EXPLAIN QUERY PLAN SELECT a, b FROM t1 WHERE a=1; 
QUERY PLAN
`--SEARCH t1 USING COVERING INDEX i2 (a=?)



  All joins in SQLite are implemented using nested scans. When a
  SELECT query that features a join is analyzed using EXPLAIN QUERY PLAN, one
  SCAN or SEARCH record is output for each nested loop. For example:
sqlite&gt; EXPLAIN QUERY PLAN SELECT t1.*, t2.* FROM t1, t2 WHERE t1.a=1 AND t1.b>2;
QUERY PLAN
|--SEARCH t1 USING INDEX i2 (a=? AND b>?)
`--SCAN t2



  The order of the entries indicates the nesting order. In
  this case, the scan of table t1 using index i2 is the outer loop (since it
  appears first)
  and the full-table scan of table t2 is the inner loop (since it appears
  last).
  In the following example, the positions of t1 and t2 in the FROM 
  clause of the SELECT are reversed. The query strategy remains the same.
  The output from EXPLAIN QUERY PLAN shows how the query is actually
  evaluated, not how it is specified in the SQL statement.
sqlite&gt; EXPLAIN QUERY PLAN SELECT t1.*, t2.* FROM t2, t1 WHERE t1.a=1 AND t1.b>2;
QUERY PLAN
|--SEARCH t1 USING INDEX i2 (a=? AND b>?)
`--SCAN t2





  If the WHERE clause of a query contains an OR expression, then SQLite might
  use the "OR by union" strategy (also known as the
  OR optimization). In this case there will be single top-level record
  for the search, with two sub-records, one for each index:
sqlite&gt; CREATE INDEX i3 ON t1(b);
sqlite&gt; EXPLAIN QUERY PLAN SELECT * FROM t1 WHERE a=1 OR b=2;
QUERY PLAN
`--MULTI-INDEX OR
   |--SEARCH t1 USING COVERING INDEX i2 (a=?)
   `--SEARCH t1 USING INDEX i3 (b=?)


c2: EXPLAIN QUERY PLAN
c3: 1.2. Temporary Sorting B-Trees
c5: eqp.html#temporary_sorting_b_trees
c4:
If a SELECT query contains an ORDER BY, GROUP BY or DISTINCT clause, 
  SQLite may need to use a temporary b-tree structure to sort the output
  rows. Or, it might use an index. Using an index is 
  almost always much more efficient than performing a sort.
  If a temporary b-tree is required, a record is added to the EXPLAIN
  QUERY PLAN output with the "detail" field set to a string value of
  the form "USE TEMP B-TREE FOR xxx", where xxx is one of "ORDER BY",
  "GROUP BY" or "DISTINCT". For example:

sqlite&gt; EXPLAIN QUERY PLAN SELECT c, d FROM t2 ORDER BY c;
QUERY PLAN
|--SCAN t2
`--USE TEMP B-TREE FOR ORDER BY



  In this case using the temporary b-tree can be avoided by creating an index
  on t2(c), as follows:

sqlite&gt; CREATE INDEX i4 ON t2(c);
sqlite&gt; EXPLAIN QUERY PLAN SELECT c, d FROM t2 ORDER BY c; 
QUERY PLAN
`--SCAN t2 USING INDEX i4


c2: EXPLAIN QUERY PLAN
c3: 1.3. Subqueries
c5: eqp.html#subqueries
c4:
In all the examples above, there has only been a single SELECT statement.
  If a query contains sub-selects, those are shown as being children of
  the outer SELECT.  For example:

sqlite&gt; EXPLAIN QUERY PLAN SELECT (SELECT b FROM t1 WHERE a=0), (SELECT a FROM t1 WHERE b=t2.c) FROM t2;
|--SCAN TABLE t2 USING COVERING INDEX i4
|--SCALAR SUBQUERY
|  `--SEARCH t1 USING COVERING INDEX i2 (a=?)
`--CORRELATED SCALAR SUBQUERY
   `--SEARCH t1 USING INDEX i3 (b=?)



  The example above contains two "SCALAR" subqueries.  The subqueries
  are SCALAR in the sense that they return a single value - a one-row,
  one-column table.  If the actual query returns more than that, then
  only the first column of the first row is used.

  The first subquery above is constant with respect to the outer query.
  The value for the first subquery can be computed once and then reused
  for each row of the outer SELECT.  The second subquery, however, is
  "CORRELATED".  The value of the second subquery changes depending
  on values in the current row of the outer query.  Hence, the second
  subquery must be run once for each output row in the outer SELECT.


  Unless the flattening optimization is applied, if a subquery appears in
  the FROM clause of a SELECT statement, SQLite can either run the subquery and
  stores the results in a temporary table, or it can run the subquery as a
  co-routine.  The following query is an example of the latter.  The subquery
  is run by a co-routine.  The outer query blocks whenever it needs another
  row of input from the subquery.  Control switches to the co-routine which
  produces the desired output row, then control switches back to the main
  routine which continues processing.  

sqlite&gt; EXPLAIN QUERY PLAN SELECT count(*)
      &gt; FROM (SELECT max(b) AS x FROM t1 GROUP BY a) AS qqq
      &gt; GROUP BY x;
QUERY PLAN
|--CO-ROUTINE qqq
|  `--SCAN t1 USING COVERING INDEX i2
|--SCAN qqqq
`--USE TEMP B-TREE FOR GROUP BY



  
  If the flattening optimization is used on a subquery in the FROM clause
  of a SELECT statement, that effectively merges the subquery into the outer
  query.  The output of EXPLAIN QUERY PLAN reflects this, as in the following
  example:

sqlite&gt; EXPLAIN QUERY PLAN SELECT * FROM (SELECT * FROM t2 WHERE c=1) AS t3, t1;
QUERY PLAN
|--SEARCH t2 USING INDEX i4 (c=?)
`--SCAN t1




  If the content of a subquery might need to be visited more than once, then
  the use of a co-routine is undesirable, as the co-routine would then have to
  compute the data more than once.  And if the subquery cannot be flattened,
  that means the subquery must be manifested into a transient table.

sqlite&gt; SELECT * FROM
      &gt;   (SELECT * FROM t1 WHERE a=1 ORDER BY b LIMIT 2) AS x,
      &gt;   (SELECT * FROM t2 WHERE c=1 ORDER BY d LIMIT 2) AS y;
QUERY PLAN
|--MATERIALIZE x
|  `--SEARCH t1 USING COVERING INDEX i2 (a=?)
|--MATERIALIZE y
|  |--SEARCH t2 USING INDEX i4 (c=?)
|  `--USE TEMP B-TREE FOR ORDER BY
|--SCAN x
`--SCAN y


c2: EXPLAIN QUERY PLAN
c3: 1.4. Compound Queries
c5: eqp.html#compound_queries
c4:
Each component query of a compound query (UNION, UNION ALL, EXCEPT or 
  INTERSECT) is assigned computed separately and is given its own line in
  the EXPLAIN QUERY PLAN output.

sqlite&gt; EXPLAIN QUERY PLAN SELECT a FROM t1 UNION SELECT c FROM t2;
QUERY PLAN
`--COMPOUND QUERY
   |--LEFT-MOST SUBQUERY
   |  `--SCAN t1 USING COVERING INDEX i1
   `--UNION USING TEMP B-TREE
      `--SCAN t2 USING COVERING INDEX i4



  The "USING TEMP B-TREE" clause in the above output indicates that a 
  temporary b-tree structure is used to implement the UNION of the results
  of the two sub-selects.  An alternative method of computing a compound
  is to run each subquery as a co-routine, arrange for their outputs to
  appear in sorted order, and merge the results together.  When the query
  planner chooses this latter approach, the EXPLAIN QUERY PLAN output
  looks like this:

sqlite&gt; EXPLAIN QUERY PLAN SELECT a FROM t1 EXCEPT SELECT d FROM t2 ORDER BY 1;
QUERY PLAN
`--MERGE (EXCEPT)
   |--LEFT
   |  `--SCAN t1 USING COVERING INDEX i1
   `--RIGHT
      |--SCAN t2
      `--USE TEMP B-TREE FOR ORDER BY

This page last modified on  2022-01-08 05:02:57 UTC


c2: The Schema Table
c3: 1. Introduction
c5: schematab.html#introduction
c4:
Every SQLite database contains a single "schema table" that stores the
schema for that database.  The schema for a database is a description of
all of the other tables, indexes, triggers, and views that are
contained within the database.  The schema table looks like this:


CREATE TABLE sqlite_schema(
  type text,
  name text,
  tbl_name text,
  rootpage integer,
  sql text
);


The sqlite_schema table contains one row for each table, index, view,
and trigger (collectively "objects") in the schema, except there
is no entry for the sqlite_schema table itself.  See the
schema storage subsection of the file format documentation for
additional information on how SQLite uses the sqlite_schema table
internally.


c2: The Schema Table
c3: 2. Alternative Names
c5: schematab.html#alternative_names
c4:
The schema table can always be referenced using the name "sqlite_schema",
especially if qualifed by the schema name like 
"main.sqlite_schema" or "temp.sqlite_schema".  But for historical
compatibility, some alternative names are also recognized, including:


 sqlite_master
 sqlite_temp_schema
 sqlite_temp_master



Alternatives (2) and (3) only work for the TEMP database associated
with each database connection, but alternative (1) works anywhere.
For historical reasons, callbacks from the sqlite3_set_authorizer()
interface always refer to the schema table using names (1) or (3).


c2: The Schema Table
c3: 3. Interpretation Of The Schema Table
c5: schematab.html#interpretation_of_the_schema_table
c4:
The meanings of the fields of the schema table are as follows:


type

The sqlite_schema.type column will be one
of the following text strings:  'table', 'index', 'view', or 'trigger'
according to the type of object defined.  The 'table' string is used
for both ordinary and virtual tables.


name

The sqlite_schema.name column will hold the name of the object.
UNIQUE and PRIMARY KEY constraints on tables cause SQLite to create
internal indexes with names of the form "sqlite_autoindex_TABLE_N"
where TABLE is replaced by the name of the table that contains the
constraint and N is an integer beginning with 1 and increasing by one
with each constraint seen in the table definition.
In a WITHOUT ROWID table, there is no sqlite_schema entry for the
PRIMARY KEY, but the "sqlite_autoindex_TABLE_N" name is set aside
for the PRIMARY KEY as if the sqlite_schema entry did exist.  This
will affect the numbering of subsequent UNIQUE constraints.
The "sqlite_autoindex_TABLE_N" name is never allocated for an
INTEGER PRIMARY KEY, either in rowid tables or WITHOUT ROWID tables.



tbl_name

The sqlite_schema.tbl_name column holds the name of a table or view
that the object is associated with.  For a table or view, the
tbl_name column is a copy of the name column.  For an index, the tbl_name
is the name of the table that is indexed.  For a trigger, the tbl_name
column stores the name of the table or view that causes the trigger 
to fire.


rootpage

The sqlite_schema.rootpage column stores the page number of the root
b-tree page for tables and indexes.  For rows that define views, triggers,
and virtual tables, the rootpage column is 0 or NULL.


sql

The sqlite_schema.sql column stores SQL text that describes the
object.  This SQL text is a CREATE TABLE, CREATE VIRTUAL TABLE,
CREATE INDEX,
CREATE VIEW, or CREATE TRIGGER statement that if evaluated against
the database file when it is the main database of a database connection
would recreate the object.  The text is usually a copy of the original
statement used to create the object but with normalizations applied so
that the text conforms to the following rules:


The CREATE, TABLE, VIEW, TRIGGER, and INDEX keywords at the beginning
of the statement are converted to all upper case letters.
The TEMP or TEMPORARY keyword is removed if it occurs after the 
initial CREATE keyword.
Any database name qualifier that occurs prior to the name of the
object being created is removed.
Leading spaces are removed.
All spaces following the first two keywords are converted into a single
space.


The text in the sqlite_schema.sql column is a copy of the original
CREATE statement text that created the object, except normalized as
described above and as modified by subsequent ALTER TABLE statements.
The sqlite_schema.sql is NULL for the internal indexes that are
automatically created by UNIQUE or PRIMARY KEY constraints.


c2: The Schema Table
c3: 4. Creation and Modification Of The Schema Table
c5: schematab.html#creation_and_modification_of_the_schema_table
c4:
SQLite creates the schema table upon database creation and modifies
its content as SQLite users submit DDL statements for execution. There
is no need for users to modify it under normal circumstances, and they
bear the risk of database corruption if they do modify it.


c2: Result and Error Codes
c3: Overview
c5: rescode.html#overview
c4:
Many of the routines in the SQLite C-language Interface return
numeric result codes indicating either success or failure, and 
in the event of a failure, providing some idea of the cause of
the failure.  This document strives to explain what each
of those numeric result codes means.


c2: Result and Error Codes
c3: 1. Result Codes versus Error Codes
c5: rescode.html#result_codes_versus_error_codes
c4:
"Error codes" are a subset of "result codes" that indicate that
something has gone wrong.  There are only a few non-error result
codes:  SQLITE_OK, SQLITE_ROW, and SQLITE_DONE.  The term
"error code" means any result code other than these three.


c2: Result and Error Codes
c3: 2. Primary Result Codes versus Extended Result Codes
c5: rescode.html#primary_result_codes_versus_extended_result_codes
c4:
Result codes are signed 32-bit integers.
The least significant 8 bits of the result code define a broad category
and are called the "primary result code".  More significant bits provide
more detailed information about the error and are called the
"extended result code"


Note that the primary result code is always a part of the extended
result code.  Given a full 32-bit extended result code, the application
can always find the corresponding primary result code merely by extracting
the least significant 8 bits of the extended result code.


All extended result codes are also error codes.  Hence the terms
"extended result code" and "extended error code" are interchangeable.


For historic compatibility, the C-language interfaces return
primary result codes by default.  
The extended result code for the most recent error can be
retrieved using the sqlite3_extended_errcode() interface.
The sqlite3_extended_result_codes() interface can be used to put
a database connection into a mode where it returns the
extended result codes instead of the primary result codes.


c2: Result and Error Codes
c3: 3. Definitions
c5: rescode.html#definitions
c4:
All result codes are integers.
Symbolic names for all result codes are created using
"#define" macros in the sqlite3.h header file.
There are separate sections in the sqlite3.h header file for
the result code definitions and the extended result code definitions.


Primary result code symbolic names are of the form "SQLITE_XXXXXX" where
XXXXXX is a sequence of uppercase alphabetic characters.  Extended
result code names are of the form "SQLITE_XXXXXX_YYYYYYY" where
the XXXXXX part is the corresponding primary result code and the
YYYYYYY is an extension that further classifies the result code.


The names and numeric values for existing result codes are fixed
and unchanging.  However, new result codes, and especially new extended
result codes, might appear in future releases of SQLite.


c2: Result and Error Codes
c3: 4. Primary Result Code List
c5: rescode.html#primary_result_code_list
c4:
The 31 result codes are
   defined in sqlite3.h and are listed in
   alphabetical order below:


SQLITE_ABORT&nbsp;(4)
SQLITE_AUTH&nbsp;(23)
SQLITE_BUSY&nbsp;(5)
SQLITE_CANTOPEN&nbsp;(14)
SQLITE_CONSTRAINT&nbsp;(19)
SQLITE_CORRUPT&nbsp;(11)
SQLITE_DONE&nbsp;(101)
SQLITE_EMPTY&nbsp;(16)
SQLITE_ERROR&nbsp;(1)
SQLITE_FORMAT&nbsp;(24)
SQLITE_FULL&nbsp;(13)
SQLITE_INTERNAL&nbsp;(2)
SQLITE_INTERRUPT&nbsp;(9)
SQLITE_IOERR&nbsp;(10)
SQLITE_LOCKED&nbsp;(6)
SQLITE_MISMATCH&nbsp;(20)
SQLITE_MISUSE&nbsp;(21)
SQLITE_NOLFS&nbsp;(22)
SQLITE_NOMEM&nbsp;(7)
SQLITE_NOTADB&nbsp;(26)
SQLITE_NOTFOUND&nbsp;(12)
SQLITE_NOTICE&nbsp;(27)
SQLITE_OK&nbsp;(0)
SQLITE_PERM&nbsp;(3)
SQLITE_PROTOCOL&nbsp;(15)
SQLITE_RANGE&nbsp;(25)
SQLITE_READONLY&nbsp;(8)
SQLITE_ROW&nbsp;(100)
SQLITE_SCHEMA&nbsp;(17)
SQLITE_TOOBIG&nbsp;(18)
SQLITE_WARNING&nbsp;(28)


c2: Result and Error Codes
c3: 5. Extended Result Code List
c5: rescode.html#extended_result_code_list
c4:
The 74 extended result codes
  are defined in sqlite3.h and are
  listed in alphabetical order below:


SQLITE_ABORT_ROLLBACK&nbsp;(516)
SQLITE_AUTH_USER&nbsp;(279)
SQLITE_BUSY_RECOVERY&nbsp;(261)
SQLITE_BUSY_SNAPSHOT&nbsp;(517)
SQLITE_BUSY_TIMEOUT&nbsp;(773)
SQLITE_CANTOPEN_CONVPATH&nbsp;(1038)
SQLITE_CANTOPEN_DIRTYWAL&nbsp;(1294)
SQLITE_CANTOPEN_FULLPATH&nbsp;(782)
SQLITE_CANTOPEN_ISDIR&nbsp;(526)
SQLITE_CANTOPEN_NOTEMPDIR&nbsp;(270)
SQLITE_CANTOPEN_SYMLINK&nbsp;(1550)
SQLITE_CONSTRAINT_CHECK&nbsp;(275)
SQLITE_CONSTRAINT_COMMITHOOK&nbsp;(531)
SQLITE_CONSTRAINT_DATATYPE&nbsp;(3091)
SQLITE_CONSTRAINT_FOREIGNKEY&nbsp;(787)
SQLITE_CONSTRAINT_FUNCTION&nbsp;(1043)
SQLITE_CONSTRAINT_NOTNULL&nbsp;(1299)
SQLITE_CONSTRAINT_PINNED&nbsp;(2835)
SQLITE_CONSTRAINT_PRIMARYKEY&nbsp;(1555)
SQLITE_CONSTRAINT_ROWID&nbsp;(2579)
SQLITE_CONSTRAINT_TRIGGER&nbsp;(1811)
SQLITE_CONSTRAINT_UNIQUE&nbsp;(2067)
SQLITE_CONSTRAINT_VTAB&nbsp;(2323)
SQLITE_CORRUPT_INDEX&nbsp;(779)
SQLITE_CORRUPT_SEQUENCE&nbsp;(523)
SQLITE_CORRUPT_VTAB&nbsp;(267)
SQLITE_ERROR_MISSING_COLLSEQ&nbsp;(257)
SQLITE_ERROR_RETRY&nbsp;(513)
SQLITE_ERROR_SNAPSHOT&nbsp;(769)
SQLITE_IOERR_ACCESS&nbsp;(3338)
SQLITE_IOERR_AUTH&nbsp;(7178)
SQLITE_IOERR_BEGIN_ATOMIC&nbsp;(7434)
SQLITE_IOERR_BLOCKED&nbsp;(2826)
SQLITE_IOERR_CHECKRESERVEDLOCK&nbsp;(3594)
SQLITE_IOERR_CLOSE&nbsp;(4106)
SQLITE_IOERR_COMMIT_ATOMIC&nbsp;(7690)
SQLITE_IOERR_CONVPATH&nbsp;(6666)
SQLITE_IOERR_CORRUPTFS&nbsp;(8458)
SQLITE_IOERR_DATA&nbsp;(8202)
SQLITE_IOERR_DELETE&nbsp;(2570)
SQLITE_IOERR_DELETE_NOENT&nbsp;(5898)
SQLITE_IOERR_DIR_CLOSE&nbsp;(4362)
SQLITE_IOERR_DIR_FSYNC&nbsp;(1290)
SQLITE_IOERR_FSTAT&nbsp;(1802)
SQLITE_IOERR_FSYNC&nbsp;(1034)
SQLITE_IOERR_GETTEMPPATH&nbsp;(6410)
SQLITE_IOERR_LOCK&nbsp;(3850)
SQLITE_IOERR_MMAP&nbsp;(6154)
SQLITE_IOERR_NOMEM&nbsp;(3082)
SQLITE_IOERR_RDLOCK&nbsp;(2314)
SQLITE_IOERR_READ&nbsp;(266)
SQLITE_IOERR_ROLLBACK_ATOMIC&nbsp;(7946)
SQLITE_IOERR_SEEK&nbsp;(5642)
SQLITE_IOERR_SHMLOCK&nbsp;(5130)
SQLITE_IOERR_SHMMAP&nbsp;(5386)
SQLITE_IOERR_SHMOPEN&nbsp;(4618)
SQLITE_IOERR_SHMSIZE&nbsp;(4874)
SQLITE_IOERR_SHORT_READ&nbsp;(522)
SQLITE_IOERR_TRUNCATE&nbsp;(1546)
SQLITE_IOERR_UNLOCK&nbsp;(2058)
SQLITE_IOERR_VNODE&nbsp;(6922)
SQLITE_IOERR_WRITE&nbsp;(778)
SQLITE_LOCKED_SHAREDCACHE&nbsp;(262)
SQLITE_LOCKED_VTAB&nbsp;(518)
SQLITE_NOTICE_RECOVER_ROLLBACK&nbsp;(539)
SQLITE_NOTICE_RECOVER_WAL&nbsp;(283)
SQLITE_OK_LOAD_PERMANENTLY&nbsp;(256)
SQLITE_READONLY_CANTINIT&nbsp;(1288)
SQLITE_READONLY_CANTLOCK&nbsp;(520)
SQLITE_READONLY_DBMOVED&nbsp;(1032)
SQLITE_READONLY_DIRECTORY&nbsp;(1544)
SQLITE_READONLY_RECOVERY&nbsp;(264)
SQLITE_READONLY_ROLLBACK&nbsp;(776)
SQLITE_WARNING_AUTOINDEX&nbsp;(284)


c2: Result and Error Codes
c3: 6. Result Code Meanings
c5: rescode.html#result_code_meanings
c4:
The meanings for all 105
result code values are shown below,
in numeric order.



(0) SQLITE_OK

  The SQLITE_OK result code means that the operation was successful and
  that there were no errors.  Most other result codes indicate an error.




(1) SQLITE_ERROR

  The SQLITE_ERROR result code is a generic error code that is used when
  no other more specific error code is available.




(2) SQLITE_INTERNAL

  The SQLITE_INTERNAL result code indicates an internal malfunction.
  In a working version of SQLite, an application should never see this
  result code.  If application does encounter this result code, it shows
  that there is a bug in the database engine.
  
  SQLite does not currently generate this result code.
  However, application-defined SQL functions or
  virtual tables, or VFSes, or other extensions might cause this 
  result code to be returned.




(3) SQLITE_PERM

  The SQLITE_PERM result code indicates that the requested access mode
  for a newly created database could not be provided.




(4) SQLITE_ABORT

  The SQLITE_ABORT result code indicates that an operation was aborted
  prior to completion, usually be application request.
  See also: SQLITE_INTERRUPT.
  
  If the callback function to sqlite3_exec() returns non-zero, then
  sqlite3_exec() will return SQLITE_ABORT.
  
  If a ROLLBACK operation occurs on the same database connection as
  a pending read or write, then the pending read or write may fail with
  an SQLITE_ABORT or SQLITE_ABORT_ROLLBACK error.
  
  In addition to being a result code,
  the SQLITE_ABORT value is also used as a conflict resolution mode
  returned from the sqlite3_vtab_on_conflict() interface.




(5) SQLITE_BUSY

  The SQLITE_BUSY result code indicates that the database file could not
  be written (or in some cases read) because of concurrent activity by 
  some other database connection, usually a database connection in a
  separate process.
  
  For example, if process A is in the middle of a large write transaction
  and at the same time process B attempts to start a new write transaction,
  process B will get back an SQLITE_BUSY result because SQLite only supports
  one writer at a time.  Process B will need to wait for process A to finish
  its transaction before starting a new transaction.  The
  sqlite3_busy_timeout() and sqlite3_busy_handler() interfaces and
  the busy_timeout pragma are available to process B to help it deal
  with SQLITE_BUSY errors.
  
  An SQLITE_BUSY error can occur at any point in a transaction: when the
  transaction is first started, during any write or update operations, or
  when the transaction commits.
  To avoid encountering SQLITE_BUSY errors in the middle of a transaction,
  the application can use BEGIN IMMEDIATE instead of just BEGIN to
  start a transaction.  The BEGIN IMMEDIATE command might itself return
  SQLITE_BUSY, but if it succeeds, then SQLite guarantees that no 
  subsequent operations on the same database through the next COMMIT 
  will return SQLITE_BUSY.
  
  See also:  SQLITE_BUSY_RECOVERY and SQLITE_BUSY_SNAPSHOT.
  
  The SQLITE_BUSY result code differs from SQLITE_LOCKED in that
  SQLITE_BUSY indicates a conflict with a
  separate database connection, probably in a separate process,
  whereas SQLITE_LOCKED 
  indicates a conflict within the same database connection (or sometimes
  a database connection with a shared cache).




(6) SQLITE_LOCKED

  The SQLITE_LOCKED result code indicates that a write operation could not
  continue because of a conflict within the same database connection or
  a conflict with a different database connection that uses a shared cache.
  
  For example, a DROP TABLE statement cannot be run while another thread
  is reading from that table on the same database connection because 
  dropping the table would delete the table out from under the concurrent
  reader.
  
  The SQLITE_LOCKED result code differs from SQLITE_BUSY in that
  SQLITE_LOCKED indicates a conflict on the same database connection
  (or on a connection with a shared cache) whereas SQLITE_BUSY indicates
  a conflict with a different database connection, probably in a different
  process.




(7) SQLITE_NOMEM

  The SQLITE_NOMEM result code indicates that SQLite was unable to allocate
  all the memory it needed to complete the operation.  In other words, an
  internal call to sqlite3_malloc() or sqlite3_realloc() has failed in
  a case where the memory being allocated was required in order to continue
  the operation.




(8) SQLITE_READONLY

  The SQLITE_READONLY result code is returned when an attempt is made to 
  alter some data for which the current database connection does not have
  write permission.




(9) SQLITE_INTERRUPT

  The SQLITE_INTERRUPT result code indicates that an operation was
  interrupted by the sqlite3_interrupt() interface.
  See also: SQLITE_ABORT




(10) SQLITE_IOERR

  The SQLITE_IOERR result code says that the operation could not finish
  because the operating system reported an I/O error.
  
  A full disk drive will normally give an SQLITE_FULL error rather than
  an SQLITE_IOERR error.
  
  There are many different extended result codes for I/O errors that
  identify the specific I/O operation that failed.




(11) SQLITE_CORRUPT

  The SQLITE_CORRUPT result code indicates that the database file has
  been corrupted.  See the How To Corrupt Your Database Files for
  further discussion on how corruption can occur.




(12) SQLITE_NOTFOUND

  The SQLITE_NOTFOUND result code is exposed in three ways:
  
  
  SQLITE_NOTFOUND can be returned by the sqlite3_file_control() interface
  to indicate that the file control opcode passed as the third argument
  was not recognized by the underlying VFS.
  
  SQLITE_NOTFOUND can also be returned by the xSetSystemCall() method of
  an sqlite3_vfs object.
  
  SQLITE_NOTFOUND can be returned by sqlite3_vtab_rhs_value() to indicate
  that the right-hand operand of a constraint is not available to the
  xBestIndex method that made the call.
  
  
  The SQLITE_NOTFOUND result code is also used
  internally by the SQLite implementation, but those internal uses are
  not exposed to the application.




(13) SQLITE_FULL

  The SQLITE_FULL result code indicates that a write could not complete
  because the disk is full.  Note that this error can occur when trying
  to write information into the main database file, or it can also
  occur when writing into temporary disk files.
  
  Sometimes applications encounter this error even though there is an
  abundance of primary disk space because the error occurs when writing
  into temporary disk files on a system where temporary files are stored
  on a separate partition with much less space that the primary disk.




(14) SQLITE_CANTOPEN

  The SQLITE_CANTOPEN result code indicates that SQLite was unable to
  open a file.  The file in question might be a primary database file
  or one of several temporary disk files.




(15) SQLITE_PROTOCOL

  The SQLITE_PROTOCOL result code indicates a problem with the file locking
  protocol used by SQLite.  The SQLITE_PROTOCOL error is currently only
  returned when using WAL mode and attempting to start a new transaction.
  There is a race condition that can occur when two separate 
  database connections both try to start a transaction at the same time
  in WAL mode.  The loser of the race backs off and tries again, after
  a brief delay.  If the same connection loses the locking race dozens
  of times over a span of multiple seconds, it will eventually give up and
  return SQLITE_PROTOCOL.  The SQLITE_PROTOCOL error should appear in practice
  very, very rarely, and only when there are many separate processes all
  competing intensely to write to the same database.




(16) SQLITE_EMPTY

  The SQLITE_EMPTY result code is not currently used.




(17) SQLITE_SCHEMA

  The SQLITE_SCHEMA result code indicates that the database schema
  has changed.  This result code can be returned from sqlite3_step() for
  a prepared statement that was generated using sqlite3_prepare() or
  sqlite3_prepare16().  If the database schema was changed by some other
  process in between the time that the statement was prepared and the time
  the statement was run, this error can result.
  
  If a prepared statement is generated from sqlite3_prepare_v2() then
  the statement is automatically re-prepared if the schema changes, up to
  SQLITE_MAX_SCHEMA_RETRY times (default: 50).  The sqlite3_step()
  interface will only return SQLITE_SCHEMA back to the application if 
  the failure persists after these many retries.




(18) SQLITE_TOOBIG

  The SQLITE_TOOBIG error code indicates that a string or BLOB was
  too large.  The default maximum length of a string or BLOB in SQLite is
  1,000,000,000 bytes.  This maximum length can be changed at compile-time
  using the SQLITE_MAX_LENGTH compile-time option, or at run-time using
  the sqlite3_limit(db,SQLITE_LIMIT_LENGTH,...) interface.  The
  SQLITE_TOOBIG error results when SQLite encounters a string or BLOB
  that exceeds the compile-time or run-time limit.
  
  The SQLITE_TOOBIG error code can also result when an oversized SQL
  statement is passed into one of the sqlite3_prepare_v2() interfaces.
  The maximum length of an SQL statement defaults to a much smaller
  value of 1,000,000,000 bytes.  The maximum SQL statement length can be
  set at compile-time using SQLITE_MAX_SQL_LENGTH or at run-time
  using sqlite3_limit(db,SQLITE_LIMIT_SQL_LENGTH,...).




(19) SQLITE_CONSTRAINT

  The SQLITE_CONSTRAINT error code means that an SQL constraint violation
  occurred while trying to process an SQL statement.  Additional information
  about the failed constraint can be found by consulting the
  accompanying error message (returned via sqlite3_errmsg() or
  sqlite3_errmsg16()) or by looking at the extended error code.
  
  The SQLITE_CONSTRAINT code can also be used as the return value from
  the xBestIndex() method of a virtual table implementation.  When
  xBestIndex() returns SQLITE_CONSTRAINT, that indicates that the particular
  combination of inputs submitted to xBestIndex() cannot result in a
  usable query plan and should not be given further consideration.




(20) SQLITE_MISMATCH

  The SQLITE_MISMATCH error code indicates a datatype mismatch.
  
  SQLite is normally very forgiving about mismatches between the type of
  a value and the declared type of the container in which that value is
  to be stored.  For example, SQLite allows the application to store
  a large BLOB in a column with a declared type of BOOLEAN.  But in a few
  cases, SQLite is strict about types.  The SQLITE_MISMATCH error is
  returned in those few cases when the types do not match.
  
  The rowid of a table must be an integer.  Attempt to set the rowid
  to anything other than an integer (or a NULL which will be automatically
  converted into the next available integer rowid) results in an
  SQLITE_MISMATCH error.




(21) SQLITE_MISUSE

  The SQLITE_MISUSE return code might be returned if the application uses
  any SQLite interface in a way that is undefined or unsupported.  For
  example, using a prepared statement after that prepared statement has
  been finalized might result in an SQLITE_MISUSE error.
  
  SQLite tries to detect misuse and report the misuse using this result code.
  However, there is no guarantee that the detection of misuse will be
  successful.  Misuse detection is probabilistic.  Applications should
  never depend on an SQLITE_MISUSE return value.
  
  If SQLite ever returns SQLITE_MISUSE from any interface, that means that
  the application is incorrectly coded and needs to be fixed.  Do not ship
  an application that sometimes returns SQLITE_MISUSE from a standard
  SQLite interface because that application contains potentially serious bugs.




(22) SQLITE_NOLFS

  The SQLITE_NOLFS error can be returned on systems that do not support
  large files when the database grows to be larger than what the filesystem
  can handle.  "NOLFS" stands for "NO Large File Support".




(23) SQLITE_AUTH

  The SQLITE_AUTH error is returned when the
  authorizer callback indicates that an
  SQL statement being prepared is not authorized.




(24) SQLITE_FORMAT

  The SQLITE_FORMAT error code is not currently used by SQLite.




(25) SQLITE_RANGE

  The SQLITE_RANGE error indices that the parameter number argument
  to one of the sqlite3_bind routines or the
  column number in one of the sqlite3_column
  routines is out of range.




(26) SQLITE_NOTADB

  When attempting to open a file, the SQLITE_NOTADB error indicates that
  the file being opened does not appear to be an SQLite database file.




(27) SQLITE_NOTICE

  The SQLITE_NOTICE result code is not returned by any C/C++ interface.
  However, SQLITE_NOTICE (or rather one of its extended error codes)
  is sometimes used as the first argument in an sqlite3_log() callback
  to indicate that an unusual operation is taking place.




(28) SQLITE_WARNING

  The SQLITE_WARNING result code is not returned by any C/C++ interface.
  However, SQLITE_WARNING (or rather one of its extended error codes)
  is sometimes used as the first argument in an sqlite3_log() callback
  to indicate that an unusual and possibly ill-advised operation is
  taking place.




(100) SQLITE_ROW

  The SQLITE_ROW result code returned by
  sqlite3_step() indicates that another row of output is available.




(101) SQLITE_DONE

  The SQLITE_DONE result code indicates that an operation has completed.
  The SQLITE_DONE result code is most commonly seen as a return value
  from sqlite3_step() indicating that the SQL statement has run to
  completion.  But SQLITE_DONE can also be returned by other multi-step
  interfaces such as sqlite3_backup_step().




(256) SQLITE_OK_LOAD_PERMANENTLY

  The sqlite3_load_extension() interface loads an
  extension into a single
  database connection.  The default behavior is for that extension to be
  automatically unloaded when the database connection closes.  However,
  if the extension entry point returns SQLITE_OK_LOAD_PERMANENTLY instead
  of SQLITE_OK, then the extension remains loaded into the process address
  space after the database connection closes.  In other words, the
  xDlClose methods of the sqlite3_vfs object is not called for the
  extension when the database connection closes.
  
  The SQLITE_OK_LOAD_PERMANENTLY return code is useful to
  loadable extensions that register new VFSes, for example.




(257) SQLITE_ERROR_MISSING_COLLSEQ

  The SQLITE_ERROR_MISSING_COLLSEQ result code means that an SQL
  statement could not be prepared because a collating sequence named
  in that SQL statement could not be located.
  
  Sometimes when this error code is encountered, the
  sqlite3_prepare_v2() routine will convert the error into
  SQLITE_ERROR_RETRY and try again to prepare the SQL statement
  using a different query plan that does not require the use of
  the unknown collating sequence.




(261) SQLITE_BUSY_RECOVERY

  The SQLITE_BUSY_RECOVERY error code is an extended error code
  for SQLITE_BUSY that indicates that an operation could not continue
  because another process is busy recovering a WAL mode database file
  following a crash.  The SQLITE_BUSY_RECOVERY error code only occurs
  on WAL mode databases.




(262) SQLITE_LOCKED_SHAREDCACHE

  The SQLITE_LOCKED_SHAREDCACHE result code indicates that access to
  an SQLite data record is blocked by another database connection that
  is using the same record in shared cache mode.  When two or more
  database connections share the same cache and one of the connections is
  in the middle of modifying a record in that cache, then other connections
  are blocked from accessing that data while the modifications are on-going
  in order to prevent the readers from seeing a corrupt or partially
  completed change.




(264) SQLITE_READONLY_RECOVERY

  The SQLITE_READONLY_RECOVERY error code is an extended error code
  for SQLITE_READONLY.  The SQLITE_READONLY_RECOVERY error code indicates
  that a WAL mode database cannot be opened because the database file
  needs to be recovered and recovery requires write access but only
  read access is available.




(266) SQLITE_IOERR_READ

  The SQLITE_IOERR_READ error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to read from a file on disk.  This error might result
  from a hardware malfunction or because a filesystem came unmounted
  while the file was open.




(267) SQLITE_CORRUPT_VTAB

  The SQLITE_CORRUPT_VTAB error code is an extended error code
  for SQLITE_CORRUPT used by virtual tables.  A virtual table might
  return SQLITE_CORRUPT_VTAB to indicate that content in the virtual table
  is corrupt.




(270) SQLITE_CANTOPEN_NOTEMPDIR

  The SQLITE_CANTOPEN_NOTEMPDIR error code is no longer used.




(275) SQLITE_CONSTRAINT_CHECK

  The SQLITE_CONSTRAINT_CHECK error code is an extended error code
  for SQLITE_CONSTRAINT indicating that a CHECK constraint failed.




(279) SQLITE_AUTH_USER

  The SQLITE_AUTH_USER error code is an extended error code
  for SQLITE_AUTH indicating that an operation was attempted on a
  database for which the logged in user lacks sufficient authorization.




(283) SQLITE_NOTICE_RECOVER_WAL

  The SQLITE_NOTICE_RECOVER_WAL result code is
  passed to the callback of
  sqlite3_log() when a WAL mode database file is recovered.




(284) SQLITE_WARNING_AUTOINDEX

  The SQLITE_WARNING_AUTOINDEX result code is
  passed to the callback of
  sqlite3_log() whenever automatic indexing is used.
  This can serve as a warning to application designers that the
  database might benefit from additional indexes.




(513) SQLITE_ERROR_RETRY

  The SQLITE_ERROR_RETRY is used internally to provoke sqlite3_prepare_v2()
  (or one of its sibling routines for creating prepared statements) to
  try again to prepare a statement that failed with an error on the
  previous attempt.




(516) SQLITE_ABORT_ROLLBACK

  The SQLITE_ABORT_ROLLBACK error code is an extended error code
  for SQLITE_ABORT indicating that an SQL statement aborted because
  the transaction that was active when the SQL statement first started
  was rolled back.  Pending write operations always fail with this error
  when a rollback occurs.  A ROLLBACK will cause a pending read operation
  to fail only if the schema was changed within the transaction being rolled
  back.




(517) SQLITE_BUSY_SNAPSHOT

  The SQLITE_BUSY_SNAPSHOT error code is an extended error code
  for SQLITE_BUSY that occurs on WAL mode databases when a database
  connection tries to promote a read transaction into a write transaction
  but finds that another database connection has already written to the
  database and thus invalidated prior reads.
  
  The following scenario illustrates how an SQLITE_BUSY_SNAPSHOT error
  might arise:
  
   Process A starts a read transaction on the database and does one
       or more SELECT statement.  Process A keeps the transaction open.
   Process B updates the database, changing values previous read by
       process A.
   Process A now tries to write to the database.  But process A's view
       of the database content is now obsolete because process B has
       modified the database file after process A read from it.  Hence
       process A gets an SQLITE_BUSY_SNAPSHOT error.
  




(518) SQLITE_LOCKED_VTAB

  The SQLITE_LOCKED_VTAB result code is not used by the SQLite core, but
  it is available for use by extensions.  Virtual table implementations
  can return this result code to indicate that they cannot complete the
  current operation because of locks held by other threads or processes.
  
  The R-Tree extension returns this result code when an attempt is made
  to update the R-Tree while another prepared statement is actively reading
  the R-Tree.  The update cannot proceed because any change to an R-Tree
  might involve reshuffling and rebalancing of nodes, which would disrupt
  read cursors, causing some rows to be repeated and other rows to be
  omitted.




(520) SQLITE_READONLY_CANTLOCK

  The SQLITE_READONLY_CANTLOCK error code is an extended error code
  for SQLITE_READONLY.  The SQLITE_READONLY_CANTLOCK error code indicates
  that SQLite is unable to obtain a read lock on a WAL mode database
  because the shared-memory file associated with that database is read-only.




(522) SQLITE_IOERR_SHORT_READ

  The SQLITE_IOERR_SHORT_READ error code is an extended error code
  for SQLITE_IOERR indicating that a read attempt in the VFS layer
  was unable to obtain as many bytes as was requested.  This might be
  due to a truncated file.




(523) SQLITE_CORRUPT_SEQUENCE

  The SQLITE_CORRUPT_SEQUENCE result code means that the schema of
  the sqlite_sequence table is corrupt.  The sqlite_sequence table
  is used to help implement the AUTOINCREMENT feature.  The
  sqlite_sequence table should have the following format:
  
  CREATE TABLE sqlite_sequence(name,seq);
  
  If SQLite discovers that the sqlite_sequence table has any other
  format, it returns the SQLITE_CORRUPT_SEQUENCE error.




(526) SQLITE_CANTOPEN_ISDIR

  The SQLITE_CANTOPEN_ISDIR error code is an extended error code
  for SQLITE_CANTOPEN indicating that a file open operation failed because
  the file is really a directory.




(531) SQLITE_CONSTRAINT_COMMITHOOK

  The SQLITE_CONSTRAINT_COMMITHOOK error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a
  commit hook callback returned non-zero that thus
  caused the SQL statement to be rolled back.




(539) SQLITE_NOTICE_RECOVER_ROLLBACK

  The SQLITE_NOTICE_RECOVER_ROLLBACK result code is
  passed to the callback of
  sqlite3_log() when a hot journal is rolled back.




(769) SQLITE_ERROR_SNAPSHOT

  The SQLITE_ERROR_SNAPSHOT result code might be returned when attempting
  to start a read transaction on an historical version of the database
  by using the sqlite3_snapshot_open() interface.  If the historical
  snapshot is no longer available, then the read transaction will fail
  with the SQLITE_ERROR_SNAPSHOT.  This error code is only possible if
  SQLite is compiled with -DSQLITE_ENABLE_SNAPSHOT.




(773) SQLITE_BUSY_TIMEOUT

  The SQLITE_BUSY_TIMEOUT error code indicates that a blocking Posix
  advisory file lock request in the VFS layer failed due to a timeout.
  Blocking Posix advisory locks are only
  available as a proprietary SQLite extension and even then are only
  supported if SQLite is compiled with the SQLITE_EANBLE_SETLK_TIMEOUT
  compile-time option.




(776) SQLITE_READONLY_ROLLBACK

  The SQLITE_READONLY_ROLLBACK error code is an extended error code
  for SQLITE_READONLY.  The SQLITE_READONLY_ROLLBACK error code indicates
  that a database cannot be opened because it has a hot journal that
  needs to be rolled back but cannot because the database is readonly.




(778) SQLITE_IOERR_WRITE

  The SQLITE_IOERR_WRITE error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to write into a file on disk.  This error might result
  from a hardware malfunction or because a filesystem came unmounted
  while the file was open.  This error should not occur if the filesystem
  is full as there is a separate error code (SQLITE_FULL) for that purpose.




(779) SQLITE_CORRUPT_INDEX

  The SQLITE_CORRUPT_INDEX result code means that SQLite detected
  an entry is or was missing from an index.  This is a special case of
  the SQLITE_CORRUPT error code that suggests that the problem might
  be resolved by running the REINDEX command, assuming no other
  problems exist elsewhere in the database file.




(782) SQLITE_CANTOPEN_FULLPATH

  The SQLITE_CANTOPEN_FULLPATH error code is an extended error code
  for SQLITE_CANTOPEN indicating that a file open operation failed because
  the operating system was unable to convert the filename into a full pathname.




(787) SQLITE_CONSTRAINT_FOREIGNKEY

  The SQLITE_CONSTRAINT_FOREIGNKEY error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a foreign key constraint failed.




(1032) SQLITE_READONLY_DBMOVED

  The SQLITE_READONLY_DBMOVED error code is an extended error code
  for SQLITE_READONLY.  The SQLITE_READONLY_DBMOVED error code indicates
  that a database cannot be modified because the database file has been
  moved since it was opened, and so any attempt to modify the database
  might result in database corruption if the processes crashes because the
  rollback journal would not be correctly named.




(1034) SQLITE_IOERR_FSYNC

  The SQLITE_IOERR_FSYNC error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to flush previously written content out of OS and/or
  disk-control buffers and into persistent storage.  In other words,
  this code indicates a problem with the fsync() system call in unix
  or the FlushFileBuffers() system call in windows.




(1038) SQLITE_CANTOPEN_CONVPATH

  The SQLITE_CANTOPEN_CONVPATH error code is an extended error code
  for SQLITE_CANTOPEN used only by Cygwin VFS and indicating that
  the cygwin_conv_path() system call failed while trying to open a file.
  See also: SQLITE_IOERR_CONVPATH




(1043) SQLITE_CONSTRAINT_FUNCTION

  The SQLITE_CONSTRAINT_FUNCTION error code is not currently used
  by the SQLite core.  However, this error code is available for use
  by extension functions.




(1288) SQLITE_READONLY_CANTINIT

  The SQLITE_READONLY_CANTINIT result code originates in the xShmMap method
  of a VFS to indicate that the shared memory region used by WAL mode
  exists buts its content is unreliable and unusable by the current process
  since the current process does not have write permission on the shared
  memory region.  (The shared memory region for WAL mode is normally a
  file with a "-wal" suffix that is mmapped into the process space.  If
  the current process does not have write permission on that file, then it
  cannot write into shared memory.)
  
  Higher level logic within SQLite will normally intercept the error code
  and create a temporary in-memory shared memory region so that the current
  process can at least read the content of the database.  This result code
  should not reach the application interface layer.




(1290) SQLITE_IOERR_DIR_FSYNC

  The SQLITE_IOERR_DIR_FSYNC error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to invoke fsync() on a directory.  The unix VFS attempts
  to fsync() directories after creating or deleting certain files to
  ensure that those files will still appear in the filesystem following
  a power loss or system crash.  This error code indicates a problem
  attempting to perform that fsync().




(1294) SQLITE_CANTOPEN_DIRTYWAL

  The SQLITE_CANTOPEN_DIRTYWAL result code is not used at this time.




(1299) SQLITE_CONSTRAINT_NOTNULL

  The SQLITE_CONSTRAINT_NOTNULL error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a NOT NULL constraint failed.




(1544) SQLITE_READONLY_DIRECTORY

  The SQLITE_READONLY_DIRECTORY result code indicates that the database
  is read-only because process does not have permission to create
  a journal file in the same directory as the database and the creation of
  a journal file is a prerequisite for writing.




(1546) SQLITE_IOERR_TRUNCATE

  The SQLITE_IOERR_TRUNCATE error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to truncate a file to a smaller size.




(1550) SQLITE_CANTOPEN_SYMLINK

  The SQLITE_CANTOPEN_SYMLINK result code is returned by the
  sqlite3_open() interface and its siblings when the
  SQLITE_OPEN_NOFOLLOW flag is used and the database file is
  a symbolic link.




(1555) SQLITE_CONSTRAINT_PRIMARYKEY

  The SQLITE_CONSTRAINT_PRIMARYKEY error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a PRIMARY KEY constraint failed.




(1802) SQLITE_IOERR_FSTAT

  The SQLITE_IOERR_FSTAT error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the VFS layer
  while trying to invoke fstat() (or the equivalent) on a file in order
  to determine information such as the file size or access permissions.




(1811) SQLITE_CONSTRAINT_TRIGGER

  The SQLITE_CONSTRAINT_TRIGGER error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a RAISE function within
  a trigger fired, causing the SQL statement to abort.




(2058) SQLITE_IOERR_UNLOCK

  The SQLITE_IOERR_UNLOCK error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within xUnlock method on the sqlite3_io_methods object.




(2067) SQLITE_CONSTRAINT_UNIQUE

  The SQLITE_CONSTRAINT_UNIQUE error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a UNIQUE constraint failed.




(2314) SQLITE_IOERR_RDLOCK

  The SQLITE_IOERR_RDLOCK error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within xLock method on the sqlite3_io_methods object while trying
  to obtain a read lock.




(2323) SQLITE_CONSTRAINT_VTAB

  The SQLITE_CONSTRAINT_VTAB error code is not currently used
  by the SQLite core.  However, this error code is available for use
  by application-defined virtual tables.




(2570) SQLITE_IOERR_DELETE

  The SQLITE_IOERR_DELETE error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within xDelete method on the sqlite3_vfs object.




(2579) SQLITE_CONSTRAINT_ROWID

  The SQLITE_CONSTRAINT_ROWID error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that a rowid is not unique.




(2826) SQLITE_IOERR_BLOCKED

  The SQLITE_IOERR_BLOCKED error code is no longer used.




(2835) SQLITE_CONSTRAINT_PINNED

  The SQLITE_CONSTRAINT_PINNED error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that an UPDATE trigger attempted
  do delete the row that was being updated in the middle of the update.




(3082) SQLITE_IOERR_NOMEM

  The SQLITE_IOERR_NOMEM error code is sometimes returned by the VFS
  layer to indicate that an operation could not be completed due to the
  inability to allocate sufficient memory.  This error code is normally
  converted into SQLITE_NOMEM by the higher layers of SQLite before
  being returned to the application.




(3091) SQLITE_CONSTRAINT_DATATYPE

  The SQLITE_CONSTRAINT_DATATYPE error code
  is an extended error code
  for SQLITE_CONSTRAINT indicating that an insert or update attempted
  to store a value inconsistent with the column's declared type
  in a table defined as STRICT.




(3338) SQLITE_IOERR_ACCESS

  The SQLITE_IOERR_ACCESS error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xAccess method on the sqlite3_vfs object.




(3594) SQLITE_IOERR_CHECKRESERVEDLOCK

  The SQLITE_IOERR_CHECKRESERVEDLOCK error code is
  an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xCheckReservedLock method on the sqlite3_io_methods object.




(3850) SQLITE_IOERR_LOCK

  The SQLITE_IOERR_LOCK error code is an extended error code
  for SQLITE_IOERR indicating an I/O error in the
  advisory file locking logic.
  Usually an SQLITE_IOERR_LOCK error indicates a problem obtaining
  a PENDING lock.  However it can also indicate miscellaneous
  locking errors on some of the specialized VFSes used on Macs.




(4106) SQLITE_IOERR_CLOSE

  The SQLITE_IOERR_CLOSE error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xClose method on the sqlite3_io_methods object.




(4362) SQLITE_IOERR_DIR_CLOSE

  The SQLITE_IOERR_DIR_CLOSE error code is no longer used.




(4618) SQLITE_IOERR_SHMOPEN

  The SQLITE_IOERR_SHMOPEN error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xShmMap method on the sqlite3_io_methods object
  while trying to open a new shared memory segment.




(4874) SQLITE_IOERR_SHMSIZE

  The SQLITE_IOERR_SHMSIZE error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xShmMap method on the sqlite3_io_methods object
  while trying to enlarge a "shm" file as part of
  WAL mode transaction processing.  This error may indicate that
  the underlying filesystem volume is out of space.




(5130) SQLITE_IOERR_SHMLOCK

  The SQLITE_IOERR_SHMLOCK error code is no longer used.




(5386) SQLITE_IOERR_SHMMAP

  The SQLITE_IOERR_SHMMAP error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xShmMap method on the sqlite3_io_methods object
  while trying to map a shared memory segment into the process address space.




(5642) SQLITE_IOERR_SEEK

  The SQLITE_IOERR_SEEK error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xRead or xWrite methods on the sqlite3_io_methods object
  while trying to seek a file descriptor to the beginning point of the
  file where the read or write is to occur.




(5898) SQLITE_IOERR_DELETE_NOENT

  The SQLITE_IOERR_DELETE_NOENT error code
  is an extended error code
  for SQLITE_IOERR indicating that the
  xDelete method on the sqlite3_vfs object failed because the
  file being deleted does not exist.




(6154) SQLITE_IOERR_MMAP

  The SQLITE_IOERR_MMAP error code is an extended error code
  for SQLITE_IOERR indicating an I/O error
  within the xFetch or xUnfetch methods on the sqlite3_io_methods object
  while trying to map or unmap part of the database file into the
  process address space.




(6410) SQLITE_IOERR_GETTEMPPATH

  The SQLITE_IOERR_GETTEMPPATH error code is an extended error code
  for SQLITE_IOERR indicating that the VFS is unable to determine
  a suitable directory in which to place temporary files.




(6666) SQLITE_IOERR_CONVPATH

  The SQLITE_IOERR_CONVPATH error code is an extended error code
  for SQLITE_IOERR used only by Cygwin VFS and indicating that
  the cygwin_conv_path() system call failed.
  See also: SQLITE_CANTOPEN_CONVPATH




(6922) SQLITE_IOERR_VNODE

  The SQLITE_IOERR_VNODE error code is a code reserved for use
  by extensions.  It is not used by the SQLite core.




(7178) SQLITE_IOERR_AUTH

  The SQLITE_IOERR_AUTH error code is a code reserved for use
  by extensions.  It is not used by the SQLite core.




(7434) SQLITE_IOERR_BEGIN_ATOMIC

  The SQLITE_IOERR_BEGIN_ATOMIC error code indicates that the
  underlying operating system reported and error on the
  SQLITE_FCNTL_BEGIN_ATOMIC_WRITE file-control.  This only comes
  up when SQLITE_ENABLE_ATOMIC_WRITE is enabled and the database
  is hosted on a filesystem that supports atomic writes.




(7690) SQLITE_IOERR_COMMIT_ATOMIC

  The SQLITE_IOERR_COMMIT_ATOMIC error code indicates that the
  underlying operating system reported and error on the
  SQLITE_FCNTL_COMMIT_ATOMIC_WRITE file-control.  This only comes
  up when SQLITE_ENABLE_ATOMIC_WRITE is enabled and the database
  is hosted on a filesystem that supports atomic writes.




(7946) SQLITE_IOERR_ROLLBACK_ATOMIC

  The SQLITE_IOERR_ROLLBACK_ATOMIC error code indicates that the
  underlying operating system reported and error on the
  SQLITE_FCNTL_ROLLBACK_ATOMIC_WRITE file-control.  This only comes
  up when SQLITE_ENABLE_ATOMIC_WRITE is enabled and the database
  is hosted on a filesystem that supports atomic writes.




(8202) SQLITE_IOERR_DATA

  The SQLITE_IOERR_DATA error code is an extended error code
  for SQLITE_IOERR used only by checksum VFS shim to indicate that
  the checksum on a page of the database file is incorrect.




(8458) SQLITE_IOERR_CORRUPTFS

  The SQLITE_IOERR_CORRUPTFS error code is an extended error code
  for SQLITE_IOERR used only by a VFS to indicate that a seek or read
  failure was due to the request not falling within the file's boundary
  rather than an ordinary device failure. This often indicates a
  corrupt filesystem.



This page last modified on  2023-09-14 14:42:56 UTC


c2: The Checksum VFS Shim
c3: 1. Overview
c5: cksumvfs.html#overview
c4:
The checksum VFS extension is a VFS shim that adds an 8-byte
checksum to the end of every page in an SQLite database.  The checksum
is added as each page is written and verified as each page is read.
The checksum is intended to help detect database corruption caused by
random bit-flips in the mass storage device.

The checksum VFS extension requires 
SQLite version 3.32.0 (2020-05-22) or later.  It will not
work with earlier versions of SQLite.


c2: The Checksum VFS Shim
c3: 2. Compiling
c5: cksumvfs.html#compiling
c4:
The checksum VFS module is a loadable extension.  It is not
included in the amalgamation.  It must be added to SQLite
either at compile-time or at run-time.  The source code to
the checksum VFS module is in the
ext/misc/cksumvfs.c
source file in the
SQLite source tree.

To build the checksum VFS module into a run-time loadable
extension, use commands similar to the following:


(linux) &rarr; gcc -fPIC -shared cksumvfs.c -o cksumvfs.so
(mac) &rarr; clang -fPIC -dynamiclib cksumvfs.c -o cksumvfs.dylib
(windows) &rarr; cl cksumvfs.c -link -dll -out:cksumvfs.dll


You may want to add additional compiler options, of course,
according to the needs of your project.

To statically link this extension with your product,
compile it like any other C-language module but add the
"-DSQLITE_CKSUMVFS_STATIC" option so that this module knows that
it is being statically linked rather than dynamically linked.


c2: The Checksum VFS Shim
c3: 3. Loading
c5: cksumvfs.html#loading
c4:
To load this extension as a shared library, you first have to
bring up a dummy SQLite database connection to use as the argument
to the sqlite3_load_extension() API call.  Then you invoke the
sqlite3_load_extension() API and shutdown the dummy database
connection.  All subsequent database connections that are opened
will include this extension.  For example:

sqlite3 *db;
sqlite3_open(":memory:", &db);
sqlite3_load_extension(db, "./cksumvfs");
sqlite3_close(db);


If this extension is compiled with -DSQLITE_CKSUMVFS_STATIC and
statically linked against the application, initialize it using
a single API call as follows:

sqlite3_cksumvfs_init();


Cksumvfs is a VFS shim. When loaded, "cksmvfs" becomes the new
default VFS, and it uses the prior default VFS as the next VFS
down in the stack.  This is normally what you want.  However, in
complex situations where multiple VFS shims are being loaded,
it might be important to ensure that cksumvfs is loaded in the
correct order so that it sequences itself into the default VFS
Shim stack in the right order.


c2: The Checksum VFS Shim
c3: 4. Usage
c5: cksumvfs.html#usage
c4:
Open database connections using the sqlite3_open() or 
sqlite3_open_v2() interfaces, as normal.  Ordinary database files
(without a checksum) will operate normally.  Databases with 
checksums will return an SQLITE_IOERR_DATA error if a page is
encountered that contains an invalid checksum.

Checksumming only works on databases that have a reserve bytes
value of exactly 8.  The default value for reserve-bytes is 0.
Hence, newly created database files will omit the checksum by
default.  To create a database that includes a checksum, change
the reserve-bytes value to 8 by running code similar to this:

int n = 8;
sqlite3_file_control(db, 0, SQLITE_FCNTL_RESERVE_BYTES, &n);


If you do this immediately after creating a new database file,
before anything else has been written into the file, then that
might be all that you need to do.  Otherwise, the API call
above should be followed by:

sqlite3_exec(db, "VACUUM", 0, 0, 0);


It never hurts to run the VACUUM, even if you don't need it.
If the database is in WAL mode, you should shutdown and
reopen all database connections before continuing.

From the CLI, use the ".filectrl reserve_bytes 8" command, 
followed by "VACUUM;".

Note that SQLite allows the number of reserve-bytes to be
increased but not decreased.  So if a database file already
has a reserve-bytes value greater than 8, there is no way to
activate checksumming on that database, other than to dump
and restore the database file.  Note also that other extensions
might also make use of the reserve-bytes.  Checksumming will
be incompatible with those other extensions.


c2: The Checksum VFS Shim
c3: 5. Verification Of Checksums
c5: cksumvfs.html#verification_of_checksums
c4:
If any checksum is incorrect, the "PRAGMA quick_check" command
will find it.  To verify that checksums are actually enabled
and running, use SQL like the following:

SELECT count(*), verify_checksum(data)
  FROM sqlite_dbpage
 GROUP BY 2;


There are three possible outputs from the verify_checksum()
function: 1, 0, and NULL.  1 is returned if the checksum is
correct.  0 is returned if the checksum is incorrect.  NULL
is returned if the page is unreadable.  If checksumming is
enabled, the read will fail if the checksum is wrong, so the
usual result from verify_checksum() on a bad checksum is NULL.

If everything is OK, the query above should return a single
row where the second column is 1.  Any other result indicates
either that there is a checksum error, or checksum validation
is disabled.


c2: The Checksum VFS Shim
c3: 6. Controlling Checksum Verification
c5: cksumvfs.html#controlling_checksum_verification
c4:
The cksumvfs extension implements a new PRAGMA statement that can
be used to disable, re-enable, or query the status of checksum
verification:

PRAGMA checksum_verification;          -- query status
PRAGMA checksum_verification=OFF;      -- disable verification
PRAGMA checksum_verification=ON;       -- re-enable verification


The "checksum_verification" pragma will return "1" (true) or "0"
(false) if checksum verification is enabled or disabled, respectively.
"Verification" in this context means the feature that causes
SQLITE_IOERR_DATA errors if a checksum mismatch is detected while
reading.  Checksums are always kept up-to-date as long as the
reserve bytes value of the database is 8, regardless of the setting
of this pragma.  Checksum verification can be disabled (for example)
to do forensic analysis of a database that has previously reported
a checksum error.

The "checksum_verification" pragma will always respond with "0" if
the database file does not have a reserve bytes value of 8.  The
pragma will return no rows at all if the cksumvfs extension is
not loaded.
This page last modified on  2022-01-08 05:02:57 UTC


c2: SQLite's Built-in printf()
c3: 1. Overview
c5: printf.html#overview
c4:
SQLite contains its own implementation of the string formatting routine "printf()",
accessible via the following interfaces:


 format() &rarr; an SQL function returning the formatted string
 sqlite3_mprintf() &rarr; Store the formatted string in memory obtained
     sqlite3_malloc64().
 sqlite3_snprintf() &rarr; Store the formatted string in a static buffer
 sqlite3_str_appendf() &rarr; Append formatted text to a dynamic string
 sqlite3_vmprintf() &rarr; Varargs version of sqlite3_mprintf()
 sqlite3_vsnprintf() &rarr; Varargs version of sqlite3_snprintf()
 sqlite3_str_vappendf() &rarr; Varargs version of sqlite3_str_appendf()


The same core string formatter is also used internally by SQLite.


c2: SQLite's Built-in printf()
c3: 1.1. Advantages
c5: printf.html#advantages
c4:
Why does SQLite have its own private built-in printf() implementation?
Why not use the printf() implementation from the standard C library?
Several reasons:




By using its own built-in implementation, SQLite guarantees that the
output will be the same on all platforms and in all LOCALEs.
This is important for consistency and for testing.  It would be problematic
if one machine gave an answer of "5.25e+08" and another gave an answer
of "5.250e+008".  Both answers are correct, but it is better when SQLite
always gives the same answer.


We know of no way to use the standard library printf() C interface to
implement the format() SQL function feature of SQLite.  The built-in
printf() implementation can be easily adapted to that task, however.


The printf() in SQLite supports new non-standard substitution
types (%q, %Q, %w, and %z), and enhanced substitution
behavior (%s and %z) that are useful both internally to SQLite
and to applications using SQLite.
Standard library printf()s cannot normally be extended in this way.


Via the sqlite3_mprintf() and sqlite3_vmprintf() interfaces,
the built-in printf() implementation supports the ability to render an
arbitrary-length string into a memory buffer obtained from sqlite3_malloc64().
This is safer and less error prone than trying to precompute an upper size
limit on the result string, allocate an appropriately sized buffer, and
then calling snprintf().


The SQLite-specific printf() supports a new flag (!) called the
"alternate-form-2" flag.  The alternate-form-2 flag changes the processing
of floating-point conversions in subtle ways so that the output is always
an SQL-compatible text representation of a floating-point number - something
that is not possible to achieve with standard-library printf().  For
string substitutions, the alternate-form-2 flag causes the width and
precision to be measured in characters instead of bytes, which simplifies
processing of strings containing multi-byte UTF8 characters.


The built-in SQLite has compile-time options such as
SQLITE_PRINTF_PRECISION_LIMIT that provide defense against 
denial-of-service attacks for application that expose the
printf() functionality to untrusted users.


Using a built-in printf() implementation means that SQLite has one
fewer dependency on the host environment, making it more portable.


c2: SQLite's Built-in printf()
c3: 1.2. Disadvantages
c5: printf.html#disadvantages
c4:
In fairness, having a built-in implementation of printf() also comes with
some disadvantages.  To wit:



The built-in printf() implementation uses extra code space 
(about 7800 bytes on GCC 5.4 with -Os).


The floating-point to text conversion subfunction for the built-in printf()
is limited in precision to 16 significant digits or 26 significant digits
if the "!" alternate-form-2 flag is used.
Every IEEE-754 double can be represented exactly as a decimal
value, but for many doubles the exact decimal representation requires
more than 16 or 26 significant digits.
The SQLite printf() function only renders the first 16 or 26 significant digits
because that can be done efficient and because 16 decimal digits it is sufficient
to distinguish every possible double value.  Use the decimal extension to get
the exact decimal equivalent of double value for the rare cases where that is
required.


The order of the buffer pointer and buffer size parameters in the built-in
snprintf() implementation is reversed from the order used in standard-library
implementations.


The built-in printf() implementation does not handle posix positional referencing
modifiers that allow the order of arguments to printf() to be different from the
order of the %-substitutions.  In the built-in printf(), the order of the arguments
must exactly match the order of the %-substitutions.



In spite of the disadvantages, the developers believe that having a built-in
printf() implementation inside of SQLite is a net positive.


c2: SQLite's Built-in printf()
c3: 2. Formatting Details
c5: printf.html#formatting_details
c4:
The format string for printf() is a template for the generated
string.  Substitutions are made whenever a "%" character appears in
the format string.  The "%" is followed by one or more additional
characters that describe the substitution.  Each substitution has
the following format:


%&#91;flags&#93;&#91;width&#93;&#91;.precision&#93;&#91;length&#93;type


All substitutions begin with a single "%" and end with a single type character.
The other elements of the substitution are optional.

To include a single "%" character in the output, put two consecutive
"%" characters in the template.


c2: SQLite's Built-in printf()
c3: 2.1. Substitution Types
c5: printf.html#substitution_types
c4:
The following chart shows the substitution types supported by SQLite:




Substitution TypeMeaning

%
Two "%" characters in a row are translated into a single "%" in the output,
    without substituting any values.

d, i
The argument is a signed integer which is displayed in decimal.

u
The argument is an unsigned integer which is displayed in decimal.

f
The argument is a double which is displayed in decimal.

e, E
The argument is a double which is displayed in exponential notation.
    The exponent character is 'e' or 'E' depending on the type.

g, G
The argument is a double which is displayed in either normal decimal
    notation or if the exponent is not close to zero, in exponential
    notation.

x, X
The argument is an integer which is displayed in hexadecimal.
    Lower-case hexadecimal is used for %x and upper-case is used
    for %X

o
The argument is an integer which is displayed in octal.

s, z



    The argument is either a zero-terminated string that is displayed,
    or a null pointer which is treated as an empty string.  For
    the %z type in C-language interfaces, sqlite3_free() is invoked
    on the string after it has been copied into the output. The %s and %z
    substitutions are identical for the SQL printf() function, with
    a NULL argument treated as an empty string.
    The %s substitution is universal among printf functions, but
    the %z substitution and safe treatment of null pointers
    are SQLite enhancements, not found in other
    printf() implementations.

c
For the C-language interfaces, the argument is an integer which
    is interpreted as a character.  For the format() SQL function the
    argument is a string from which the first character is extracted and
    displayed.

p
The argument is a pointer which is displayed as a hexadecimal address.
    Since the SQL language has no concept of a pointer, the %p substitution
    for the format() SQL function works like %x.

n
The argument is a pointer to an integer.  Nothing is displayed for
    this substitution type.  Instead, the integer to which the argument
    points is overwritten with the number of characters in the generated
    string that result from all format symbols to the left of the %n.

q, Q


    The argument is a zero-terminated string.  The string is printed with
    all single quote (') characters doubled so that the string can safely
    appear inside an SQL string literal.  The %Q substitution type also
    puts single-quotes on both ends of the substituted string.
    If the argument
    to %Q is a null pointer then the output is an unquoted "NULL".  In other
    words, a null pointer generates an SQL NULL, and a non-null pointer generates
    a valid SQL string literal.  If the argument to %q is a null pointer
    then no output is generated.  Thus a null-pointer to %q is the same as
    an empty string.
    For these  substitutions, the precision is the number of bytes or
    characters taken from the argument, not the number of bytes or characters that
    are written into the output.
    
    The %q and %Q substitutions are SQLite enhancements, not found in
    most other printf() implementations.

w


    This substitution works like %q except that it doubles all double-quote
    characters (") instead of single-quotes, making the result suitable for
    using with a double-quoted identifier name in an SQL statement.
    
    The %w substitution is an SQLite enhancements, not found in
    most other printf() implementations.


c2: SQLite's Built-in printf()
c3: 2.2. The Optional Length Field
c5: printf.html#the_optional_length_field
c4:
The length of the argument value can be specified by one or more letters
that occur just prior to the substitution type letter.  In SQLite, the
length only matter for integer types.  The length is ignored for the
format() SQL function which always uses 64-bit values.  The following
table shows the length specifiers allowed by SQLite:




Length Specifier
Meaning

(default)
An "int" or "unsigned int".  32-bits on all modern systems.

l
A "long int" or "long unsigned int".  Also 32-bits on all modern systems.

ll
A "long long int" or "long long unsigned" or an "sqlite3_int64" or
    "sqlite3_uint64" value.  These are 64-bit integers on all modern systems.



Only the "ll" length modifier ever makes a difference for SQLite.  And
it only makes a difference when using the C-language interfaces.


c2: SQLite's Built-in printf()
c3: 2.3. The Optional Width Field
c5: printf.html#the_optional_width_field
c4:
The width field specifies the minimum width of the substituted value in
the output.  If the string or number that is written into the output is shorter
than the width, then the value is padded.  Padding is on the left (the
value is right-justified) by default.  If the "-" flag is used, then the
padding is on the right and the value is left-justified.

The width is measured in bytes by default.  However, if the "!" flag is
present then the width is in characters.  This only makes a difference for
multi-byte utf-8 characters, and those only occur on string substitutions.

If the width is a single "*" character instead of a number, then the
actual width value is read as an integer from the argument list.  If the
value read is negative, then the absolute value is used for the width and
the value is left-justified as if the "-" flag were present.

If the value being substituted is larger than the width, then full value
is added to the output.  In other words, the width is the minimum width of
the value as it is rendered in the output.


c2: SQLite's Built-in printf()
c3: 2.4. The Optional Precision Field
c5: printf.html#the_optional_precision_field
c4:
The precision field, if it is present, must follow the width separated
by a single "." character.  If there is no width, then the "." that introduces
the precision immediately follows either the flags (if there are any) or
the initial "%".

For string substitutions (%s, %z, %q, %Q, or %w) the precision is the number
of byte or character used from the argument.  The number is bytes by default but
is characters if the "!" flag is present.  If there is no precision, then the
entire string is substituted.  Examples:  "%.3s" substitutes the first 3 bytes
of the argument string.  "%!.3s" substitutes the first three characters of the
argument string.

For integer substitutions (%d, %i, %x, %X, %o, and %p) the precision specifies
minimum number of digits to display.  Leading zeros are added if necessary, to
expand the output to the minimum number of digits.

For floating-point substitutions (%e, %E, %f, %g, %G) the precision specifies 
the number of digits to display to the right of the decimal point.

For the character substitution (%c) a precision N greater than 1 causes the
character to be repeated N times.  This is a non-standard extension found only
in SQLite.

If the precision is a single "*" character instead of a number, then the
actual precision value is read as an integer from the argument list.


c2: SQLite's Built-in printf()
c3: 2.5. The Options Flags Field
c5: printf.html#the_options_flags_field
c4:
Flags consist of zero or more characters that immediately follow the
"%" that introduces the substitution.  The various flags and their meanings
are as follows:




Flag
Meaning

-
Left-justify the value in the output.  The default is to right-justify.
If the width is zero or is otherwise less than the length of the value being
substituted, then there is no padding and the "-" flag is a no-op.

+
For signed numeric substitutions, include a "+" sign before positive numbers.
A "-" sign always appears before negative numbers regardless of flag settings.

(space)
For signed numeric substitutions, prepend a single space before positive
numbers.

0
(The zero-padding option)
Prepend as many "0" characters to numeric substitutions as necessary to
expand the value out to the specified width.  If the width field is omitted,
then this flag is a no-op.
Infinity and NaN (Not-A-Number) floating point values are normally rendered
as "Inf" and "NaN", respectively, but with the zero-padding option enabled
they are rendered as "9.0e+999" and "null".  In other words, with the zero-padding
option, floating-point Infinity and NaN are rendered as valid SQL and JSON literals.

#
This is the "alternate-form-1" flag.
For %g and %G substitutions, this causes trailing zeros to be removed.
This flag forces a decimal point to appear for all floating-point substitutions.
For %o, %x, and %X substitutions, the alternate-form-1 flag cause the value
to be prepended with "0", "0x", or "0X", respectively.



,
The comma option causes comma separators to be added to the output of numeric
substitutions (%d, %f, and similar) before every 3rd digits to the left of
the decimal point.  No commas are added for digits to the right of the decimal point.
This can help humans to more easily discern the magnitude of large integer values.
For example,
the value 2147483647 would be rendered as "2147483647" using "%d" but would
appear as "2,147,483,647" with "%,d".  This flag is a non-standard extension.

!
This is the "alternate-form-2 flag.
For string substitutions, this flag causes the width and precision to be understand
in terms of characters rather than bytes.
For floating point substitutions, the alternate-form-2 flag increases the 
maximum number of significant digits displayed from 16 to 26,
forces the display of the decimal point and causes at least one digit
to appear after the decimal point.
The alternate-form-2 flag is a non-standard extension that appears in no
other printf() implementations, as far as we know.


c2: SQLite's Built-in printf()
c3: 3. Implementation And History
c5: printf.html#implementation_and_history
c4:
The core string formatting routine is the sqlite3VXPrintf() function found in the
printf.c source file.  All the
various interfaces invoke (sometimes indirectly) this one core function.
The sqlite3VXPrintf() function began as code written by the first author
of SQLite (Hipp) when he was a graduate student at Duke University in the
late 1980s.  Hipp kept this printf() implementation in his personal toolbox until
he started working on SQLite in 2000.  The code was incorporated into the
SQLite source tree on 2000-10-08
for SQLite version 1.0.9.


The Fossil Version Control System uses its own
printf() implementation that is derived from an early version of the SQLite
printf() implementation, but those two implementations have since diverged.


The sqlite3_snprintf() function has its buffer pointer and buffer size
arguments reversed from what is found in the standard C library snprintf()
routine. This is because there was no snprintf() routine in the
standard C library
when Hipp was first implementing his version, and he chose a different order
than the designers of the standard C library.
This page last modified on  2023-06-29 23:18:37 UTC


c2: Command Line Shell For SQLite
c3: 1. Getting Started
c5: cli.html#getting_started
c4:
The SQLite project provides a simple command-line program named
sqlite3 (or sqlite3.exe on Windows)
that allows the user to manually enter and execute SQL
statements against an SQLite database or against a
ZIP archive.  This document provides a brief
introduction on how to use the sqlite3 program.

Start the sqlite3 program by typing "sqlite3" at the
command prompt, optionally followed
by the name of the file that holds the SQLite database
(or ZIP archive).  If the named
file does not exist, a new database file with the given name will be
created automatically.  If no database file is specified on the
command-line, a temporary database is created and automatically deleted when
the "sqlite3" program exits.

On startup, the sqlite3 program will show a brief banner
message then prompt you to enter SQL.  Type in SQL statements (terminated
by a semicolon), press "Enter" and the SQL will be executed.

For example, to create a new SQLite database named "ex1"
with a single table named "tbl1", you might do this:

$ sqlite3 ex1
SQLite version 3.36.0 2021-06-18 18:36:39
Enter ".help" for usage hints.
sqlite&gt; create table tbl1(one text, two int);
sqlite&gt; insert into tbl1 values('hello!',10);
sqlite&gt; insert into tbl1 values('goodbye', 20);
sqlite&gt; select * from tbl1;
hello!|10
goodbye|20
sqlite&gt;


Terminate the sqlite3 program by typing your system
End-Of-File character (usually a Control-D).  Use the interrupt
character (usually a Control-C) to stop a long-running SQL statement.

Make sure you type a semicolon at the end of each SQL command!
The sqlite3 program looks for a semicolon to know when your SQL command is
complete.  If you omit the semicolon, sqlite3 will give you a
continuation prompt and wait for you to enter more text to
complete the SQL command.  This feature allows you to
enter SQL commands that span multiple lines.  For example:


sqlite&gt; CREATE TABLE tbl2 (
   ...&gt;   f1 varchar(30) primary key,
   ...&gt;   f2 text,
   ...&gt;   f3 real
   ...&gt; );
sqlite&gt;


c2: Command Line Shell For SQLite
c3: 2. Double-click Startup On Windows
c5: cli.html#double_click_startup_on_windows
c4:
Windows users can double-click on the sqlite3.exe icon to cause
the command-line shell to pop-up a terminal window running SQLite.  However,
because double-clicking starts the sqlite3.exe without command-line arguments,
no database file will have been specified, so SQLite will use a temporary
database that is deleted when the session exits.
To use a persistent disk file as the database, enter the ".open" command
immediately after the terminal window starts up:

SQLite version 3.36.0 2021-06-18 18:36:39
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite&gt; .open ex1.db
sqlite&gt;


The example above causes the database file named "ex1.db" to be opened
and used.  The "ex1.db" file is created if it does not previously exist.
You might want to
use a full pathname to ensure that the file is in the directory that you
think it is in.  Use forward-slashes as the directory separator character.
In other words use "c:/work/ex1.db", not "c:\work\ex1.db".

Alternatively, you can create a new database using the default temporary
storage, then save that database into a disk file using the ".save" command:

SQLite version 3.36.0 2021-06-18 18:36:39
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite&gt; ... many SQL commands omitted ...
sqlite&gt; .save ex1.db
sqlite&gt;


Be careful when using the ".save" command as it will overwrite any
preexisting database files having the same name without prompting for
confirmation.  As with the ".open" command, you might want to use a
full pathname with forward-slash directory separators to avoid ambiguity.


c2: Command Line Shell For SQLite
c3: 3. Special commands to sqlite3 (dot-commands)
c5: cli.html#special_commands_to_sqlite3_dot_commands_
c4:
Most of the time, sqlite3 just reads lines of input and passes them
on to the SQLite library for execution.
But input lines that begin with a dot (".")
are intercepted and interpreted by the sqlite3 program itself.
These "dot commands" are typically used to change the output format
of queries, or to execute certain prepackaged query statements.
There were originally just a few dot commands, but over the years
many new features have accumulated so that today there are over 60.



For a listing of the available dot commands, you can enter ".help" with
no arguments.  Or enter ".help TOPIC" for detailed information about TOPIC.
The list of available dot-commands follows:


sqlite&gt; .help
.archive ...             Manage SQL archives
.auth ON|OFF             Show authorizer callbacks
.backup ?DB? FILE        Backup DB (default "main") to FILE
.bail on|off             Stop after hitting an error.  Default OFF
.cd DIRECTORY            Change the working directory to DIRECTORY
.changes on|off          Show number of rows changed by SQL
.check GLOB              Fail if output since .testcase does not match
.clone NEWDB             Clone data into NEWDB from the existing database
.connection &#91;close&#93; &#91;#&#93;  Open or close an auxiliary database connection
.databases               List names and files of attached databases
.dbconfig ?op? ?val?     List or change sqlite3_db_config() options
.dbinfo ?DB?             Show status information about the database
.dump ?OBJECTS?          Render database content as SQL
.echo on|off             Turn command echo on or off
.eqp on|off|full|...     Enable or disable automatic EXPLAIN QUERY PLAN
.excel                   Display the output of next command in spreadsheet
.exit ?CODE?             Exit this program with return-code CODE
.expert                  EXPERIMENTAL. Suggest indexes for queries
.explain ?on|off|auto?   Change the EXPLAIN formatting mode.  Default: auto
.filectrl CMD ...        Run various sqlite3_file_control() operations
.fullschema ?--indent?   Show schema and the content of sqlite_stat tables
.headers on|off          Turn display of headers on or off
.help ?-all? ?PATTERN?   Show help text for PATTERN
.import FILE TABLE       Import data from FILE into TABLE
.imposter INDEX TABLE    Create imposter table TABLE on index INDEX
.indexes ?TABLE?         Show names of indexes
.limit ?LIMIT? ?VAL?     Display or change the value of an SQLITE_LIMIT
.lint OPTIONS            Report potential schema issues.
.load FILE ?ENTRY?       Load an extension library
.log FILE|off            Turn logging on or off.  FILE can be stderr/stdout
.mode MODE ?TABLE?       Set output mode
.nonce STRING            Disable safe mode for one command if the nonce matches
.nullvalue STRING        Use STRING in place of NULL values
.once ?OPTIONS? ?FILE?   Output for the next SQL command only to FILE
.open ?OPTIONS? ?FILE?   Close existing database and reopen FILE
.output ?FILE?           Send output to FILE or stdout if FILE is omitted
.parameter CMD ...       Manage SQL parameter bindings
.print STRING...         Print literal STRING
.progress N              Invoke progress handler after every N opcodes
.prompt MAIN CONTINUE    Replace the standard prompts
.quit                    Stop interpreting input stream, exit if primary.
.read FILE               Read input from FILE
.recover                 Recover as much data as possible from corrupt db.
.restore ?DB? FILE       Restore content of DB (default "main") from FILE
.save FILE               Write in-memory database into FILE
.scanstats on|off        Turn sqlite3_stmt_scanstatus() metrics on or off
.schema ?PATTERN?        Show the CREATE statements matching PATTERN
.selftest ?OPTIONS?      Run tests defined in the SELFTEST table
.separator COL ?ROW?     Change the column and row separators
.session ?NAME? CMD ...  Create or control sessions
.sha3sum ...             Compute a SHA3 hash of database content
.shell CMD ARGS...       Run CMD ARGS... in a system shell
.show                    Show the current values for various settings
.stats ?ARG?             Show stats or turn stats on or off
.system CMD ARGS...      Run CMD ARGS... in a system shell
.tables ?TABLE?          List names of tables matching LIKE pattern TABLE
.testcase NAME           Begin redirecting output to 'testcase-out.txt'
.testctrl CMD ...        Run various sqlite3_test_control() operations
.timeout MS              Try opening locked tables for MS milliseconds
.timer on|off            Turn SQL timer on or off
.trace ?OPTIONS?         Output each SQL statement as it is run
.vfsinfo ?AUX?           Information about the top-level VFS
.vfslist                 List all available VFSes
.vfsname ?AUX?           Print the name of the VFS stack
.width NUM1 NUM2 ...     Set minimum column widths for columnar output
sqlite&gt;


c2: Command Line Shell For SQLite
c3: 4.1. Line Structure
c5: cli.html#line_structure
c4:
The CLI's input is parsed into a sequence consisting of:
  
    SQL statements;
    dot-commands; or
    CLI comments
  
SQL statements are free-form, and can be spread across multiple lines,
  with whitespace or SQL comments embedded anywhere.
  They are terminated by either a ';' character at the end of an input line,
  or a '/' character or the word "go" on a line by itself.
  When not at the end of an input line, the ';' character
  acts to separate SQL statements.
  Trailing whitespace is ignored for purposes of termination.
A dot-command has a more restrictive structure:

It must begin with its "." at the left margin
    with no preceding whitespace.
It must be entirely contained on a single input line.
It cannot occur in the middle of an ordinary SQL
    statement.  In other words, it cannot occur at a
    continuation prompt.
There is no comment syntax for dot-commands.

The CLI also accepts whole-line comments that
begin with a '#' character and extend to the end of the line.
There can be no with whitespace prior to the '#'.


c2: Command Line Shell For SQLite
c3: 4.2. Dot-command arguments
c5: cli.html#dot_command_arguments
c4:
The arguments passed to dot-commands are parsed from the command tail,
  per these rules:

  The trailing newline and any other trailing whitespace is discarded;
  Whitespace immediately following the dot-command name, or any argument
    input end bound is discarded;
  An argument input begins with any non-whitespace character;
  An argument input ends with a character which
    depends upon its leading character thusly:
  
    for a leading single-quote ('), a single-quote acts
      as the end delimiter;
    for a leading double-quote ("), an unescaped double-quote
      acts as the end delimiter;
    for any other leading character, the end delimiter is
      any whitespace; and
    the command tail end acts as the end delimiter for any argument;
  
  Within a double-quoted argument input, a backslash-escaped double-quote
    is part of the argument rather than its terminating quote;
  Within a double-quoted argument, traditional C-string literal, backslash
    escape sequence translation is done; and
  Argument input delimiters (the bounding quotes or whitespace)
    are discarded to yield the passed argument.


c2: Command Line Shell For SQLite
c3: 4.3. Dot-command execution
c5: cli.html#dot_command_execution
c4:
The dot-commands
are interpreted by the sqlite3.exe command-line program, not by
SQLite itself.  So none of the dot-commands will work as an argument
to SQLite interfaces such as sqlite3_prepare() or sqlite3_exec().


c2: Command Line Shell For SQLite
c3: 5. Changing Output Formats
c5: cli.html#changing_output_formats
c4:
The sqlite3 program is able to show the results of a query
in 14 different output formats:


 ascii
 box
 csv
 column
 html
 insert
 json
 line
 list
 markdown
 quote
 table
 tabs
 tcl


You can use the ".mode" dot command to switch between these output
formats.
The default output mode is "list".  In
list mode, each row of a query result is written on one line of
output and each column within that row is separated by a specific
separator string.  The default separator is a pipe symbol ("|").
List mode is especially useful when you are going to send the output
of a query to another program (such as AWK) for additional processing.

sqlite&gt; .mode list
sqlite&gt; select * from tbl1;
hello!|10
goodbye|20
sqlite&gt;


Type ".mode" with no arguments to show the current mode:

sqlite&gt; .mode
current output mode: list
sqlite&gt;


Use the ".separator" dot command to change the separator.
For example, to change the separator to a comma and
a space, you could do this:

sqlite&gt; .separator ", "
sqlite&gt; select * from tbl1;
hello!, 10
goodbye, 20
sqlite&gt;


The next ".mode" command might reset the ".separator" back to some
default value (depending on its arguments).
So you will likely need to repeat the ".separator" command whenever you
change modes if you want to continue using a non-standard separator.



In "quote" mode, the output is formatted as SQL literals.  Strings are
enclosed in single-quotes and internal single-quotes are escaped by doubling.
Blobs are displayed in hexadecimal blob literal notation (Ex: x'abcd').
Numbers are displayed as ASCII text and NULL values are shown as "NULL".
All columns are separated from each other by a comma (or whatever alternative
character is selected using ".separator").

sqlite&gt; .mode quote
sqlite&gt; select * from tbl1;
'hello!',10
'goodbye',20
sqlite&gt;


In "line" mode, each column in a row of the database
is shown on a line by itself.  Each line consists of the column
name, an equal sign and the column data.  Successive records are
separated by a blank line.  Here is an example of line mode
output:

sqlite&gt; .mode line
sqlite&gt; select * from tbl1;
one = hello!
two = 10

one = goodbye
two = 20
sqlite&gt;





In column mode, each record is shown on a separate line with the
data aligned in columns.  For example:

sqlite&gt; .mode column
sqlite&gt; select * from tbl1;
one       two
--------  ---
hello!    10
goodbye   20
sqlite&gt;


In "column" mode (and also in "box", "table", and "markdown" modes)
the width of columns adjusts automatically.  But you can override this,
providing a speicified width for each column using the ".width" command.
The arguments to ".width" are integers which are the number of
characters to devote to each column.  Negative numbers mean right-justify.
Thus:

sqlite&gt; .width 12 -6
sqlite&gt; select * from tbl1;
one              two
------------  ------
hello!            10
goodbye           20
sqlite&gt;


A width of 0 means the column width is chosen automatically.
Unspecified column widths become zero.  Hence, the command
".width" with no arguments resets all column widths to zero and
hence causes all column widths to be determined automatically.

The "column" mode is a tabular output format.  Other
tabular output formats are "box", "markdown", and "table":

sqlite&gt; .width
sqlite&gt; .mode markdown
sqlite&gt; select * from tbl1;
|   one   | two |
|---------|-----|
| hello!  | 10  |
| goodbye | 20  |
sqlite&gt; .mode table
sqlite&gt; select * from tbl1;
+---------+-----+
|   one   | two |
+---------+-----+
| hello!  | 10  |
| goodbye | 20  |
+---------+-----+
sqlite&gt; .mode box
sqlite&gt; select * from tbl1;
┌─────────┬─────┐
│   one   │ two │
├─────────┼─────┤
│ hello!  │ 10  │
│ goodbye │ 20  │
└─────────┴─────┘
sqlite&gt;





The columnar modes accept some addition options to control formatting.
The "--wrap N" option (where N is an integer) causes columns
to wrap text that is longer than N characters.  Wrapping is disabled if
N is zero.

sqlite&gt; insert into tbl1 values('The quick fox jumps over a lazy brown dog.',90);
sqlite&gt; .mode box --wrap 30
sqlite&gt; select * from tbl1 where two&gt;50;
┌────────────────────────────────┬─────┐
│              one               │ two │
├────────────────────────────────┼─────┤
│ The quick fox jumps over a laz │ 90  │
│ y brown dog.                   │     │
└────────────────────────────────┴─────┘
sqlite&gt;


Wrapping happens after exactly N characters,
which might be in the middle of a word.
To wrap at a word boundary, add the "--wordwrap on" option
(or just "-ww" for short):

sqlite&gt; .mode box --wrap 30 -ww
sqlite&gt; select * from tbl1 where two&gt;50;
┌─────────────────────────────┬─────┐
│             one             │ two │
├─────────────────────────────┼─────┤
│ The quick fox jumps over a  │ 90  │
│ lazy brown dog.             │     │
└─────────────────────────────┴─────┘
sqlite&gt;


The "--quote" option causes the results in each column to be
quoted like an SQL literal, as in the "quote" mode.  See the on-line
help for additional options.



The command ".mode box --wrap 60 --quote" is so useful for general-purpose
database queries that it is given its own alias.  Instead of typing out
that whole 27-character command, you can just say ".mode qbox".

Another useful output mode is "insert".  In insert mode, the output
is formatted to look like SQL INSERT statements.  Use insert
mode to generate text that can later be used to input data into a
different database.

When specifying insert mode, you have to give an extra argument
which is the name of the table to be inserted into.  For example:

sqlite&gt; .mode insert new_table
sqlite&gt; select * from tbl1 where two&lt;50;
INSERT INTO "new_table" VALUES('hello',10);
INSERT INTO "new_table" VALUES('goodbye',20);
sqlite&gt;



Other output modes include "csv", "json", and "tcl".  Try these
yourself to see what they do.


c2: Command Line Shell For SQLite
c3: 6. Querying the database schema
c5: cli.html#querying_the_database_schema
c4:
The sqlite3 program provides several convenience commands that
are useful for looking at the schema of the database.  There is
nothing that these commands do that cannot be done by some other
means.  These commands are provided purely as a shortcut.



For example, to see a list of the tables in the database, you
can enter ".tables".


sqlite&gt; .tables
tbl1 tbl2
sqlite&gt;



The ".tables" command is similar to setting list mode then
executing the following query:

SELECT name FROM sqlite_schema
WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%'
ORDER BY 1


But the ".tables" command does more.  It queries the sqlite_schema table
for all attached databases, not just the primary database.  And it arranges
its output into neat columns.

The ".indexes" command works in a similar way to list all of
the indexes. If the ".indexes" command is given an argument which is
the name of a table, then it shows just indexes on that table.



The ".schema" command shows the complete schema for the database,
or for a single table if an optional tablename argument is provided:

sqlite&gt; .schema
create table tbl1(one varchar(10), two smallint)
CREATE TABLE tbl2 (
  f1 varchar(30) primary key,
  f2 text,
  f3 real
);
sqlite&gt; .schema tbl2
CREATE TABLE tbl2 (
  f1 varchar(30) primary key,
  f2 text,
  f3 real
);
sqlite&gt;



The ".schema" command is roughly the same as setting
list mode, then entering the following query:

SELECT sql FROM sqlite_schema
ORDER BY tbl_name, type DESC, name


As with ".tables", the ".schema" command shows the schema for
all attached databases.  If you only want to see the schema for
a single database (perhaps "main") then you can add an argument
to ".schema" to restrict its output:

sqlite&gt; .schema main.*


The ".schema" command can be augmented with the "--indent" option,
in which case it tries to reformat the various CREATE statements of
the schema so that they are more easily readable by humans.



The ".databases" command shows a list of all databases open in
the current connection.  There will always be at least 2.  The first
one is "main", the original database opened.  The second is "temp",
the database used for temporary tables. There may be additional
databases listed for databases attached using the ATTACH statement.
The first output column is the name the database is attached with,
and the second result column is the filename of the external file.
There may be a third result column which will be either "'r/o'" or
"'r/w'" depending on whether the database file is read-only or read-write.
And there might be a fourth result column showing the result of
sqlite3_txn_state() for that database file.

sqlite&gt; .databases




The ".fullschema" dot-command works like the ".schema" command in
that it displays the entire database schema.  But ".fullschema" also
includes dumps of the statistics tables "sqlite_stat1", "sqlite_stat3",
and "sqlite_stat4", if they exist.  The ".fullschema" command normally
provides all of the information needed to exactly recreate a query
plan for a specific query.  When reporting suspected problems with
the SQLite query planner to the SQLite development team, developers
are requested to provide the complete ".fullschema" output as part
of the trouble report.  Note that the sqlite_stat3 and sqlite_stat4
tables contain samples of index entries and so might contain sensitive
data, so do not send the ".fullschema" output of a proprietary database
over a public channel.


c2: Command Line Shell For SQLite
c3: 7. Opening Database Files
c5: cli.html#opening_database_files
c4:
The ".open" command opens a new database connection, after first closing the
previously opened database command.  In its simplest form, the ".open" command merely
invokes sqlite3_open() on the file named as its argument.  Use the name ":memory:"
to open a new in-memory database that disappears when the CLI exits or when the
".open" command is run again.
Or use no name to open a private, temporary on-disk database which
will also disappear upon exit or use of ".open".

If the --new option is included with ".open", then the database is reset prior
to being opened.  Any prior data is destroyed.  This is a destructive overwrite of
prior data and no confirmation is requested, so use this option carefully.

The --readonly option opens the database in read-only mode.  Write will be
prohibited.

The --deserialize option causes the entire content of the on-disk file to be
read into memory and then opened as an in-memory database using the
sqlite3_deserialize() interface.  This will, of course, require a lot of memory
if you have a large database.  Also, any changes you make to the database will not
be saved back to disk unless you explicitly save them using the ".save" or ".backup"
commands.

The --append option causes the SQLite database to be appended to an existing
file rather than working as a stand-alone file.  See the
appendvfs extension for
more information.

The --zip option causes the specified input file to be interpreted as a ZIP archive
instead of as an SQLite database file.

The --hexdb option causes the database content to be to be read from subsequent
lines of input in a hex format, rather than from a separate file on disk.
The "dbtotxt" command-line tool can be used to generate
the appropriate text for a database.  The --hexdb option is intended for use by the
SQLite developers for testing purposes.  We do not know of any use cases for this
option outside of internal SQLite testing and development.


c2: Command Line Shell For SQLite
c3: 8.1. Writing results to a file
c5: cli.html#writing_results_to_a_file
c4:
By default, sqlite3 sends query results to standard output.  You
can change this using the ".output" and ".once" commands.  Just put
the name of an output file as an argument to .output and all subsequent
query results will be written to that file.  Or use the .once command
instead of .output and output will only be redirected for the single next
command before reverting to the console.  Use .output with no arguments to
begin writing to standard output again.  For example:

sqlite&gt; .mode list
sqlite&gt; .separator |
sqlite&gt; .output test_file_1.txt
sqlite&gt; select * from tbl1;
sqlite&gt; .exit
$ cat test_file_1.txt
hello|10
goodbye|20
$


If the first character of the ".output" or ".once" filename is a pipe
symbol ("|") then the remaining characters are treated as a command and the
output is sent to that command.  This makes it easy to pipe the results
of a query into some other process.  For example, the
"open -f" command on a Mac opens a text editor to display the content that
it reads from standard input.  So to see the results of a query
in a text editor, one could type:

sqlite&gt; .once | open -f
sqlite&gt; SELECT * FROM bigTable;


If the ".output" or ".once" commands have an argument of "-e" then
output is collected into a temporary file and the system text editor is
invoked on that text file.  Thus, the command ".once -e" achieves the
same result as ".once '|open -f'" but with the benefit of being portable
across all systems.

If the ".output" or ".once" commands have a "-x" argument, that causes
them to accumulate output as Comma-Separated-Values (CSV) in a temporary
file, then invoke the default system utility for viewing CSV files
(usually a spreadsheet program) on the result.  This is a quick way of
sending the result of a query to a spreadsheet for easy viewing:

sqlite&gt; .once -x
sqlite&gt; SELECT * FROM bigTable;




The ".excel" command is an alias for ".once -x".  It does exactly the same
thing.


c2: Command Line Shell For SQLite
c3: 8.2. Reading SQL from a file
c5: cli.html#reading_sql_from_a_file
c4:
In interactive mode, sqlite3 reads input text (either SQL statements
or dot-commands) from the keyboard.  You can also redirect input from
a file when you launch sqlite3, of course, but then you do not have the
ability to interact with the program.  Sometimes it is useful to run an
SQL script contained in a file entering other commands from the command-line.
For this, the ".read" dot-command is provided.

The ".read" command takes a single argument which is (usually) the name
of a file from which to read input text.

sqlite&gt; .read myscript.sql


The ".read" command temporarily stops reading from the keyboard and instead
takes its input from the file named.  Upon reaching the end of the file,
input reverts back to the keyboard.  The script file may contain dot-commands,
just like ordinary interactive input.

If the argument to ".read" begins with the "|" character, then instead of
opening the argument as a file, it runs the argument (without the leading "|")
as a command, then uses the output of that command as its input.  Thus, if you
have a script that generates SQL, you can execute that SQL directly using
a command similar to the following:

sqlite&gt; .read |myscript.bat


c2: Command Line Shell For SQLite
c3: 8.3. File I/O Functions
c5: cli.html#file_i_o_functions
c4:
The command-line shell adds two application-defined SQL functions that
facilitate reading content from a file into a table column, and writing the
content of a column into a file, respectively.

The readfile(X) SQL function reads the entire content of the file named
X and returns that content as a BLOB.  This can be used to load content into
a table.  For example:

sqlite&gt; CREATE TABLE images(name TEXT, type TEXT, img BLOB);
sqlite&gt; INSERT INTO images(name,type,img)
   ...&gt;   VALUES('icon','jpeg',readfile('icon.jpg'));


The writefile(X,Y) SQL function write the blob Y into the file named X
and returns the number of bytes written.  Use this function to extract
the content of a single table column into a file.  For example:

sqlite&gt; SELECT writefile('icon.jpg',img) FROM images WHERE name='icon';


Note that the readfile(X) and writefile(X,Y) functions are extension
functions and are not built into the core SQLite library.  These routines
are available as a loadable extension in the
ext/misc/fileio.c
source file in the SQLite source code repositories.


c2: Command Line Shell For SQLite
c3: 8.4. The edit() SQL function
c5: cli.html#the_edit_sql_function
c4:
The CLI has another built-in SQL function named edit().  Edit() takes
one or two arguments.  The first argument is a value - often a large
multi-line string to be edited.  The second argument is the invocation
for a text editor. (It may include options to affect the editor's
behavior.) If the second argument is omitted, the VISUAL environment
variable is used.  The edit() function writes its first argument into a
temporary file, invokes the editor on the temporary file, rereads the file
back into memory after the editor is done, then returns the edited text.

The edit() function can be used to make changes to large text
values.  For example:

sqlite&gt; UPDATE docs SET body=edit(body) WHERE name='report-15';


In this example, the content of the docs.body field for the entry where
docs.name is "report-15" will be sent to the editor.  After the editor returns,
the result will be written back into the docs.body field.

The default operation of edit() is to invoke a text editor.  But by using
an alternative edit program in the second argument, you can also get it to edit
images or other non-text resources.  For example, if you want to modify a JPEG
image that happens to be stored in a field of a table, you could run:

sqlite&gt; UPDATE pics SET img=edit(img,'gimp') WHERE id='pic-1542';


The edit program can also be used as a viewer, by simply ignoring the
return value.  For example, to merely look at the image above, you might run:

sqlite&gt; SELECT length(edit(img,'gimp')) WHERE id='pic-1542';


c2: Command Line Shell For SQLite
c3: 8.5. Importing files as CSV or other formats
c5: cli.html#importing_files_as_csv_or_other_formats
c4:
Use the ".import" command to import CSV (comma separated value)
or similarly delimited data into an SQLite table.
The ".import" command takes two arguments which are the
source from which data is to be read and the name of the
SQLite table into which the data is to be inserted. The source argument
is the name of a file to be read or, if it begins with a "|" character,
it specifies a command which will be run to produce the input data.

Note that it may be important to set the "mode" before running the
 ".import" command.  This is prudent to prevent the command-line shell
from trying to interpret the input file text as some format other than
how the file is structured. If the --csv or --ascii options are used,
they control import input delimiters. Otherwise, the delimiters are
those in effect for the current output mode.

To import into a table not in the "main" schema, the --schema option
may be used to specify that the table is in some other schema. This can
be useful for ATTACH'ed databases or to import into a TEMP table.

When .import is run, its treatment of the first input row depends
upon whether the target table already exists. If it does not exist,
the table is automatically created and the content of the first input
row is used to set the name of all the columns in the table. In this
case, the table data content is taken from the second and subsequent
input rows. If the target table already exists, every row of the
input, including the first, is taken to be actual data content.  If
the input file contains an initial row of column labels, you can make
the .import command skip that initial row using the "--skip 1" option.

Here is an example usage, loading a pre-existing temporary table
from a CSV file which has column names in its first row:

sqlite&gt; .import --csv --skip 1 --schema temp C:/work/somedata.csv tab1


While reading input data in modes other than 'ascii', ".import"
interprets input as records composed of fields according to the RFC 4180
specification with this exception: The input record and field separators
are as set by the mode or by use of the .separator command. Fields are
always subject to quote removal to reverse quoting done per RFC 4180,
except in ascii mode.

To import data with arbitrary delimiters and no quoting,
first set ascii mode (".mode&nbsp;ascii"), then set the field
and record delimiters using the ".separator" command. This
will suppress dequoting. Upon ".import", the data will be split
into fields and records according to the delimiters so specified.


c2: Command Line Shell For SQLite
c3: 8.6. Export to CSV
c5: cli.html#export_to_csv
c4:
To export an SQLite table (or part of a table) as CSV, simply set
the "mode" to "csv" and then run a query to extract the desired rows
of the table. The output will formatted as CSV per RFC 4180.

sqlite&gt; .headers on
sqlite&gt; .mode csv
sqlite&gt; .once c:/work/dataout.csv
sqlite&gt; SELECT * FROM tab1;
sqlite&gt; .system c:/work/dataout.csv


In the example above, the ".headers on" line causes column labels to
be printed as the first row of output.  This means that the first row of
the resulting CSV file will contain column labels.  If column labels are
not desired, set ".headers off" instead. (The ".headers off" setting is
the default and can be omitted if the headers have not been previously
turned on.)

The line ".once FILENAME" causes all query output to go into
the named file instead of being printed on the console.  In the example
above, that line causes the CSV content to be written into a file named
"C:/work/dataout.csv".

The final line of the example (the ".system c:/work/dataout.csv")
has the same effect as double-clicking on the c:/work/dataout.csv file
in windows.  This will typically bring up a spreadsheet program to display
the CSV file.

That command only works as written on Windows.
The equivalent line on a Mac would be:

sqlite&gt; .system open dataout.csv


On Linux and other unix systems you will need to enter something like:


sqlite&gt; .system xdg-open dataout.csv


c2: Command Line Shell For SQLite
c3: 8.6.1. Export to Excel
c5: cli.html#_export_to_excel_
c4:
To simplify export to a spreadsheet, the CLI provides the
".excel" command which captures the output of a single query and sends
that output to the default spreadsheet program on the host computer.
Use it like this:

sqlite&gt; .excel
sqlite&gt; SELECT * FROM tab;



The command above writes the output of the query as CSV into a temporary
file, invokes the default handler for CSV files (usually the preferred
spreadsheet program such as Excel or LibreOffice), then deletes the
temporary file.  This is essentially a short-hand method of doing
the sequence of ".csv", ".once", and ".system" commands described above.


The ".excel" command is really an alias for ".once -x".  The -x option
to .once causes it to writes results as CSV into a temporary file that
is named with a ".csv" suffix, then invoke the systems default handler
for CSV files.


There is also a ".once -e" command which works similarly, except that
it names the temporary file with a ".txt" suffix so that the default
text editor for the system will be invoked, instead of the default
spreadsheet.


c2: Command Line Shell For SQLite
c3: 8.6.2. Export to TSV (tab separated values)
c5: cli.html#_export_to_tsv_tab_separated_values_
c4:
Exporting to pure TSV, without any field quoting, can be done by
entering ".mode tabs" before running a query. However, the output
will not be read correctly in tabs mode by the ".import" command
if it contains doublequote characters. To get TSV quoted per
RFC 4180 so that it can be input in tabs mode with ".import",
first enter ".mode&nbsp;csv", then enter '.separator&nbsp;"\t"'
before running a query.


c2: Command Line Shell For SQLite
c3: 9. Accessing ZIP Archives As Database Files
c5: cli.html#accessing_zip_archives_as_database_files
c4:
In addition to reading and writing SQLite database files,
the sqlite3 program will also read and write ZIP archives.
Simply specify a ZIP archive filename in place of an SQLite database
filename on the initial command line, or in the ".open" command,
and sqlite3 will automatically detect that the file is a
ZIP archive instead of an SQLite database and will open it as such.
This works regardless of file suffix.  So you can open JAR, DOCX,
and ODP files and any other file format that is really a ZIP
archive and SQLite will read it for you.

A ZIP archive appears to be a database containing a single table
with the following schema:

CREATE TABLE zip(
  name,     // Name of the file
  mode,     // Unix-style file permissions
  mtime,    // Timestamp, seconds since 1970
  sz,       // File size after decompression
  rawdata,  // Raw compressed file data
  data,     // Uncompressed file content
  method    // ZIP compression method code
);


So, for example, if you wanted to see the compression efficiency
(expressed as the size of the compressed content relative to the
original uncompressed file size) for all files in the ZIP archive,
sorted from most compressed to least compressed, you could run a
query like this:

sqlite&gt; SELECT name, (100.0*length(rawdata))/sz FROM zip ORDER BY 2;


Or using file I/O functions, you can extract elements of the
ZIP archive:

sqlite&gt; SELECT writefile(name,content) FROM zip
   ...&gt; WHERE name LIKE 'docProps/%';


c2: Command Line Shell For SQLite
c3: 9.1. How ZIP archive access is implemented
c5: cli.html#how_zip_archive_access_is_implemented
c4:
The command-line shell uses the Zipfile virtual table to
access ZIP archives.  You can see this by running the ".schema"
command when a ZIP archive is open:

sqlite&gt; .schema
CREATE VIRTUAL TABLE zip USING zipfile('document.docx')
/* zip(name,mode,mtime,sz,rawdata,data,method) */;


When opening a file, if the command-line client discovers that the
file is ZIP archive instead of an SQLite database, it actually opens
an in-memory database and then in that in-memory database it creates
an instance of the Zipfile virtual table that is attached to the
ZIP archive.

The special processing for opening ZIP archives is a trick of the
command-line shell, not the core SQLite library.  So if you want to
open a ZIP archive as a database in your application, you will need to
activate the Zipfile virtual table module then run an appropriate
CREATE VIRTUAL TABLE statement.


c2: Command Line Shell For SQLite
c3: 10. Converting An Entire Database To A Text File
c5: cli.html#converting_an_entire_database_to_a_text_file
c4:
Use the ".dump" command to convert the entire contents of a
database into a single UTF-8 text file.  This file can be converted
back into a database by piping it back into sqlite3.

A good way to make an archival copy of a database is this:


$ sqlite3 ex1 .dump | gzip -c &gt;ex1.dump.gz



This generates a file named ex1.dump.gz that contains everything
you need to reconstruct the database at a later time, or on another
machine.  To reconstruct the database, just type:


$ zcat ex1.dump.gz | sqlite3 ex2



The text format is pure SQL so you
can also use the .dump command to export an SQLite database
into other popular SQL database engines.  Like this:


$ createdb ex2
$ sqlite3 ex1 .dump | psql ex2


c2: Command Line Shell For SQLite
c3: 11. Recover Data From a Corrupted Database
c5: cli.html#recover_data_from_a_corrupted_database
c4:
Like the ".dump" command, ".recover" attempts to convert the entire
contents of a database file to text. The difference is that instead of
reading data using the normal SQL database interface, ".recover"
attempts to reassemble the database based on data extracted directly from
as many database pages as possible. If the database is corrupt, ".recover"
is usually able to recover data from all uncorrupted parts of the database,
whereas ".dump" stops when the first sign of corruption is encountered.

If the ".recover" command recovers one or more rows that it cannot
attribute to any database table, the output script creates a "lost_and_found"
table to store the orphaned rows. The schema of the lost_and_found
table is as follows:

CREATE TABLE lost_and_found(
    rootpgno INTEGER,             -- root page of tree pgno is a part of
    pgno INTEGER,                 -- page number row was found on
    nfield INTEGER,               -- number of fields in row
    id INTEGER,                   -- value of rowid field, or NULL
    c0, c1, c2, c3...             -- columns for fields of row
);


The "lost_and_found" table contains one row for each orphaned row recovered
from the database. Additionally, there is one row for each recovered index
entry that cannot be attributed to any SQL index. This is because, in an
SQLite database, the same format is used to store SQL index entries and
WITHOUT ROWID table entries.


  ColumnContents
  rootpgno Even though it may not be possible to attribute the
      row to a specific database table, it may be part of a tree structure
      within the database file. In this case, the root page number of that
      tree structure is stored in this column. Or, if the page the row was
      found on is not part of a tree structure, this column stores a copy of
      the value in column "pgno" - the page number of the page the row was
      found on. In many, although not all, cases, all rows in the
      lost_and_found table with the same value in this column belong to the
      same table.

  pgno The page number of the page on which this row was found.

  nfield The number of fields in this row.

  id If the row comes from a WITHOUT ROWID table, this column
      contains NULL. Otherwise, it contains the 64-bit integer rowid value for
      the row.

  c0,&nbsp;c1,&nbsp;c2... The values for each column of the row
      are stored in these columns. The ".recover" command creates the
      lost_and_found table with as many columns as required by the longest
      orphaned row.


If the recovered database schema already contains a table named
"lost_and_found", the ".recover" command uses the name "lost_and_found0". If
the name "lost_and_found0" is also already taken, "lost_and_found1", and so
on. The default name "lost_and_found" may be overridden by invoking ".recover"
with the --lost-and-found switch. For example, to have the output script call
the table "orphaned_rows":

sqlite&gt; .recover --lost-and-found orphaned_rows


c2: Command Line Shell For SQLite
c3: 12. Loading Extensions
c5: cli.html#loading_extensions
c4:
You can add new custom application-defined SQL functions,
collating sequences, virtual tables, and VFSes to the command-line
shell at run-time using the ".load" command.  First, build the
extension as a DLL or shared library (as described in the
Run-Time Loadable Extensions document) then type:

sqlite&gt; .load /path/to/my_extension


Note that SQLite automatically adds the appropriate extension suffix
(".dll" on windows, ".dylib" on Mac, ".so" on most other unixes) to the
extension filename.  It is generally a good idea to specify the full
pathname of the extension.

SQLite computes the entry point for the extension based on the extension
filename.  To override this choice, simply add the name of the extension
as a second argument to the ".load" command.

Source code for several useful extensions can be found in the
ext/misc
subdirectory of the SQLite source tree.  You can use these extensions
as-is, or as a basis for creating your own custom extensions to address
your own particular needs.


c2: Command Line Shell For SQLite
c3: 13. Cryptographic Hashes Of Database Content
c5: cli.html#cryptographic_hashes_of_database_content
c4:
The ".sha3sum" dot-command computes a
SHA3 hash of the content
of the database.  To be clear, the hash is computed over the database content,
not its representation on disk.  This means, for example, that a VACUUM
or similar data-preserving transformation does not change the hash.

The ".sha3sum" command supports options "--sha3-224", "--sha3-256",
"--sha3-384", and "--sha3-512" to define which variety of SHA3 to use
for the hash.  The default is SHA3-256.

The database schema (in the sqlite_schema table) is not normally
included in the hash, but can be added by the "--schema" option.

The ".sha3sum" command takes a single optional argument which is a
LIKE pattern.  If this option is present, only tables whose names match
the LIKE pattern will be hashed.

The ".sha3sum" command is implemented with the help of the
extension function "sha3_query()"
that is included with the command-line shell.


c2: Command Line Shell For SQLite
c3: 14. Database Content Self-Tests
c5: cli.html#database_content_self_tests
c4:
The ".selftest" command attempts to verify that a database is
intact and is not corrupt.
The .selftest command looks for a table in schema named "selftest"
and defined as follows:

CREATE TABLE selftest(
  tno INTEGER PRIMARY KEY,  -- Test number
  op TEXT,                  -- 'run' or 'memo'
  cmd TEXT,                 -- SQL command to run, or text of "memo"
  ans TEXT                  -- Expected result of the SQL command
);


The .selftest command reads the rows of the selftest table in
selftest.tno order.
For each 'memo' row, it writes the text in 'cmd' to the output.  For
each 'run' row, it runs the 'cmd' text as SQL and compares the result
to the value in 'ans', and shows an error message if the results differ.

If there is no selftest table, the ".selftest" command runs
PRAGMA integrity_check.

The ".selftest --init" command creates the selftest table if it
does not already exists, then appends entries that check the SHA3
hash of the content of all tables.  Subsequent runs of ".selftest"
will verify that the database has not been changed in any way.  To
generate tests to verify that a subset of the tables is unchanged,
simply run ".selftest --init" then DELETE the selftest rows that
refer to tables that are not constant.


c2: Command Line Shell For SQLite
c3: 15. SQLite Archive Support
c5: cli.html#sqlite_archive_support
c4:
The ".archive" dot-command and the "-A" command-line option
provide built-in support for the
SQLite Archive format. The interface is similar to
that of the "tar" command on unix systems. Each invocation of the ".ar"
command must specify a single command option. The following commands
are available for ".archive":


  OptionLong&nbsp;OptionPurpose
  -c--createCreate a new archive containing specified files.
  -x--extractExtract specified files from archive.
  -i--insertAdd files to existing archive.
  -r--removeRemove files from the archive.
  -t--listList the files in the archive.
  -u--updateAdd files to existing archive if they have changed.


As well as the command option, each invocation of ".ar" may specify
one or more modifier options. Some modifier options require an argument,
some do not. The following modifier options are available:


  OptionLong&nbsp;OptionPurpose
  -v--verboseList each file as it is processed.
  -f FILE--file FILEIf specified, use file FILE as the
  archive. Otherwise, assume that the current "main" database is the
  archive to be operated on.
  -a FILE--append FILELike --file, use file FILE as the
  archive, but open the file using the
  apndvfs VFS so that
  the archive will be appended to the end of FILE if FILE already exists.
  -C DIR--directory DIRIf specified, interpret all relative
  paths as relative to DIR, instead of the current working directory.
  -g--globUse glob(Y,X) to match arguments
  against names in the archive.
  -n--dryrunShow the SQL that would be run to carry out the
                       archive operation, but do not actually change anything.
  ----All subsequent command line words are command arguments,
  not options.



For command-line usage, add the short style command-line options immediately
following the "-A", without an intervening space.  All subsequent arguments
are considered to be part of the .archive command.  For example, the following
commands are equivalent:

sqlite3 new_archive.db -Acv file1 file2 file3
sqlite3 new_archive.db ".ar -cv file1 file2 file3"



Long and short style options may be mixed. For example, the following are
equivalent:

-- Two ways to create a new archive named "new_archive.db" containing
-- files "file1", "file2" and "file3".
.ar -c --file new_archive.db file1 file2 file3
.ar -f new_archive.db --create file1 file2 file3


Alternatively, the first argument following to ".ar" may be the concatenation
of the short form of all required options (without the "-" characters). In
this case arguments for options requiring them are read from the command line
next, and any remaining words are considered command arguments. For example:

-- Create a new archive "new_archive.db" containing files "file1" and
-- "file2" from directory "dir1".
.ar cCf dir1 new_archive.db file1 file2 file3


c2: Command Line Shell For SQLite
c3: 15.1. SQLite Archive Create Command
c5: cli.html#_sqlite_archive_create_command_
c4:
Create a new archive, overwriting any existing archive (either in the current
"main" db or in the file specified by a --file option). Each argument following
the options is a file to add to the archive. Directories are imported
recursively. See above for examples.


c2: Command Line Shell For SQLite
c3: 15.2. SQLite Archive Extract Command
c5: cli.html#_sqlite_archive_extract_command_
c4:
Extract files from the archive (either to the current working directory or
to the directory specified by a --directory option).
Files or directories whose names match the arguments,
as affected by the --glob option, are extracted.
Or, if no arguments follow the options, all files and directories are extracted.
Any specified directories are extracted recursively. It is an error if any
specified names or match patterns cannot be found in the archive.

-- Extract all files from the archive in the current "main" db to the
-- current working directory. List files as they are extracted. 
.ar --extract --verbose

-- Extract file "file1" from archive "ar.db" to directory "dir1".
.ar fCx ar.db dir1 file1

-- Extract files with ".h" extension to directory "headers".
.ar -gCx headers *.h


c2: Command Line Shell For SQLite
c3: 15.3. SQLite Archive List Command
c5: cli.html#_sqlite_archive_list_command_
c4:
List the contents of the archive. If no arguments are specified, then all
files are listed. Otherwise, only those which match the arguments,
as affected by the --glob option, are listed. Currently,
the --verbose option does not change the behaviour of this command. That may
change in the future.

-- List contents of archive in current "main" db..
.ar --list


c2: Command Line Shell For SQLite
c3: 15.4. SQLite Archive Insert And Update Commands
c5: cli.html#_sqlite_archive_insert_and_update_commands_
c4:
The --update and --insert commands work like --create command, except that
they do not delete the current archive before commencing. New versions of
files silently replace existing files with the same names, but otherwise
the initial contents of the archive (if any) remain intact.

 For the --insert command, all files listed are inserted into the archive.
For the --update command, files are only inserted if they do not previously
exist in the archive, or if their "mtime" or "mode" is different from what
is currently in the archive.

 Compatibility node:  Prior to SQLite version 3.28.0 (2019-04-16) only
the --update option was supported but that option worked like --insert in that
it always reinserted every file regardless of whether or not it had changed.


c2: Command Line Shell For SQLite
c3: 15.5. SQLite Archive Remove Command
c5: cli.html#_sqlite_archive_remove_command_
c4:
The --remove command deletes files and directories which match the
provided arguments (if any) as affected by the --glob option.
It is an error to provide arguments which match nothing in the archive.


c2: Command Line Shell For SQLite
c3: 15.6. Operations On ZIP Archives
c5: cli.html#_operations_on_zip_archives_
c4:
If FILE is a ZIP archive rather than an SQLite Archive, the ".archive"
command and the "-A" command-line option still work.  This is accomplished
using of the zipfile extension.
Hence, the following commands are roughly equivalent,
differing only in output formatting:


  Traditional CommandEquivalent sqlite3.exe Command
  unzip archive.zipsqlite3 -Axf archive.zip
  unzip -l archive.zipsqlite3 -Atvf archive.zip
  zip -r archive2.zip dirsqlite3 -Acf archive2.zip dir


c2: Command Line Shell For SQLite
c3: 15.7. SQL Used To Implement SQLite Archive Operations
c5: cli.html#_sql_used_to_implement_sqlite_archive_operations_
c4:
The various SQLite Archive Archive commands are implemented using SQL statements.
Application developers can easily add SQLite Archive Archive reading and writing
support to their own projects by running the appropriate SQL.

To see what SQL statements are used to implement an SQLite Archive
operation, add the --dryrun or -n option.  This causes the SQL to be
displayed but inhibits the execution of the SQL.

The SQL statements used to implement SQLite Archive operations make use of
various loadable extensions.  These extensions are all available in
the SQLite source tree in the
ext/misc/ subfolder.
The extensions needed for full SQLite Archive support include:



fileio.c &mdash;
This extension adds SQL functions readfile() and writefile() for
reading and writing content from files on disk.  The fileio.c
extension also includes fsdir() table-valued function for listing
the contents of a directory and the lsmode() function for converting
numeric st_mode integers from the stat() system call into human-readable
strings after the fashion of the "ls -l" command.


sqlar.c &mdash;
This extension adds the sqlar_compress() and sqlar_uncompress()
functions that are needed to compress and uncompress file content
as it is inserted and extracted from an SQLite Archive.


zipfile.c &mdash;
This extension implements the "zipfile(FILE)" table-valued function
which is used to read ZIP archives.  This extension is only needed
when reading ZIP archives instead of SQLite archives.


appendvfs.c &mdash;
This extension implements a new VFS that allows an SQLite database
to be appended to some other file, such as an executable.  This
extension is only needed if the --append option to the .archive
command is used.


c2: Command Line Shell For SQLite
c3: 16. SQL Parameters
c5: cli.html#sql_parameters
c4:
SQLite allows bound parameters to appear in an SQL statement anywhere
that a literal value is allowed.  The values for these parameters are set
using the sqlite3_bind_...() family of APIs.

Parameters can be either named or unnamed.  An unnamed parameter is a single
question mark ("?").  Named parameters are a "?" followed immediately by a number
(ex: "?15" or "?123") or one of the characters "$", ":", or "@" followed by an
alphanumeric name (ex: "$var1", ":xyz", "@bingo").

This command-line shell leaves unnamed parameters unbound, meaning that they
will have a value of an SQL NULL, but named parameters might be assigned values.
If there exists a TEMP table named "sqlite_parameters" with a schema like this:

CREATE TEMP TABLE sqlite_parameters(
  key TEXT PRIMARY KEY,
  value
) WITHOUT ROWID;


And if there is an entry in that table where the key column exactly matches
the name of parameter (including the initial "?", "$", ":", or "@" character)
then the parameter is assigned the value of the value column.  If no entry exists,
the parameter defaults to NULL.

The ".parameter" command exists to simplify managing this table.  The
".parameter init" command (often abbreviated as just ".param init") creates
the temp.sqlite_parameters table if it does not already exist.  The ".param list"
command shows all entries in the temp.sqlite_parameters table.  The ".param clear"
command drops the temp.sqlite_parameters table.  The ".param set KEY VALUE" and
".param unset KEY" commands create or delete entries from the
temp.sqlite_parameters table.

The VALUE passed to ".param set KEY VALUE" can be either a SQL literal
or any other SQL expression or query which can be evaluated to yield a value.
This allows values of differing types to be set.
If such evaluation fails, the provided VALUE is instead quoted and inserted
as text.
Because such initial evaluation may or may not fail depending upon
the VALUE content, the reliable way to get a text value is to enclose it
with single-quotes protected from the above-described command-tail parsing.
For example, (unless one intends a value of -1365):
.parameter init
.parameter set @phoneNumber "'202-456-1111'"

Note that the double-quotes serve to protect the single-quotes
and ensure that the quoted text is parsed as one argument.

The temp.sqlite_parameters table only provides values for parameters in the
command-line shell.  The temp.sqlite_parameter table has no effect on queries
that are run directly using the SQLite C-language API.  Individual applications
are expected to implement their own parameter binding.  You can search for
"sqlite_parameters" in the
command-line shell source code
to see how the command-line shell does parameter binding, and use that as
a hint for how to implement it yourself.


c2: Command Line Shell For SQLite
c3: 17. Index Recommendations (SQLite Expert)
c5: cli.html#index_recommendations_sqlite_expert_
c4:
Note: This command is experimental. It may be removed or the
interface modified in incompatible ways at some point in the future.

For most non-trivial SQL databases, the key to performance is creating
the right SQL indexes. In this context "the right SQL indexes" means those
that cause the queries that an application needs to optimize run fast. The
".expert" command can assist with this by proposing indexes that might
assist with specific queries, were they present in the database.

The ".expert" command is issued first, followed by the SQL query
on a separate line. For example, consider the following session:

sqlite&gt; CREATE TABLE x1(a, b, c);                  -- Create table in database 
sqlite&gt; .expert
sqlite&gt; SELECT * FROM x1 WHERE a=? AND b&gt;?;        -- Analyze this SELECT 
CREATE INDEX x1_idx_000123a7 ON x1(a, b);

0|0|0|SEARCH TABLE x1 USING INDEX x1_idx_000123a7 (a=? AND b&gt;?)

sqlite&gt; CREATE INDEX x1ab ON x1(a, b);             -- Create the recommended index 
sqlite&gt; .expert
sqlite&gt; SELECT * FROM x1 WHERE a=? AND b&gt;?;        -- Re-analyze the same SELECT 
(no new indexes)

0|0|0|SEARCH TABLE x1 USING INDEX x1ab (a=? AND b&gt;?)


In the above, the user creates the database schema (a single table - "x1"),
and then uses the ".expert" command to analyze a query, in this case
"SELECT * FROM x1 WHERE a=? AND b&gt;?". The shell tool recommends that the
user create a new index (index "x1_idx_000123a7") and outputs the plan
that the query would use in EXPLAIN QUERY PLAN format. The user then creates
an index with an equivalent schema and runs the analysis on the same query
again. This time the shell tool does not recommend any new indexes, and
outputs the plan that SQLite will use for the query given the existing
indexes.

The ".expert" command accepts the following options:


 Option  Purpose
 &#8209;&#8209;verbose
     If present, output a more verbose report for each query analyzed.
 &#8209;&#8209;sample&nbsp;PERCENT
     This parameter defaults to 0, causing the ".expert" command to
         recommend indexes based on the query and database schema alone.
         This is similar to the way the SQLite query planner selects
         indexes for queries if the user has not run the ANALYZE command
         on the database to generate data distribution statistics.
         
         If this option is passed a non-zero argument, the ".expert" command
         generates similar data distribution statistics for all indexes
         considered based on PERCENT percent of the rows currently stored in
         each database table. For databases with unusual data distributions,
         this may lead to better index recommendations, particularly if the
         application intends to run ANALYZE.
         
         For small databases and modern CPUs, there is usually no reason not
         to pass "--sample 100". However, gathering data distribution
         statistics can be expensive for large database tables. If the
         operation is too slow, try passing a smaller value for the --sample
         option.


The functionality described in this section may be integrated into other
applications or tools using the

SQLite expert extension code.

A database schema which incorporate SQL custom functions made available
via the extension load mechanism may need special provision to work with
the .expert feature. Because the feature uses additional connections to
implement its functionality, those custom functions must be made available
to those additional connections. This can be done by means of the extension
load/usage options described at

Automatically Load Statically Linked Extensions
and 
Persistent Loadable Extensions.


c2: Command Line Shell For SQLite
c3: 18. Working With Multiple Database Connections
c5: cli.html#working_with_multiple_database_connections
c4:
Beginning with version 3.37.0 (2021-11-27), the CLI has the ability to
hold multiple database connections open at once.  Only one database connection
is active at a time.  The inactive connections are still open but are idle.


Use the ".connection" dot-command (often abbreviated as just ".conn") to see a
list of database connections and an indication of which one is currently active.
Each database connection is identified by an integer between 0 and 9.  (There
can be at most 10 simultaneously open connections.)  Change to another database
connection, creating it if it does not already exist, by typing the ".conn"
command followed by its number.  Close a database connection by typing
".conn close N" where N is the connection number.


Though the underlying SQLite database connections are completely independent
of one another, many of the CLI settings, such as the output format, are
shared across all database connections.  Thus, changing the output mode in
one connection will change it in them all.  On the other hand, some
dot-commands such as .open only affect the current connection.


c2: Command Line Shell For SQLite
c3: 19. Miscellaneous Extension Features
c5: cli.html#miscellaneous_extension_features
c4:
The CLI is built with several SQLite extensions that are not
included with the SQLite library. A few add features
not described in the preceding sections, namely:

the UINT collating sequence which treats
unsigned integers embedded in text according to
their value, along with other text, for ordering;
decimal arithmetic as provided by the decimal extension;
the generate_series() table-valued function;




the base64() and base85() functions which encode a
blob to base64 or base85 text or decode the same to a blob; and
support for POSIX extended regular expressions
bound to the REGEXP operator.


c2: Command Line Shell For SQLite
c3: 20. Other Dot Commands
c5: cli.html#other_dot_commands
c4:
There are many other dot-commands available in the command-line
shell.  See the ".help" command for a complete list for any particular
version and build of SQLite.


c2: Command Line Shell For SQLite
c3: 21. Using sqlite3 in a shell script
c5: cli.html#using_sqlite3_in_a_shell_script
c4:
One way to use sqlite3 in a shell script is to use "echo" or
"cat" to generate a sequence of commands in a file, then invoke sqlite3
while redirecting input from the generated command file.  This
works fine and is appropriate in many circumstances.  But as
an added convenience, sqlite3 allows a single SQL command to be
entered on the command line as a second argument after the
database name.  When the sqlite3 program is launched with two
arguments, the second argument is passed to the SQLite library
for processing, the query results are printed on standard output
in list mode, and the program exits.  This mechanism is designed
to make sqlite3 easy to use in conjunction with programs like
"awk".  For example:

$ sqlite3 ex1 'select * from tbl1' \
&gt;  | awk '{printf "&lt;tr&gt;&lt;td&gt;%s&lt;td&gt;%s\n",$1,$2 }'
&lt;tr&gt;&lt;td&gt;hello&lt;td&gt;10
&lt;tr&gt;&lt;td&gt;goodbye&lt;td&gt;20
$


c2: Command Line Shell For SQLite
c3: 22. Marking The End Of An SQL Statement
c5: cli.html#marking_the_end_of_an_sql_statement
c4:
SQLite commands are normally terminated by a semicolon.  In the CLI
you can also use the word "GO" (case-insensitive) or a slash character
"/" on a line by itself to end a command.  These are used by SQL Server
and Oracle, respectively, and are supported by the SQLite CLI for
compatibility.  These won't work in sqlite3_exec(),
because the CLI translates these inputs into a semicolon before passing
them down into the SQLite core.


c2: Command Line Shell For SQLite
c3: 23. Command-line Options
c5: cli.html#command_line_options
c4:
There are many command-line options available to the CLI.  Use the --help
command-line option to see a list:

$ sqlite3 --help
Usage: ./sqlite3 &#91;OPTIONS&#93; FILENAME &#91;SQL&#93;
FILENAME is the name of an SQLite database. A new database is created
if the file does not previously exist.
OPTIONS include:
   -A ARGS...           run ".archive ARGS" and exit
   -append              append the database to the end of the file
   -ascii               set output mode to 'ascii'
   -bail                stop after hitting an error
   -batch               force batch I/O
   -box                 set output mode to 'box'
   -column              set output mode to 'column'
   -cmd COMMAND         run "COMMAND" before reading stdin
   -csv                 set output mode to 'csv'
   -deserialize         open the database using sqlite3_deserialize()
   -echo                print commands before execution
   -init FILENAME       read/process named file
   -&#91;no&#93;header          turn headers on or off
   -help                show this message
   -html                set output mode to HTML
   -interactive         force interactive I/O
   -json                set output mode to 'json'
   -line                set output mode to 'line'
   -list                set output mode to 'list'
   -lookaside SIZE N    use N entries of SZ bytes for lookaside memory
   -markdown            set output mode to 'markdown'
   -maxsize N           maximum size for a --deserialize database
   -memtrace            trace all memory allocations and deallocations
   -mmap N              default mmap size set to N
   -newline SEP         set output row separator. Default: '\n'
   -nofollow            refuse to open symbolic links to database files
   -nonce STRING        set the safe-mode escape nonce
   -nullvalue TEXT      set text string for NULL values. Default ''
   -pagecache SIZE N    use N slots of SZ bytes each for page cache memory
   -quote               set output mode to 'quote'
   -readonly            open the database read-only
   -safe                enable safe-mode
   -separator SEP       set output column separator. Default: '|'
   -stats               print memory stats before each finalize
   -table               set output mode to 'table'
   -tabs                set output mode to 'tabs'
   -no-utf8             do not setup for console I/O using UTF-8
   -version             show SQLite version
   -vfs NAME            use NAME as the default VFS
   -zip                 open the file as a ZIP Archive


The CLI is flexible regarding command-line option formatting.
Either one or two leading "-" characters are permitted.
Thus "-box" and "--box" mean the same thing.
Command-line options are processed from left to right.
Hence a "--box" option will override a prior "--quote" option.


Most of the command-line options are self-explanatory, but a few merit additional
discussion below.


c2: Command Line Shell For SQLite
c3: 23.1. The --safe command-line option
c5: cli.html#the_safe_command_line_option
c4:
The --safe command-line option attempts to disable all features of the CLI that
might cause any changes to the host computer other than changes to the specific database
file named on the command-line.  The idea is that if you receive a large SQL script
from an unknown or untrusted source, you can run that script to see what it does without
risking an exploit by using the --safe option.  The --safe option disables (among other
things):


 The .open command, unless the --hexdb option is used or the filename is ":memory:".
     This prevents the script from reading or writing any database files not named on
     the original command-line.
 The ATTACH SQL command.
 SQL functions that have potentially harmful side-effects, such as
     edit(), fts3_tokenizer(), load_extension(), readfile() and writefile().
 The .archive command.
 The .backup and .save commands.
 The .import command.
 The .load command.
 The .log command.
 The .shell and .system commands.
 The .excel, .once and .output commands.
 Other commands that can have deleterious side effects.



Basically, any feature of the CLI that reads or writes from a file on disk other
than the main database file is disabled.


c2: Command Line Shell For SQLite
c3: 23.1.1. Bypassing --safe restrictions for specific commands
c5: cli.html#bypassing_safe_restrictions_for_specific_commands
c4:
If the "--nonce NONCE" option is also included on the command-line, for some
large and arbitrary NONCE string, then the ".nonce NONCE" command (with the
same large nonce string) will permit the next SQL statement or dot-command
to bypass the --safe restrictions.

Suppose you want to run a suspicious script and the script requires one or
two of the features that --safe normally disables.  For example, suppose it
needs to ATTACH one additional database.  Or suppose the script needs to load
a specific extension. This can be accomplished by preceding the (carefully
audited) ATTACH statement or the ".load" command with an appropriate ".nonce"
command and supplying the same nonce value using the "--nonce" command-line
option.  Those specific commands will then be allowed to execute normally,
but all other unsafe commands will still be restricted.

The use of ".nonce" is dangerous in the sense that a mistake can allow a
hostile script to damage your system.  Therefore, use ".nonce" carefully,
sparingly, and as a last resort when there are no other ways to get a
script to run under --safe mode.


c2: Command Line Shell For SQLite
c3: 23.2. The --unsafe-testing command-line option
c5: cli.html#the_unsafe_testing_command_line_option
c4:
The --unsafe-testing command-line option supports use of the CLI for
internal testing of the SQLite library. It is not needed or useful for using
the CLI as a utility for creating, modifying or querying SQLite databases.
Its intended use is to permit scripted testing with direct schema changes,
defensive measures defeated, and certain special-purpose, undocumented,
test-oriented dot-commands enabled.

Misbehavior which requires use of the --unsafe-testing option to be induced
will generally not be considered a bug for that reason alone. CLI behavior
with --unsafe-testing is not supported or defined.


c2: Command Line Shell For SQLite
c3: 23.3. The --no-utf8 command-line option
c5: cli.html#the_no_utf8_command_line_option
c4:
On the Windows platform, when the console is used for input or output,
its input or output code page, or both are normally set for UTF-8 I/O
during the session, then restored afterward.
This allows copy and paste to transfer UTF-8
content properly and screen output to render properly.

This console UTF-8 I/O is not done if the CLI is invoked
with the --no-utf8 option or if the operating system
does not support UTF-8 console I/O. In either case,
I/O is translated between the CLI's internal UTF-8
encoding and the MBCS encoding in effect for the console.

In all cases, non-console text I/O is UTF-8 encoded.

On non-Windows platforms, this option is ignored and invisible.


c2: Command Line Shell For SQLite
c3: 24. Compiling the sqlite3 program from sources
c5: cli.html#compiling_the_sqlite3_program_from_sources
c4:
To compile the command-line shell on unix systems and on Windows with MinGW,
the usual configure-make command works:

sh configure; make



The configure-make works whether you are building from the canonical sources
from the source tree, or from an amalgamated bundle.  There are few
dependencies.  When building from canonical sources, a working
tclsh is required.
If using an amalgamation bundle, all the preprocessing work normally
done by tclsh will have already been carried out and only normal build
tools are required.


A working zlib compression library is
needed in order for the .archive command to operate.


On Windows with MSVC, use nmake with the Makefile.msc:

nmake /f Makefile.msc



For correct operation of the .archive command, make a copy of the
zlib source code into the compat/zlib subdirectory
of the source tree and compile this way:

nmake /f Makefile.msc USE_ZLIB=1


c2: Command Line Shell For SQLite
c3: 24.1. Do-It-Yourself Builds
c5: cli.html#_do_it_yourself_builds_
c4:
The source code to the sqlite3 command line interface is in a single
file named "shell.c".  The shell.c source file is generated from other
sources, but most of the code for shell.c can be found in
src/shell.c.in.
(Regenerate shell.c by typing "make shell.c" from the canonical source tree.)
Compile the shell.c file (together
with the sqlite3 library source code) to generate
the executable.  For example:

gcc -o sqlite3 shell.c sqlite3.c -ldl -lpthread -lz -lm



The following additional compile-time options are recommended in order to
provide a full-featured command-line shell:


 -DSQLITE_THREADSAFE=0
 -DSQLITE_ENABLE_EXPLAIN_COMMENTS
 -DSQLITE_HAVE_ZLIB
 -DSQLITE_INTROSPECTION_PRAGMAS
 -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION
 -DSQLITE_ENABLE_STMTVTAB
 -DSQLITE_ENABLE_DBPAGE_VTAB
 -DSQLITE_ENABLE_DBSTAT_VTAB
 -DSQLITE_ENABLE_OFFSET_SQL_FUNC
 -DSQLITE_ENABLE_JSON1
 -DSQLITE_ENABLE_RTREE
 -DSQLITE_ENABLE_FTS4
 -DSQLITE_ENABLE_FTS5

This page last modified on  2023-10-26 19:49:47 UTC


c2: 35% Faster Than The Filesystem
c3: 1. Summary
c5: fasterthanfs.html#summary
c4:
SQLite reads and writes small blobs (for example, thumbnail images)
35% faster&sup1; than the same blobs
can be read from or written to individual files on disk using
fread() or fwrite().

Furthermore, a single SQLite database holding
10-kilobyte blobs uses about 20% less disk space than
storing the blobs in individual files.

The performance difference arises (we believe) because when
working from an SQLite database, the open() and close() system calls
are invoked only once, whereas
open() and close() are invoked once for each blob
when using blobs stored in individual files.  It appears that the
overhead of calling open() and close() is greater than the overhead
of using the database.  The size reduction arises from the fact that
individual files are padded out to the next multiple of the filesystem
block size, whereas the blobs are packed more tightly into an SQLite
database.


The measurements in this article were made during the week of 2017-06-05
using a version of SQLite in between 3.19.2 and 3.20.0.  You may expect
future versions of SQLite to perform even better.


c2: 35% Faster Than The Filesystem
c3: 1.1. Caveats
c5: fasterthanfs.html#caveats
c4:
&sup1;The 35% figure above is approximate.  Actual timings vary
depending on hardware, operating system, and the
details of the experiment, and due to random performance fluctuations
on real-world hardware.  See the text below for more detail.
Try the experiments yourself.  Report significant deviations on
the SQLite forum.



The 35% figure is based on running tests on every machine
that the author has easily at hand.
Some reviewers of this article report that SQLite has higher 
latency than direct I/O on their systems.  We do not yet understand
the difference.  We also see indications that SQLite does not
perform as well as direct I/O when experiments are run using
a cold filesystem cache.


So let your take-away be this: read/write latency for
SQLite is competitive with read/write latency of individual files on
disk.  Often SQLite is faster.  Sometimes SQLite is almost
as fast.  Either way, this article disproves the common
assumption that a relational database must be slower than direct
filesystem I/O.


c2: 35% Faster Than The Filesystem
c3: 1.2. Related Studies
c5: fasterthanfs.html#related_studies
c4:
A 2022 study
(alternative link on GitHub) found that
SQLite is roughly twice as fast at real-world workloads compared to Btrfs and Ext4 on Linux.


Jim Gray
and others studied the read performance of BLOBs
versus file I/O for Microsoft SQL Server and found that reading BLOBs 
out of the 
database was faster for BLOB sizes less than between 250KiB and 1MiB.
(Paper).
In that study, the database still stores the filename of the content even
if the content is held in a separate file.  So the database is consulted
for every BLOB, even if it is only to extract the filename.  In this
article, the key for the BLOB is the filename, so no preliminary database
access is required.  Because the database is never used at all when
reading content from individual files in this article, the threshold
at which direct file I/O becomes faster is smaller than it is in Gray's
paper.


The Internal Versus External BLOBs article on this website is an
earlier investigation (circa 2011) that uses the same approach as the
Jim Gray paper &mdash; storing the blob filenames as entries in the
database &mdash; but for SQLite instead of SQL Server.


c2: 35% Faster Than The Filesystem
c3: 2. How These Measurements Are Made
c5: fasterthanfs.html#how_these_measurements_are_made
c4:
I/O performance is measured using the
kvtest.c program
from the SQLite source tree.
To compile this test program, first gather the kvtest.c source file
into a directory with the SQLite amalgamation source
files "sqlite3.c" and "sqlite3.h".  Then on unix, run a command like
the following:

gcc -Os -I. -DSQLITE_DIRECT_OVERFLOW_READ &#92;
  kvtest.c sqlite3.c -o kvtest -ldl -lpthread


Or on Windows with MSVC:

cl -I. -DSQLITE_DIRECT_OVERFLOW_READ kvtest.c sqlite3.c


Instructions for compiling for Android
are shown below.


Use the resulting "kvtest" program to
generate a test database with 100,000 random uncompressible
blobs, each with a random
size between 8,000 and 12,000 bytes
using a command like this:

./kvtest init test1.db --count 100k --size 10k --variance 2k



If desired, you can verify the new database by running this command:

./kvtest stat test1.db



Next, make copies of all the blobs into individual files in a directory
using a command like this:

./kvtest export test1.db test1.dir



At this point, you can measure the amount of disk space used by
the test1.db database and the space used by the test1.dir directory
and all of its content.  On a standard Ubuntu Linux desktop, the
database file will be 1,024,512,000 bytes in size and the test1.dir
directory will use 1,228,800,000 bytes of space (according to "du -k"),
about 20% more than the database.


The "test1.dir" directory created above puts all the blobs into a single
folder.  It was conjectured that some operating systems would perform 
poorly when a single directory contains 100,000 objects.  To test this,
the kvtest program can also store the blobs in a hierarchy of folders with no
more than 100 files and/or subdirectories per folder.  The alternative
on-disk representation of the blobs can be created using the --tree
command-line option to the "export" command, like this:

./kvtest export test1.db test1.tree --tree



The test1.dir directory will contain 100,000 files
with names like "000000", "000001", "000002" and so forth but the
test1.tree directory will contain the same files in subdirectories like
"00/00/00", "00/00/01", and so on.  The test1.dir and test1.test
directories take up approximately the same amount of space, though
test1.test is very slightly larger due to the extra directory entries.


All of the experiments that follow operate the same with either 
"test1.dir" or "test1.tree".  Very little performance difference is
measured in either case, regardless of operating system.


Measure the performance for reading blobs from the database and from
individual files using these commands:

./kvtest run test1.db --count 100k --blob-api
./kvtest run test1.dir --count 100k --blob-api
./kvtest run test1.tree --count 100k --blob-api



Depending on your hardware and operating system, you should see that reads 
from the test1.db database file are about 35% faster than reads from 
individual files in the test1.dir or test1.tree folders.  Results can vary
significantly from one run to the next due to caching, so it is advisable
to run tests multiple times and take an average or a worst case or a best
case, depending on your requirements.

The --blob-api option on the database read test causes kvtest to use
the sqlite3_blob_read() feature of SQLite to load the content of the
blobs, rather than running pure SQL statements.  This helps SQLite to run
a little faster on read tests.  You can omit that option to compare the
performance of SQLite running SQL statements.
In that case, the SQLite still out-performs direct reads, though
by not as much as when using sqlite3_blob_read().
The --blob-api option is ignored for tests that read from individual disk
files.


Measure write performance by adding the --update option.  This causes
the blobs are overwritten in place with another random blob of
exactly the same size.

./kvtest run test1.db --count 100k --update
./kvtest run test1.dir --count 100k --update
./kvtest run test1.tree --count 100k --update



The writing test above is not completely fair, since SQLite is doing
power-safe transactions whereas the direct-to-disk writing is not.
To put the tests on a more equal footing, add either the --nosync
option to the SQLite writes to disable calling fsync() or
FlushFileBuffers() to force content to disk, or using the --fsync option
for the direct-to-disk tests to force them to invoke fsync() or
FlushFileBuffers() when updating disk files.


By default, kvtest runs the database I/O measurements all within
a single transaction.  Use the --multitrans option to run each blob
read or write in a separate transaction.  The --multitrans option makes
SQLite much slower, and uncompetitive with direct disk I/O.  This
option proves, yet again, that to get the most performance out of
SQLite, you should group as much database interaction as possible within
a single transaction.


There are many other testing options, which can be seen by running
the command:

./kvtest help


c2: 35% Faster Than The Filesystem
c3: 2.1. Read Performance Measurements
c5: fasterthanfs.html#read_performance_measurements
c4:
The chart below shows data collected using 
kvtest.c on five different
systems:


Win7: A circa-2009 Dell Inspiron laptop, Pentium dual-core
    at 2.30GHz, 4GiB RAM, Windows7.
Win10: A 2016 Lenovo YOGA 910, Intel i7-7500 at 2.70GHz,
    16GiB RAM, Windows10.
Mac: A 2015 MacBook Pro, 3.1GHz intel Core i7, 16GiB RAM,
    MacOS 10.12.5
Ubuntu: Desktop built from Intel i7-4770K at 3.50GHz, 32GiB RAM,
    Ubuntu 16.04.2 LTS
Android: Galaxy S3, ARMv7, 2GiB RAM


All machines use SSD except Win7 which has a
hard-drive. The test database is 100K blobs with sizes uniformly
distributed between 8K and 12K, for a total of about 1 gigabyte
of content.  The database page size
is 4KiB.  The -DSQLITE_DIRECT_OVERFLOW_READ compile-time option was
used for all of these tests.
Tests were run multiple times.
The first run was used to warm up the cache and its timings were discarded.


The chart below shows average time to read a blob directly from the
filesystem versus the time needed to read the same blob from the SQLite 
database.
The actual timings vary considerably from one system to another 
(the Ubuntu desktop is much
faster than the Galaxy S3 phone, for example).  
This chart shows the ratio of the
times needed to read blobs from a file divided by the time needed to
from the database.  The left-most column in the chart is the normalized
time to read from the database, for reference.


In this chart, an SQL statement ("SELECT v FROM kv WHERE k=?1") 
is prepared once.  Then for each blob, the blob key value is bound 
to the ?1 parameter and the statement is evaluated to extract the
blob content.


The chart shows that on Windows10, content can be read from the SQLite
database about 5 times faster than it can be read directly from disk.
On Android, SQLite is only about 35% faster than reading from disk.






Chart 1:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg 10KB each, random order using SQL



The performance can be improved slightly by bypassing the SQL layer
and reading the blob content directly using the
sqlite3_blob_read() interface, as shown in the next chart:






Chart 2:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg size 10KB, random order
using sqlite3_blob_read().



Further performance improves can be made by using the
memory-mapped I/O feature of SQLite.  In the next chart, the
entire 1GB database file is memory mapped and blobs are read
(in random order) using the sqlite3_blob_read() interface.
With these optimizations, SQLite is twice as fast as Android
or MacOS-X and over 10 times faster than Windows.






Chart 3:  SQLite read latency relative to direct filesystem reads.
100K blobs, avg size 10KB, random order
using sqlite3_blob_read() from a memory-mapped database.



The third chart shows that reading blob content out of SQLite can be
twice as fast as reading from individual files on disk for Mac and
Android, and an amazing ten times faster for Windows.


c2: 35% Faster Than The Filesystem
c3: 2.2. Write Performance Measurements
c5: fasterthanfs.html#write_performance_measurements
c4:
Writes are slower.
On all systems, using both direct I/O and SQLite, write performance is
between 5 and 15 times slower than reads.


Write performance measurements were made by replacing (overwriting)
an entire blob with a different blob.  All of the blobs in these
experiment are random and incompressible.  Because writes are so much
slower than reads, only 10,000 of the 100,000 blobs in the database
are replaced.  The blobs to be replaced are selected at random and
are in no particular order.


The direct-to-disk writes are accomplished using fopen()/fwrite()/fclose().
By default, and in all the results shown below, the OS filesystem buffers are
never flushed to persistent storage using fsync() or
FlushFileBuffers().  In other words, there is no attempt to make the
direct-to-disk writes transactional or power-safe.
We found that invoking fsync() or FlushFileBuffers() on each file
written causes direct-to-disk storage
to be about 10 times or more slower than writes to SQLite.


The next chart compares SQLite database updates in WAL mode
against raw direct-to-disk overwrites of separate files on disk.
The PRAGMA synchronous setting is NORMAL.
All database writes are in a single transaction.
The timer for the database writes is stopped after the transaction
commits, but before a checkpoint is run.
Note that the SQLite writes, unlike the direct-to-disk writes,
are transactional and power-safe, though because the synchronous
setting is NORMAL instead of FULL, the transactions are not durable.






Chart 4:  SQLite write latency relative to direct filesystem writes.
10K blobs, avg size 10KB, random order,
WAL mode with synchronous NORMAL,
exclusive of checkpoint time



The android performance numbers for the write experiments are omitted
because the performance tests on the Galaxy S3 are so random.  Two
consecutive runs of the exact same experiment would give wildly different
times.  And, to be fair, the performance of SQLite on android is slightly
slower than writing directly to disk.


The next chart shows the performance of SQLite versus direct-to-disk
when transactions are disabled (PRAGMA journal_mode=OFF)
and PRAGMA synchronous is set to OFF.  These settings put SQLite on an
equal footing with direct-to-disk writes, which is to say they make the
data prone to corruption due to system crashes and power failures.






Chart 5:  SQLite write latency relative to direct filesystem writes.
10K blobs, avg size 10KB, random order,
journaling disabled, synchronous OFF.



In all of the write tests, it is important to disable anti-virus software
prior to running the direct-to-disk performance tests.  We found that
anti-virus software slows down direct-to-disk by an order of magnitude
whereas it impacts SQLite writes very little.  This is probably due to the
fact that direct-to-disk changes thousands of separate files which all need
to be checked by anti-virus, whereas SQLite writes only changes the single
database file.


c2: 35% Faster Than The Filesystem
c3: 2.3. Variations
c5: fasterthanfs.html#variations
c4:
The -DSQLITE_DIRECT_OVERFLOW_READ compile-time option causes SQLite
to bypass its page cache when reading content from overflow pages.  This
helps database reads of 10K blobs run a little faster, but not all that much
faster.  SQLite still holds a speed advantage over direct filesystem reads
without the SQLITE_DIRECT_OVERFLOW_READ compile-time option.

Other compile-time options such as using -O3 instead of -Os or
using -DSQLITE_THREADSAFE=0 and/or some of the other
recommended compile-time options might help SQLite to run even faster
relative to direct filesystem reads.

The size of the blobs in the test data affects performance.
The filesystem will generally be faster for larger blobs, since
the overhead of open() and close() is amortized over more bytes of I/O,
whereas the database will be more efficient in both speed and space
as the average blob size decreases.


c2: 35% Faster Than The Filesystem
c3: 3. General Findings
c5: fasterthanfs.html#general_findings
c4:
SQLite is competitive with, and usually faster than, blobs stored in
separate files on disk, for both reading and writing.


SQLite is much faster than direct writes to disk on Windows
when anti-virus protection is turned on.  Since anti-virus software
is and should be on by default in Windows, that means that SQLite
is generally much faster than direct disk writes on Windows.


Reading is about an order of magnitude faster than writing, for all
systems and for both SQLite and direct-to-disk I/O.


I/O performance varies widely depending on operating system and hardware.
Make your own measurements before drawing conclusions.


Some other SQL database engines advise developers to store blobs in separate
files and then store the filename in the database.  In that case, where
the database must first be consulted to find the filename before opening
and reading the file, simply storing the entire blob in the database
gives much faster read and write performance with SQLite.
See the Internal Versus External BLOBs article for more information.


c2: 35% Faster Than The Filesystem
c3: 4.1. Compiling And Testing on Android
c5: fasterthanfs.html#compiling_and_testing_on_android
c4:
The kvtest program is compiled and run on Android as follows.
First install the Android SDK and NDK.  Then prepare a script
named "android-gcc" that looks approximately like this:

#!/bin/sh
#
NDK=/home/drh/Android/Sdk/ndk-bundle
SYSROOT=$NDK/platforms/android-16/arch-arm
ABIN=$NDK/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin
GCC=$ABIN/arm-linux-androideabi-gcc
$GCC --sysroot=$SYSROOT -fPIC -pie $*


Make that script executable and put it on your $PATH.  Then
compile the kvtest program as follows:

android-gcc -Os -I. kvtest.c sqlite3.c -o kvtest-android


Next, move the resulting kvtest-android executable to the Android
device:

adb push kvtest-android /data/local/tmp


Finally use "adb shell" to get a shell prompt on the Android device,
cd into the /data/local/tmp directory, and begin running the tests
as with any other unix host.
This page last modified on  2023-10-28 11:35:47 UTC


c2: Dynamic Memory Allocation In SQLite
c3: Overview
c5: malloc.html#overview
c4:
SQLite uses dynamic memory allocation to obtain
memory for storing various objects
(ex: database connections and prepared statements) and to build
a memory cache of the database file and to hold the results of queries.
Much effort has gone into making the dynamic memory allocation subsystem
of SQLite reliable, predictable, robust, secure, and efficient.

This document provides an overview of dynamic memory allocation within 
SQLite.  The target audience is software engineers who are tuning their
use of SQLite for peak performance in demanding environments.
Nothing in this document is required knowledge for using SQLite.  The
default settings and configuration for SQLite will work well in most
applications.  However, the information contained in this document may
be useful to engineers who are tuning SQLite to comply with special
requirements or to run under unusual circumstances.


c2: Dynamic Memory Allocation In SQLite
c3: 1. Features
c5: malloc.html#_features
c4:
The SQLite core and its memory allocation subsystem provides the 
following capabilities:



Robust against allocation failures.
If a memory allocation ever fails (that is to say, 
if malloc() or realloc() ever return NULL)
then SQLite will recover gracefully.   SQLite will first attempt
to free memory from unpinned cache pages then retry the allocation
request.  
Failing that, SQLite will either stop what
it is doing and return the
SQLITE_NOMEM error code back up to the application or it will
make do without the requested memory.



No memory leaks.
The application is responsible for destroying any objects it allocates.
(For example, the application must use sqlite3_finalize() on 
every prepared statement and sqlite3_close() on every 
database connection.)   But as long as
the application cooperates, SQLite will never leak memory.  This is
true even in the face of memory allocation failures or other system
errors.



Memory usage limits.
The sqlite3_soft_heap_limit64() mechanism allows the application to
set a memory usage limit that SQLite strives to stay below.  SQLite
will attempt to reuse memory from its caches rather than allocating new
memory as it approaches the soft limit.



Zero-malloc option.
The application can optionally provide SQLite with several buffers of bulk memory
at startup and SQLite will then use those provided buffers for all of
its memory allocation needs and never call system malloc() or free().



Application-supplied memory allocators.
The application can provide SQLite with pointers to alternative 
memory allocators at start-time.  The alternative memory allocator
will be used in place of system malloc() and free().



Proof against breakdown and fragmentation.
SQLite can be configured so that, subject to certain usage constraints
detailed below, it is guaranteed to never fail a memory allocation
or fragment the heap.
This property is important to long-running, high-reliability
embedded systems where a memory allocation error could contribute
to an overall system failure.



Memory usage statistics.
Applications can see how much memory they are using and detect when
memory usage is approaching or exceeding design boundaries.




Plays well with memory debuggers.
Memory allocation in SQLite is structured so that standard
third-party memory debuggers (such as dmalloc or 
valgrind) can be used to verify correct
memory allocation behavior.


Minimal calls to the allocator.
The system malloc() and free() implementations are inefficient
on many systems.  SQLite strives to reduce overall processing time
by minimizing its use of malloc() and free().



Open access.
Pluggable SQLite extensions or even the application itself can 
access to the same underlying memory allocation
routines used by SQLite through the
sqlite3_malloc(), sqlite3_realloc(), and sqlite3_free() interfaces.


c2: Dynamic Memory Allocation In SQLite
c3: 2. Testing
c5: malloc.html#_testing
c4:
Most of the code in the SQLite source tree is devoted purely to 
testing and verification.  Reliability is important to SQLite.
Among the tasks of the test infrastructure is to ensure that
SQLite does not misuse dynamically allocated memory, that SQLite
does not leak memory, and that SQLite responds
correctly to a dynamic memory allocation failure.

The test infrastructure verifies that SQLite does not misuse
dynamically allocated memory by using a specially instrumented
memory allocator.  The instrumented memory allocator is enabled
at compile-time using the SQLITE_MEMDEBUG option.  The instrumented
memory allocator is much slower than the default memory allocator and
so its use is not recommended in production.  But when
enabled during testing, 
the instrumented memory allocator performs the following checks:


Bounds checking.
The instrumented memory allocator places sentinel values at both ends
of each memory allocation to verify that nothing within SQLite writes
outside the bounds of the allocation.

Use of memory after freeing.
When each block of memory is freed, every byte is overwritten with a
nonsense bit pattern.  This helps to ensure that no memory is ever
used after having been freed.

Freeing memory not obtained from malloc.
Each memory allocation from the instrumented memory allocator contains
sentinels used to verify that every allocation freed came
from prior malloc.

Uninitialized memory.
The instrumented memory allocator initializes each memory allocation
to a nonsense bit pattern to help ensure that the user makes no
assumptions about the content of allocation memory.


Regardless of whether or not the instrumented memory allocator is
used, SQLite keeps track of how much memory is currently checked out.
There are hundreds of test scripts used for testing SQLite.  At the
end of each script, all objects are destroyed and a test is made to
ensure that all  memory has been freed.  This is how memory
leaks are detected.  Notice that memory leak detection is in force at
all times, during test builds and during production builds.  Whenever
one of the developers runs any individual test script, memory leak
detection is active.  Hence memory leaks that do arise during development
are quickly detected and fixed.


The response of SQLite to out-of-memory (OOM) errors is tested using
a specialized memory allocator overlay that can simulate memory failures.
The overlay is a layer that is inserted in between the memory allocator
and the rest of SQLite.  The overlay passes most memory allocation
requests straight through to the underlying allocator and passes the
results back up to the requester.  But the overlay can be set to 
cause the Nth memory allocation to fail.  To run an OOM test, the overlay
is first set to fail on the first allocation attempt.  Then some test
script is run and verification that the allocation was correctly caught
and handled is made.  Then the overlay is set to fail on the second
allocation and the test repeats.  The failure point continues to advance
one allocation at a time until the entire test procedure runs to
completion without hitting a memory allocation error.  This whole
test sequence run twice.  On the first pass, the
overlay is set to fail only the Nth allocation.  On the second pass,
the overlay is set to fail the Nth and all subsequent allocations.

Note that the memory leak detection logic continues to work even
when the OOM overlay is being used.  This verifies that SQLite
does not leak memory even when it encounters memory allocation errors.
Note also that the OOM overlay can work with any underlying memory
allocator, including the instrumented memory allocator that checks
for memory allocation misuse.  In this way it is verified that 
OOM errors do not induce other kinds of memory usage errors.

Finally, we observe that the instrumented memory allocator and the
memory leak detector both work over the entire SQLite test suite and
the TCL test suite provides over 99% statement test coverage and that
the TH3 test harness provides 100% branch test coverage
with no leak leaks. This is
strong evidence that dynamic memory allocation is used correctly
everywhere within SQLite.


c2: Dynamic Memory Allocation In SQLite
c3: 2.1. Use of reallocarray()
c5: malloc.html#_use_of_reallocarray_
c4:
The reallocarray() interface is a recent innovation (circa 2014)
from the OpenBSD community that grow out of efforts to prevent the
next "heartbleed" bug by avoiding 32-bit integer
arithmetic overflow on memory allocation size computations.  The
reallocarray() function has both unit-size and count parameters.
To allocate memory sufficient to hold an array of N elements each X-bytes
in size, one calls "reallocarray(0,X,N)".  This is preferred over
the traditional technique of invoking "malloc(X*N)" as reallocarray()
eliminates the risk that the X*N multiplication will overflow and
cause malloc() to return a buffer that is a different size from what
the application expected.

SQLite does not use reallocarray().  The reason is that reallocarray()
is not useful to SQLite.  It turns out that SQLite never does memory
allocations that are the simple product of two integers.  Instead, SQLite
does allocations of the form "X+C" or "N*X+C" or "M*N*X+C" or
"N*X+M*Y+C", and so forth.  The reallocarray() interface is not helpful
in avoiding integer overflow in those cases.

Nevertheless, integer overflow in the computation of memory allocation
sizes is a concern that SQLite would like to deal with.  To prevent
problems, all SQLite internal memory allocations occur using thin wrapper
functions that take a signed 64-bit integer size parameter.  The SQLite 
source code is audited to ensure that all size computations are carried 
out using 64-bit signed integers as well.  SQLite will
refuse to allocate more than about 2GB of memory at one go.  (In common
use, SQLite seldom ever allocates more than about 8KB of memory at a time
so a 2GB allocation limit is not a burden.)  So the 64-bit size parameter
provides lots of headroom for detecting overflows.  The same audit that
verifies that all size computations are done as 64-bit signed integers
also verifies that it is impossible to overflow a 64-bit integer
during the computation.

The code audits used to ensure that memory allocation size computations
do not overflow in SQLite are repeated prior to every SQLite release.


c2: Dynamic Memory Allocation In SQLite
c3: 3. Configuration
c5: malloc.html#_configuration
c4:
The default memory allocation settings in SQLite are appropriate
for most applications.  However, applications with unusual or particularly
strict requirements may want to adjust the configuration to more closely
align SQLite to their needs.
Both compile-time and start-time configuration options are available.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1. Alternative low-level memory allocators
c5: malloc.html#_alternative_low_level_memory_allocators
c4:
The SQLite source code includes several different memory allocation
modules that can be selected at compile-time, or to a limited extent
at start-time.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.1. The default memory allocator
c5: malloc.html#the_default_memory_allocator
c4:
By default, SQLite uses the malloc(), realloc(), and free() routines
from the standard C library for its memory allocation needs.  These routines
are surrounded by a thin wrapper that also provides a "memsize()" function
that will return the size of an existing allocation.  The memsize() function
is needed to keep an accurate count of the number of bytes of outstanding
memory; memsize() determines how many bytes to remove from the outstanding
count when an allocation is freed.  The default allocator implements
memsize() by always allocating 8 extra bytes on each malloc() request and
storing the size of the allocation in that 8-byte header.

The default memory allocator is recommended for most applications.
If you do not have a compelling need to use an alternative memory
allocator, then use the default.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.2. The debugging memory allocator
c5: malloc.html#the_debugging_memory_allocator
c4:
If SQLite is compiled with the SQLITE_MEMDEBUG compile-time option,
then a different, heavy wrapper is used around system malloc(), realloc(), 
and free().
The heavy wrapper allocates around 100 bytes of extra space
with each allocation.  The extra space is used to place sentinel values 
at both ends of the allocation returned to the SQLite core.  When an
allocation is freed,
these sentinels are checked to make sure the SQLite core did not overrun
the buffer in either direction.  When the system library is GLIBC, the 
heavy wrapper also makes use of the GNU backtrace() function to examine
the stack and record the ancestor functions of the malloc() call.  When
running the SQLite test suite, the heavy wrapper also records the name of
the current test case.  These latter two features are useful for
tracking down the source of memory leaks detected by the test suite.

The heavy wrapper that is used when SQLITE_MEMDEBUG is set also
makes sure each new allocation is filled with nonsense data prior to
returning the allocation to the caller.  And as soon as an allocation
is free, it is again filled with nonsense data.  These two actions help
to ensure that the SQLite core does not make assumptions about the state
of newly allocated memory and that memory allocations are not used after
they have been freed.

The heavy wrapper employed by SQLITE_MEMDEBUG is intended for use
only during testing, analysis, and debugging of SQLite.  The heavy wrapper
has a significant performance and memory overhead and probably should not
be used in production.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.3. The Win32 native memory allocator
c5: malloc.html#the_win32_native_memory_allocator
c4:
If SQLite is compiled for Windows with the SQLITE_WIN32_MALLOC
compile-time option, then a different, thin wrapper is used around
HeapAlloc(), HeapReAlloc(), and HeapFree().  The thin wrapper uses the
configured SQLite heap, which will be different from the default process
heap if the SQLITE_WIN32_HEAP_CREATE compile-time option is used.  In
addition, when an allocation is made or freed, HeapValidate() will be
called if SQLite is compiled with assert() enabled and the
SQLITE_WIN32_MALLOC_VALIDATE compile-time option.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.4. Zero-malloc memory allocator
c5: malloc.html#zero_malloc_memory_allocator
c4:
When SQLite is compiled with the SQLITE_ENABLE_MEMSYS5 option, an
alternative memory allocator that does not use malloc() is included in the
build.  The SQLite developers refer to this alternative memory allocator
as "memsys5".  Even when it is included in the build, memsys5 is 
disabled by default.
To enable memsys5, the application must invoke the following SQLite 
interface at start-time:


sqlite3_config(SQLITE_CONFIG_HEAP, pBuf, szBuf, mnReq);


In the call above, pBuf is a pointer to a large, contiguous chunk
of memory space that SQLite will use to satisfy all of its memory
allocation needs.   pBuf might point to a static array or it might
be memory obtained from some other application-specific mechanism.
szBuf is an integer that is the number of bytes of memory space
pointed to by pBuf.  mnReq is another integer that is the
minimum size of an allocation.  Any call to sqlite3_malloc(N) where
N is less than mnReq will be rounded up to mnReq.  mnReq must be
a power of two.  We shall see later that the mnReq parameter is
important in reducing the value of n and hence the minimum memory
size requirement in the Robson proof.

The memsys5 allocator is designed for use on embedded systems, 
though there is nothing to prevent its use on workstations.
The szBuf is typically between a few hundred kilobytes up to a few
dozen megabytes, depending on system requirements and memory budget.

The algorithm used by memsys5 can be called "power-of-two,
first-fit".  The sizes of all memory allocation 
requests are rounded up to a power of two and the request is satisfied
by the first free slot in pBuf that is large enough.  Adjacent freed
allocations are coalesced using a buddy system. When used appropriately,
this algorithm provides mathematical guarantees against fragmentation and
breakdown, as described further below.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.5. Experimental memory allocators
c5: malloc.html#experimental_memory_allocators
c4:
The name "memsys5" used for the zero-malloc memory allocator implies
that there are several additional memory allocators available, and indeed
there are.  The default memory allocator is "memsys1".  The debugging
memory allocator is "memsys2".  Those have already been covered.

If SQLite is compiled with SQLITE_ENABLE_MEMSYS3 then another
zero-malloc memory allocator, similar to memsys5, is included in the
source tree.  The memsys3 allocator, like memsys5, must be activated
by a call to sqlite3_config(SQLITE_CONFIG_HEAP,...).  Memsys3
uses the memory buffer supplied as its source for all memory allocations.
The difference between memsys3 and memsys5 is that memsys3 uses a
different memory allocation algorithm that seems to work well in
practice, but which does not provide mathematical
guarantees against memory fragmentation and breakdown.  Memsys3 was
a predecessor to memsys5.  The SQLite developers now believe that 
memsys5 is superior to
memsys3 and that all applications that need a zero-malloc memory
allocator should use memsys5 in preference to memsys3.  Memsys3 is
considered both experimental and deprecated and will likely be removed 
from the source tree in a future release of SQLite.

Memsys4 and memsys6 were experimental memory allocators
introduced in around 2007 and subsequently removed from the
source tree in around 2008, after it became clear that they
added no new value.

Other experimental memory allocators might be added in future releases
of SQLite.  One may anticipate that these will be called memsys7, memsys8,
and so forth.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.6. Application-defined memory allocators
c5: malloc.html#application_defined_memory_allocators
c4:
New memory allocators do not have to be part of the SQLite source tree
nor included in the sqlite3.c amalgamation.  Individual applications can
supply their own memory allocators to SQLite at start-time.

To cause SQLite to use a new memory allocator, the application
simply calls:


sqlite3_config(SQLITE_CONFIG_MALLOC, pMem);


In the call above, pMem is a pointer to an sqlite3_mem_methods object
that defines the interface to the application-specific memory allocator.
The sqlite3_mem_methods object is really just a structure containing
pointers to functions to implement the various memory allocation primitives.


In a multi-threaded application, access to the sqlite3_mem_methods
is serialized if and only if SQLITE_CONFIG_MEMSTATUS is enabled.
If SQLITE_CONFIG_MEMSTATUS is disabled then the methods in
sqlite3_mem_methods must take care of their own serialization needs.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.7. Memory allocator overlays
c5: malloc.html#memory_allocator_overlays
c4:
An application can insert layers or "overlays" in between the
SQLite core and the underlying memory allocator.
For example, the out-of-memory test logic
for SQLite uses an overlay that can simulate memory allocation
failures.

An overlay can be created by using the


sqlite3_config(SQLITE_CONFIG_GETMALLOC, pOldMem);


interface to obtain pointers to the existing memory allocator.
The existing allocator is saved by the overlay and is used as
a fallback to do real memory allocation.  Then the overlay is
inserted in place of the existing memory allocator using
the sqlite3_config(SQLITE_CONFIG_MALLOC,...) as described
above.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.8. No-op memory allocator stub
c5: malloc.html#no_op_memory_allocator_stub
c4:
If SQLite is compiled with the SQLITE_ZERO_MALLOC option, then
the default memory allocator is omitted and replaced by a stub
memory allocator that never allocates any memory.  Any calls to the
stub memory allocator will report back that no memory is available.

The no-op memory allocator is not useful by itself.  It exists only
as a placeholder so that SQLite has a memory allocator to link against
on systems that may not have malloc(), free(), or realloc() in their
standard library.
An application that is compiled with SQLITE_ZERO_MALLOC will need to
use sqlite3_config() together with SQLITE_CONFIG_MALLOC or
SQLITE_CONFIG_HEAP to specify a new alternative memory allocator
before beginning to use SQLite.


c2: Dynamic Memory Allocation In SQLite
c3: 3.2. Page cache memory
c5: malloc.html#_page_cache_memory
c4:
In most applications, the database page cache subsystem within 
SQLite uses more dynamically allocated memory than all other parts
of SQLite combined.  It is not unusual to see the database page cache
consume over 10 times more memory than the rest of SQLite combined.

SQLite can be configured to make page cache memory allocations from
a separate and distinct memory pool of fixed-size
slots.  This can have two advantages:



Because allocations are all the same size, the memory allocator can
operate much faster.  The allocator need not bother with coalescing 
adjacent free slots or searching for a slot
of an appropriate size.  All unallocated memory slots can be stored on
a linked list.  Allocating consists of removing the first entry from the
list.  Deallocating is simply adding an entry to the beginning of the list.



With a single allocation size, the n parameter in the
Robson proof is 1, and the total memory space required by the allocator
(N) is exactly equal to maximum memory used (M).  
No additional memory is required to cover fragmentation overhead, thus 
reducing memory requirements.  This is particularly important for the
page cache memory since the page cache constitutes the largest component
of the memory needs of SQLite.



The page-cache memory allocator is disabled by default.
An application can enable it at start-time as follows:


sqlite3_config(SQLITE_CONFIG_PAGECACHE, pBuf, sz, N);


The pBuf parameter is a pointer to a contiguous range of bytes that
SQLite will use for page-cache memory allocations.  The buffer must be
at least sz*N bytes in size.  The "sz" parameter
is the size of each page-cache allocation.  N is the maximum 
number of available allocations.

If SQLite needs a page-cache entry that is larger than "sz" bytes or
if it needs more than N entries, it falls back to using the
general-purpose memory allocator.


c2: Dynamic Memory Allocation In SQLite
c3: 3.3. Lookaside memory allocator
c5: malloc.html#_lookaside_memory_allocator
c4:
SQLite database connections make many
small and short-lived memory allocations.
This occurs most commonly when compiling SQL statements using
sqlite3_prepare_v2() but also to a lesser extent when running
prepared statements using sqlite3_step().  These small memory
allocations are used to hold things such as the names of tables
and columns, parse tree nodes, individual query results values,
and B-Tree cursor objects.  There are consequently
many calls to malloc() and free() - so many calls that malloc() and
free() end up using a significant fraction of the CPU time assigned
to SQLite.

SQLite version 3.6.1 (2008-08-06)
introduced the lookaside memory allocator to
help reduce the memory allocation load.  In the lookaside allocator,
each database connection preallocates a single large chunk of memory
(typically in the range of 60 to 120 kilobytes) and divides that chunk
up into small fixed-size "slots" of around 100 to 1000 byte each.  This
becomes the lookaside memory pool.  Thereafter, memory allocations
associated with the database connection and that are not too large
are satisfied using one of the lookaside pool slots rather than by calling
the general-purpose memory allocator.  Larger allocations continue to
use the general-purpose memory allocator, as do allocations that occur
when the lookaside pool slots are all checked out.  
But in many cases, the memory
allocations are small enough and there are few enough outstanding that
the new memory requests can be satisfied from the lookaside
pool.

Because lookaside allocations are always the same size, the allocation
and deallocation algorithms are very quick.  There is no
need to coalesce adjacent free slots or search for a slot
of a particular size.  Each database connection maintains a singly-linked
list of unused slots.  Allocation requests simply pull the first
element of this list.  Deallocations simply push the element back onto
the front of the list.
Furthermore, each database connection is assumed to already be
running in a single thread (there are mutexes already in
place to enforce this) so no additional mutexing is required to 
serialize access to the lookaside slot freelist.
Consequently, lookaside memory
allocations and deallocations are very fast.  In speed tests on
Linux and Mac OS X workstations, SQLite has shown overall performance
improvements as high as 10% and 15%, depending on the workload how
and lookaside is configured.

The size of the lookaside memory pool has a global default value
but can also be configured on a connection-by-connection basis.
To change the default size of the lookaside memory pool at
compile-time, use the 
-DSQLITE_DEFAULT_LOOKASIDE=SZ,N
option.
To change the default size of the lookaside memory pool at
start-time, use the sqlite3_config() interface:


sqlite3_config(SQLITE_CONFIG_LOOKASIDE, sz, cnt);


The "sz" parameter is the size in bytes of each lookaside slot.
The "cnt" parameter is
the total number of lookaside memory slots per database connection.
The total amount
of lookaside memory allocated to each database connection is
sz*cnt bytes. 


The lookaside pool can be changed for an individual
database connection "db" using this call:


sqlite3_db_config(db, SQLITE_DBCONFIG_LOOKASIDE, pBuf, sz, cnt);


The "pBuf" parameter is a pointer to memory space that will be
used for the lookaside memory pool.  If pBuf is NULL, then SQLite
will obtain its own space for the memory pool using sqlite3_malloc().
The "sz" and "cnt" parameters are the size of each lookaside slot
and the number of slots, respectively.  If pBuf is not NULL, then it
must point to at least sz*cnt bytes of memory.

The lookaside configuration can only be changed while there are
no outstanding lookaside allocations for the database connection.
Hence, the configuration should be set immediately after creating the 
database connection using sqlite3_open() (or equivalent) and before
evaluating any SQL statements on the connection.


c2: Dynamic Memory Allocation In SQLite
c3: 3.3.1. Two-Size Lookaside
c5: malloc.html#two_size_lookaside
c4:
Beginning with SQLite version 3.31.0 (2020-01-22),
lookaside supports two memory pools, each with a different size
slot.  The small-slot pool uses 128-byte slots and the large-slot
pool uses whatever size is specified by SQLITE_DBCONFIG_LOOKASIDE
(defaulting to 1200 bytes).  Splitting the pool in two like this
allows memory allocations to be covered by lookaside more often
while at the same time reducing per-database-connection heap usage
from 120KB down to 48KB.


Configuration continues to use the SQLITE_DBCONFIG_LOOKASIDE or
SQLITE_CONFIG_LOOKASIDE configuration options, as described above,
with parameters "sz" and "cnt".  The total heap space used for
lookaside continues to be sz*cnt bytes.  But the space is allocated
between the small-slot lookaside and big-slot lookaside, with
preference given to small-slot lookaside.  The total number of
slots will usually exceed "cnt", since "sz" is typically much
larger than the small-slot size of 128 bytes.


The default lookaside configuration has changed from 100 slots
of 1200 bytes each (120KB) to be 40 slots of 1200 bytes each
(48KB).  This space ends up being allocated as 93 slots of
128 bytes each and 30 slots of 1200 bytes each.  So more lookaside
slots are available but much less heap space is used.


The default lookaside configuration, the size of the small-slots,
and the details of how heap space is allocated between small-slots
and big-slots, are all subject to change from one release to the
next.


c2: Dynamic Memory Allocation In SQLite
c3: 3.4. Memory status
c5: malloc.html#_memory_status
c4:
By default, SQLite keeps statistics on its memory usage.  These
statistics are useful in helping to determine how much memory an
application really needs.  The statistics can also be used in
high-reliability system to determine
if the memory usage is coming close to or exceeding the limits 
of the Robson proof and hence that the memory allocation subsystem is 
liable to breakdown.

Most memory statistics are global, and therefore the tracking of
statistics must be serialized with a mutex.  Statistics are turned 
on by default, but an option exists to disable them.  By disabling 
memory statistics,
SQLite avoids entering and leaving a mutex on each memory allocation
and deallocation.  That savings can be noticeable on systems where
mutex operations are expensive.  To disable memory statistics, the
following interface is used at start-time:


sqlite3_config(SQLITE_CONFIG_MEMSTATUS, onoff);


The "onoff" parameter is true to enable the tracking of memory
statistics and false to disable statistics tracking.

Assuming statistics are enabled, the following routine can be used
to access them:


sqlite3_status(verb, &amp;current, &amp;highwater, resetflag);


The "verb" argument determines what statistic is accessed.
There are various verbs defined.  The
list is expected to grow as the sqlite3_status() interface matures.
The current value the selected parameter is written into integer 
"current" and the highest historical value
is written into integer "highwater".  If resetflag is true, then
the high-water mark is reset down to the current value after the call
returns.

A different interface is used to find statistics associated with a
single database connection:


sqlite3_db_status(db, verb, &amp;current, &amp;highwater, resetflag);


This interface is similar except that it takes a pointer to
a database connection as its first argument and returns statistics about
that one object rather than about the entire SQLite library.
The sqlite3_db_status() interface currently only recognizes a
single verb SQLITE_DBSTATUS_LOOKASIDE_USED, though additional verbs
may be added in the future.

The per-connection statistics do not use global variables and hence
do not require mutexes to update or access.  Consequently the
per-connection statistics continue to function even if
SQLITE_CONFIG_MEMSTATUS is turned off.


c2: Dynamic Memory Allocation In SQLite
c3: 3.5. Setting memory usage limits
c5: malloc.html#_setting_memory_usage_limits
c4:
The sqlite3_soft_heap_limit64() interface can be used to set an
upper bound on the total amount of outstanding memory that the
general-purpose memory allocator for SQLite will allow to be outstanding
at one time.  If attempts are made to allocate more memory than specified
by the soft heap limit, then SQLite will first attempt to free cache
memory before continuing with the allocation request.  The soft heap
limit mechanism only works if memory statistics are enabled and
it works best
if the SQLite library is compiled with the SQLITE_ENABLE_MEMORY_MANAGEMENT
compile-time option.

The soft heap limit is "soft" in this sense:  If SQLite is not able
to free up enough auxiliary memory to stay below the limit, it goes
ahead and allocates the extra memory and exceeds its limit.  This occurs
under the theory that it is better to use additional memory than to fail
outright.

As of SQLite version 3.6.1 (2008-08-06), 
the soft heap limit only applies to the
general-purpose memory allocator.  The soft heap limit does not know
about or interact with
the pagecache memory allocator or the lookaside memory allocator.
This deficiency will likely be addressed in a future release.


c2: Dynamic Memory Allocation In SQLite
c3: 4. Mathematical Guarantees Against Memory Allocation Failures
c5: malloc.html#_mathematical_guarantees_against_memory_allocation_failures
c4:
The problem of dynamic memory allocation, and specifically the
problem of a memory allocator breakdown, has been studied by
J. M. Robson and the results published as:


J. M. Robson.  "Bounds for Some Functions Concerning Dynamic
Storage Allocation".  Journal of the Association for
Computing Machinery, Volume 21, Number 8, July 1974,
pages 491-499.


Let us use the following notation (similar but not identical to
Robson's notation):



N

The amount of raw memory needed by the memory allocation system
in order to guarantee that no memory allocation will ever fail.

M

The maximum amount of memory that the application ever has checked out
at any point in time.

n

The ratio of the largest memory allocation to the smallest.  We assume
that every memory allocation size is an integer multiple of the smallest memory
allocation size.




Robson proves the following result:


N = M*(1 + (log2 n)/2) - n + 1


Colloquially, the Robson proof shows that in order to guarantee
breakdown-free operation, any memory allocator must use a memory pool
of size N which exceeds the maximum amount of memory ever
used M by a multiplier that depends on n, 
the ratio of the largest to the smallest allocation size.  In other
words, unless all memory allocations are of exactly the same size
(n=1) then the system needs access to more memory than it will
ever use at one time.  Furthermore, we see that the amount of surplus
memory required grows rapidly as the ratio of largest to smallest
allocations increases, and so there is strong incentive to keep all
allocations as near to the same size as possible.

Robson's proof is constructive. 
He provides an algorithm for computing a sequence of allocation
and deallocation operations that will lead to an allocation failure due to
memory fragmentation if available memory is as much as one byte
less than N.
And, Robson shows that a power-of-two first-fit memory allocator
(such as implemented by memsys5) will never fail a memory allocation
provided that available memory is N or more bytes.

The values M and n are properties of the application.
If an application is constructed in such a way that both M and
n are known, or at least have known upper bounds, and if the
application uses
the memsys5 memory allocator and is provided with N bytes of
available memory space using SQLITE_CONFIG_HEAP
then Robson proves that no memory allocation request will ever fail
within the application.
To put this another way, the application developer can select a value
for N that will guarantee that no call to any SQLite interface
will ever return SQLITE_NOMEM.  The memory pool will never become
so fragmented that a new memory allocation request cannot be satisfied.
This is an important property for
applications where a software fault could cause injury, physical harm, or
loss of irreplaceable data.


c2: Dynamic Memory Allocation In SQLite
c3: 4.1. Computing and controlling parameters M and n
c5: malloc.html#_computing_and_controlling_parameters_m_and_n
c4:
The Robson proof applies separately to each of the memory allocators
used by SQLite:


The general-purpose memory allocator (memsys5).
The pagecache memory allocator.
The lookaside memory allocator.


For allocators other than memsys5,
all memory allocations are of the same size.  Hence, n=1
and therefore N=M.  In other words, the memory pool need
be no larger than the largest amount of memory in use at any given moment.

The usage of pagecache memory is somewhat harder to control in
SQLite version 3.6.1, though mechanisms are planned for subsequent
releases that will make controlling pagecache memory much easier.
Prior to the introduction of these new mechanisms, the only way
to control pagecache memory is using the cache_size pragma.

Safety-critical applications will usually want to modify the
default lookaside memory configuration so that when the initial
lookaside memory buffer is allocated during sqlite3_open() the
resulting memory allocation is not so large as to force the n
parameter to be too large.  In order to keep n under control,
it is best to try to keep the largest memory allocation below 2 or 4
kilobytes.  Hence, a reasonable default setup for the lookaside
memory allocator might any one of the following:


sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 32, 32);  /* 1K */
sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 64, 32);  /* 2K */
sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 32, 64);  /* 2K */
sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 64, 64);  /* 4K */


Another approach is to initially disable the lookaside memory
allocator:


sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 0, 0);


Then let the application maintain a separate pool of larger
lookaside memory buffers that it can distribute to database connections
as they are created.  In the common case, the application will only
have a single database connection and so the lookaside memory pool
can consist of a single large buffer.


sqlite3_db_config(db, SQLITE_DBCONFIG_LOOKASIDE, aStatic, 256, 500);


The lookaside memory allocator is really intended as performance
optimization, not as a method for assuring breakdown-free memory allocation,
so it is not unreasonable to completely disable the lookaside memory
allocator for safety-critical operations.

The general purpose memory allocator is the most difficult memory pool
to manage because it supports allocations of varying sizes.  Since 
n is a multiplier on M we want to keep n as small
as possible.  This argues for keeping the minimum allocation size for
memsys5 as large as possible.  In most applications, the
lookaside memory allocator is able to handle small allocations.  So
it is reasonable to set the minimum allocation size for memsys5 to
2, 4 or even 8 times the maximum size of a lookaside allocation.  
A minimum allocation size of 512 is a reasonable setting.

Further to keeping n small, one desires to keep the size of
the largest memory allocations under control.
Large requests to the general-purpose memory allocator
might come from several sources:


SQL table rows that contain large strings or BLOBs.
Complex SQL queries that compile down to large prepared statements.
SQL parser objects used internally by sqlite3_prepare_v2().
Storage space for database connection objects.
Page cache memory allocations that overflow into the general-purpose
    memory allocator.
Lookaside buffer allocations for new database connections.


The last two allocations can be controlled and/or eliminated by
configuring the pagecache memory allocator,
and lookaside memory allocator appropriately, as described above.
The storage space required for database connection objects depends
to some extent on the length of the filename of the database file, but
rarely exceeds 2KB on 32-bit systems.  (More space is required on
64-bit systems due to the increased size of pointers.)
Each parser object uses about 1.6KB of memory.  Thus, elements 3 through 6
above can easily be controlled to keep the maximum memory allocation
size below 2KB.

If the application is designed to manage data in small pieces,
then the database should never contain any large strings or BLOBs
and hence element 1 above should not be a factor.  If the database
does contain large strings or BLOBs, they should be read using
incremental BLOB I/O and rows that contain the
large strings or BLOBs should never be update by any means other
than incremental BLOB I/O.  Otherwise, the 
sqlite3_step() routine will need to read the entire row into
contiguous memory at some point, and that will involve at least
one large memory allocation.

The final source of large memory allocations is the space to hold
the prepared statements that result from compiling complex SQL
operations.  Ongoing work by the SQLite developers is reducing the
amount of space required here.  But large and complex queries might
still require prepared statements that are several kilobytes in
size.  The only workaround at the moment is for the application to
break complex SQL operations up into two or more smaller and simpler 
operations contained in separate prepared statements.

All things considered, applications should normally be able to
hold their maximum memory allocation size below 2K or 4K.  This
gives a value for log2(n) of 2 or 3.  This will
limit N to between 2 and 2.5 times M.

The maximum amount of general-purpose memory needed by the application
is determined by such factors as how many simultaneous open 
database connection and prepared statement objects the application
uses, and on the complexity of the prepared statements.  For any
given application, these factors are normally fixed and can be
determined experimentally using SQLITE_STATUS_MEMORY_USED.
A typical application might only use about 40KB of general-purpose
memory.  This gives a value of N of around 100KB.


c2: Dynamic Memory Allocation In SQLite
c3: 4.2. Ductile failure
c5: malloc.html#_ductile_failure
c4:
If the memory allocation subsystems within SQLite are configured
for breakdown-free operation but the actual memory usage exceeds
design limits set by the Robson proof, SQLite will usually continue 
to operate normally.
The pagecache memory allocator
and the lookaside memory allocator automatically failover
to the memsys5 general-purpose memory allocator.  And it is usually the
case that the memsys5 memory allocator will continue to function
without fragmentation even if M and/or n exceeds the limits
imposed by the Robson proof.  The Robson proof shows that it is 
possible for a memory allocation to break down and fail in this 
circumstance, but such a failure requires an especially
despicable sequence of allocations and deallocations - a sequence that
SQLite has never been observed to follow.  So in practice it is usually
the case that the limits imposed by Robson can be exceeded by a
considerable margin with no ill effect.

Nevertheless, application developers are admonished to monitor
the state of the memory allocation subsystems and raise alarms when
memory usage approaches or exceeds Robson limits.  In this way,
the application will provide operators with abundant warning well
in advance of failure.
The memory statistics interfaces of SQLite provide the application with
all the mechanism necessary to complete the monitoring portion of
this task.


c2: Dynamic Memory Allocation In SQLite
c3: 5. Stability Of Memory Interfaces
c5: malloc.html#_stability_of_memory_interfaces
c4:
Update: As of SQLite version 3.7.0 (2010-07-21), 
all of SQLite memory allocation interfaces
are considered stable and will be supported in future releases.
This page last modified on  2022-04-18 02:55:50 UTC


c2: The dbhash.exe Utility Program
c3: 1. Overview
c5: dbhash.html#overview
c4:
The dbhash (or dbhash.exe on Windows) utility is a
command-line program that computes the SHA1 hash of the schema and content 
for an SQLite database.

Dbhash ignores extraneous formatting details and hashes only the database
schema and content.  Hence the hash is constant even if the database file
is modified by:


 VACUUM
 PRAGMA page_size
 PRAGMA journal_mode
 REINDEX
 ANALYZE
 copied via the backup API
 ... and so forth


The operations above can potentially cause vast changes the raw database file,
and hence cause very different SHA1 hashes at the file level.
But since the content represented in the database file is unchanged by these
operations, the hash computed by dbhash is also unchanged.

Dbhash can be used to compare two databases to confirm that they
are equivalent, even though their representation on disk is quite different.
Dbhash might also be used to verify the content of a remote database without having
to transmit the entire content of the remote database over a slow link.


c2: The dbhash.exe Utility Program
c3: 2. Usage
c5: dbhash.html#usage
c4:
Dbhash is a command-line utility.
To run it, type "dbhash" on a command-line prompt followed by the names of
one or more SQLite database files that are to be hashed.
The database hashes will be displayed on standard output.
For example:

drh@bella:~/sqlite/bld$ dbhash ~/Fossils/sqlite.fossil
8d3da9ff87196312aaa33076627ccb7943ef79e3 /home/drh/Fossils/sqlite.fossil


Dbhash supports command-line options that can restrict the tables of the
database file that are hashed, or restrict the hash to only content or only
the schema.  Run "dbhash --help" for further information.


c2: The dbhash.exe Utility Program
c3: 3. Building
c5: dbhash.html#building
c4:
To build a copy of the dbhash utility program on unix, get a copy of the
canonical SQLite source code and enter:

./configure
make dbhash


On Windows, enter:

nmake /f makefile.msc dbhash.exe


The dbhash program is implemented by a single file of C-code
called dbhash.c.
To build the dbhash program manually, simply compile the dbhash.c source file
and link it against the SQLite library.
This page last modified on  2022-01-08 05:02:57 UTC


c2: Database File Format
c5: fileformat.html#
c4:
This document describes and defines the on-disk database file
format used by all releases of SQLite since 
version 3.0.0 (2004-06-18).


c2: Database File Format
c3: 1. The Database File
c5: fileformat.html#the_database_file
c4:
The complete state of an SQLite database is usually
contained in a single file on disk called the "main database file".

During a transaction, SQLite stores additional information 
in a second file called the "rollback journal", or if SQLite is in
WAL mode, a write-ahead log file.


c2: Database File Format
c3: 1.1. Hot Journals
c5: fileformat.html#hot_journals
c4:
If the application or
host computer crashes before the transaction completes, then the rollback
journal or write-ahead log contains information needed 
to restore the main database file to a consistent state.  When a rollback 
journal or write-ahead log contains information necessary for recovering 
the state of the database, they are called a "hot journal" or "hot WAL file".
Hot journals and WAL files are only a factor during error recovery
scenarios and so are uncommon, but they are part of the state of an SQLite
database and so cannot be ignored.  This document defines the format
of a rollback journal and the write-ahead log file, but the focus is
on the main database file.


c2: Database File Format
c3: 1.2. Pages
c5: fileformat.html#pages
c4:
The main database file consists of one or more pages.  The size of a
page is a power of two between 512 and 65536 inclusive.  All pages within
the same database are the same size.  The page size for a database file
is determined by the 2-byte integer located at an offset of
16 bytes from the beginning of the database file.

Pages are numbered beginning with 1.  The maximum page number is
4294967294 (232 - 2).  The minimum size
SQLite database is a single 512-byte page.
The maximum size database would be 4294967294 pages at 65536 bytes per
page or 281,474,976,579,584 bytes (about 281 terabytes).  Usually SQLite will
hit the maximum file size limit of the underlying filesystem or disk
hardware long before it hits its own internal size limit.

In common use, SQLite databases tend to range in size from a few kilobytes
to a few gigabytes, though terabyte-size SQLite databases are known to exist
in production.

At any point in time, every page in the main database has a single
use which is one of the following:

The lock-byte page
A freelist page

A freelist trunk page
A freelist leaf page

A b-tree page

A table b-tree interior page
A table b-tree leaf page
An index b-tree interior page
An index b-tree leaf page

A payload overflow page
A pointer map page



All reads from and writes to the main database file begin at a page
boundary and all writes are an integer number of pages in size.  Reads
are also usually an integer number of pages in size, with the one exception
that when the database is first opened, the first 100 bytes of the
database file (the database file header) are read as a sub-page size unit.

Before any information-bearing page of the database is modified, 
the original unmodified content of that page is written into the 
rollback journal.  If a transaction is interrupted and needs to be 
rolled back, the rollback journal can then be used to restore the
database to its original state.  Freelist leaf pages bear no
information that would need to be restored on a rollback and so they
are not written to the journal prior to modification, in order to
reduce disk I/O.


c2: Database File Format
c3: 1.3. The Database Header
c5: fileformat.html#the_database_header
c4:
The first 100 bytes of the database file comprise the database file 
header.  The database file header is divided into fields as shown by
the table below.  All multibyte fields in the database file header are
stored with the most significant byte first (big-endian).


Database Header Format

OffsetSizeDescription
016
The header string: "SQLite format 3\000"
162
The database page size in bytes.  Must be a power of two between 512
and 32768 inclusive, or the value 1 representing a page size of 65536.
181
File format write version.  1 for legacy; 2 for WAL.
191
File format read version.  1 for legacy; 2 for WAL.
201
Bytes of unused "reserved" space at the end of each page.  Usually 0.
211
Maximum embedded payload fraction.  Must be 64.
221
Minimum embedded payload fraction.  Must be 32.
231
Leaf payload fraction.  Must be 32.
244
File change counter.
284
Size of the database file in pages.  The "in-header database size".
324
Page number of the first freelist trunk page.
364
Total number of freelist pages.
404
The schema cookie.
444
The schema format number.  Supported schema formats are 1, 2, 3, and 4.
484
Default page cache size.
524
The page number of the largest root b-tree page when in auto-vacuum or
incremental-vacuum modes, or zero otherwise.
564
The database text encoding.  A value of 1 means UTF-8.  A value of 2
means UTF-16le.  A value of 3 means UTF-16be.
604
The "user version" as read and set by the user_version pragma.
644
True (non-zero) for incremental-vacuum mode.  False (zero) otherwise.
684
The "Application ID" set by PRAGMA application_id.
7220
Reserved for expansion.  Must be zero.
924
The version-valid-for number.
964
SQLITE_VERSION_NUMBER


c2: Database File Format
c3: 1.3.1. Magic Header String
c5: fileformat.html#magic_header_string
c4:
Every valid SQLite database file begins with the following 16 bytes 
(in hex): 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00.  This byte sequence
corresponds to the UTF-8 string "SQLite format 3" including the nul
terminator character at the end.


c2: Database File Format
c3: 1.3.2. Page Size
c5: fileformat.html#page_size
c4:
The two-byte value beginning at offset 16 determines the page size of 
the database.  For SQLite versions 3.7.0.1 (2010-08-04)
and earlier, this value is 
interpreted as a big-endian integer and must be a power of two between
512 and 32768, inclusive.  Beginning with SQLite version 3.7.1
(2010-08-23), a page
size of 65536 bytes is supported.  The value 65536 will not fit in a
two-byte integer, so to specify a 65536-byte page size, the value
at offset 16 is 0x00 0x01.
This value can be interpreted as a big-endian
1 and thought of as a magic number to represent the 65536 page size.
Or one can view the two-byte field as a little endian number and say
that it represents the page size divided by 256.  These two 
interpretations of the page-size field are equivalent.


c2: Database File Format
c3: 1.3.3. File format version numbers
c5: fileformat.html#file_format_version_numbers
c4:
The file format write version and file format read version at offsets
18 and 19 are intended to allow for enhancements of the file format
in future versions of SQLite.  In current versions of SQLite, both of
these values are 1 for rollback journalling modes and 2 for WAL
journalling mode.  If a version of SQLite coded to the current
file format specification encounters a database file where the read
version is 1 or 2 but the write version is greater than 2, then the database
file must be treated as read-only.  If a database file with a read version
greater than 2 is encountered, then that database cannot be read or written.


c2: Database File Format
c3: 1.3.4. Reserved bytes per page
c5: fileformat.html#reserved_bytes_per_page
c4:
SQLite has the ability to set aside a small number of extra bytes at
the end of every page for use by extensions.  These extra bytes are
used, for example, by the SQLite Encryption Extension to store a nonce
and/or cryptographic checksum associated with each page.  The 
"reserved space" size in the 1-byte integer at offset 20 is the number
of bytes of space at the end of each page to reserve for extensions.
This value is usually 0.  The value can be odd.



The "usable size" of a database page is the page size specified by the
2-byte integer at offset 16 in the header less the "reserved" space size
recorded in the 1-byte integer at offset 20 in the header.  The usable
size of a page might be an odd number.  However, the usable size is not
allowed to be less than 480.  In other words, if the page size is 512,
then the reserved space size cannot exceed 32.


c2: Database File Format
c3: 1.3.5. Payload fractions
c5: fileformat.html#payload_fractions
c4:
The maximum and minimum embedded payload fractions and the leaf
payload fraction values must be 64, 32, and 32.  These values were
originally intended to be tunable parameters that could be used to
modify the storage format of the b-tree algorithm.  However, that
functionality is not supported and there are no current plans to add
support in the future.  Hence, these three bytes are fixed at the
values specified.


c2: Database File Format
c3: 1.3.6. File change counter
c5: fileformat.html#file_change_counter
c4:
The file change counter is a 4-byte big-endian integer at
offset 24 that is incremented whenever the database file is unlocked
after having been modified.
When two or more processes are reading the same database file, each 
process can detect database changes from other processes by monitoring 
the change counter.
A process will normally want to flush its database page cache when
another process modified the database, since the cache has become stale.
The file change counter facilitates this.

In WAL mode, changes to the database are detected using the wal-index
and so the change counter is not needed.  Hence, the change counter might
not be incremented on each transaction in WAL mode.


c2: Database File Format
c3: 1.3.7. In-header database size
c5: fileformat.html#in_header_database_size
c4:
The 4-byte big-endian integer at offset 28 into the header 
stores the size of the database file in pages.  If this in-header
datasize size is not valid (see the next paragraph), then the database 
size is computed by looking
at the actual size of the database file. Older versions of SQLite
ignored the in-header database size and used the actual file size
exclusively.  Newer versions of SQLite use the in-header database
size if it is available but fall back to the actual file size if
the in-header database size is not valid.

The in-header database size is only considered to be valid if
it is non-zero and if the 4-byte change counter at offset 24
exactly matches the 4-byte version-valid-for number at offset 92.
The in-header database size is always valid 
when the database is only modified using recent versions of SQLite,
versions 3.7.0 (2010-07-21) and later.
If a legacy version of SQLite writes to the database, it will not
know to update the in-header database size and so the in-header
database size could be incorrect.  But legacy versions of SQLite
will also leave the version-valid-for number at offset 92 unchanged
so it will not match the change-counter.  Hence, invalid in-header
database sizes can be detected (and ignored) by observing when
the change-counter does not match the version-valid-for number.


c2: Database File Format
c3: 1.3.8. Free page list
c5: fileformat.html#free_page_list
c4:
Unused pages in the database file are stored on a freelist.  The
4-byte big-endian integer at offset 32 stores the page number of
the first page of the freelist, or zero if the freelist is empty.
The 4-byte big-endian integer at offset 36 stores the total 
number of pages on the freelist.


c2: Database File Format
c3: 1.3.9. Schema cookie
c5: fileformat.html#schema_cookie
c4:
The schema cookie is a 4-byte big-endian integer at offset 40
that is incremented whenever the database schema changes.  A 
prepared statement is compiled against a specific version of the
database schema.  When the database schema changes, the statement
must be reprepared.  When a prepared statement runs, it first checks
the schema cookie to ensure the value is the same as when the statement
was prepared and if the schema cookie has changed, the statement either
automatically reprepares and reruns or it aborts with an SQLITE_SCHEMA 
error.


c2: Database File Format
c3: 1.3.10. Schema format number
c5: fileformat.html#schema_format_number
c4:
The schema format number is a 4-byte big-endian integer at offset 44.
The schema format number is similar to the file format read and write
version numbers at offsets 18 and 19 except that the schema format number
refers to the high-level SQL formatting rather than the low-level b-tree
formatting.  Four schema format numbers are currently defined:


Format 1 is understood by all versions of SQLite back to
version 3.0.0 (2004-06-18).
Format 2 adds the ability of rows within the same table
to have a varying number of columns, in order to support the
ALTER TABLE ... ADD COLUMN functionality.  Support for
reading and writing format 2 was added in SQLite 
version 3.1.3 on 2005-02-20.
Format 3 adds the ability of extra columns added by
ALTER TABLE ... ADD COLUMN to have non-NULL default
values.  This capability was added in SQLite version 3.1.4
on 2005-03-11.
Format 4 causes SQLite to respect the
DESC keyword on
index declarations.  (The DESC keyword is ignored in indexes for 
formats 1, 2, and 3.)
Format 4 also adds two new boolean record type values (serial types
8 and 9).  Support for format 4 was added in SQLite 3.3.0 on
2006-01-10.


New database files created by SQLite use format 4 by default.
The legacy_file_format pragma can be used to cause SQLite
to create new database files using format 1.
The format version number can be made to default to 1 instead of 4 by
setting SQLITE_DEFAULT_FILE_FORMAT=1 at compile-time.


c2: Database File Format
c3: 1.3.11. Suggested cache size
c5: fileformat.html#suggested_cache_size
c4:
The 4-byte big-endian signed integer at offset 48 is the suggested
cache size in pages for the database file.  The value is a suggestion
only and SQLite is under no obligation to honor it.  The absolute value
of the integer is used as the suggested size.  The suggested cache size
can be set using the default_cache_size pragma.


c2: Database File Format
c3: 1.3.12. Incremental vacuum settings
c5: fileformat.html#incremental_vacuum_settings
c4:
The two 4-byte big-endian integers at offsets 52 and 64 are used
to manage the auto_vacuum and incremental_vacuum modes.  If
the integer at offset 52 is zero then pointer-map (ptrmap) pages are
omitted from the database file and neither auto_vacuum nor
incremental_vacuum are supported.  If the integer at offset 52 is
non-zero then it is the page number of the largest root page in the
database file, the database file will contain ptrmap pages, and the
mode must be either auto_vacuum or incremental_vacuum.  In this latter
case, the integer at offset 64 is true for incremental_vacuum and
false for auto_vacuum.  If the integer at offset 52 is zero then
the integer at offset 64 must also be zero.


c2: Database File Format
c3: 1.3.13. Text encoding
c5: fileformat.html#text_encoding
c4:
The 4-byte big-endian integer at offset 56 determines the encoding
used for all text strings stored in the database.  
A value of 1 means UTF-8.
A value of 2 means UTF-16le.
A value of 3 means UTF-16be.
No other values are allowed.
The sqlite3.h header file defines C-preprocessor macros SQLITE_UTF8 as 1,
SQLITE_UTF16LE as 2, and SQLITE_UTF16BE as 3, to use in place of
the numeric codes for the text encoding.


c2: Database File Format
c3: 1.3.14. User version number
c5: fileformat.html#user_version_number
c4:
The 4-byte big-endian integer at offset 60 is the user version which
is set and queried by the user_version pragma.  The user version is
not used by SQLite.


c2: Database File Format
c3: 1.3.15. Application ID
c5: fileformat.html#application_id
c4:
The 4-byte big-endian integer at offset 68 is an "Application ID" that
can be set by the PRAGMA application_id command in order to identify the
database as belonging to or associated with a particular application.
The application ID is intended for database files used as an
application file-format.  The application ID can be used by utilities 
such as file(1) to determine the specific
file type rather than just reporting "SQLite3 Database".  A list of
assigned application IDs can be seen by consulting the
magic.txt
file in the SQLite source repository.


c2: Database File Format
c3: 1.3.16. Write library version number and version-valid-for number
c5: fileformat.html#write_library_version_number_and_version_valid_for_number
c4:
The 4-byte big-endian integer at offset 96 stores the 
SQLITE_VERSION_NUMBER value for the SQLite library that most
recently modified the database file.  The 4-byte big-endian integer at
offset 92 is the value of the change counter when the version number
was stored.  The integer at offset 92 indicates which transaction
the version number is valid for and is sometimes called the
"version-valid-for number".


c2: Database File Format
c3: 1.3.17. Header space reserved for expansion
c5: fileformat.html#header_space_reserved_for_expansion
c4:
All other bytes of the database file header are reserved for
future expansion and must be set to zero.


c2: Database File Format
c3: 1.4. The Lock-Byte Page
c5: fileformat.html#the_lock_byte_page
c4:
The lock-byte page is the single page of the database file
that contains the bytes at offsets between 1073741824 and 1073742335,
inclusive.  A database file that is less than or equal to 1073741824 bytes 
in size contains no lock-byte page.  A database file larger than
1073741824 contains exactly one lock-byte page.


The lock-byte page is set aside for use by the operating-system specific
VFS implementation in implementing the database file locking primitives.
SQLite does not use the lock-byte page.  The SQLite core 
will never read or write the lock-byte page,
though operating-system specific VFS 
implementations may choose to read or write bytes on the lock-byte 
page according to the 
needs and proclivities of the underlying system.  The unix and win32
VFS implementations that come built into SQLite do not write to the
lock-byte page, but third-party VFS implementations for
other operating systems might.

The lock-byte page arose from the need to support Win95 which was the
predominant operating system when this file format was designed and which 
only supported mandatory file locking.  All modern operating systems that
we know of support advisory file locking, and so the lock-byte page is
not really needed any more, but is retained for backwards compatibility.


c2: Database File Format
c3: 1.5. The Freelist
c5: fileformat.html#the_freelist
c4:
A database file might contain one or more pages that are not in
active use.  Unused pages can come about, for example, when information
is deleted from the database.  Unused pages are stored on the freelist
and are reused when additional pages are required.

The freelist is organized as a linked list of freelist trunk pages
with each trunk page containing page numbers for zero or more freelist
leaf pages.

A freelist trunk page consists of an array of 4-byte big-endian integers.
The size of the array is as many integers as will fit in the usable space
of a page.  The minimum usable space is 480 bytes so the array will always
be at least 120 entries in length.  The first integer on a freelist trunk
page is the page number of the next freelist trunk page in the list or zero 
if this is the last freelist trunk page.  The second integer on a freelist
trunk page is the number of leaf page pointers to follow.  
Call the second integer on a freelist trunk page L.
If L is greater than zero then integers with array indexes between 2 and
L+1 inclusive contain page numbers for freelist leaf pages.

Freelist leaf pages contain no information.  SQLite avoids reading or
writing freelist leaf pages in order to reduce disk I/O.

A bug in SQLite versions prior to 3.6.0 (2008-07-16)
caused the database to be
reported as corrupt if any of the last 6 entries in the freelist trunk page 
array contained non-zero values.  Newer versions of SQLite do not have
this problem.  However, newer versions of SQLite still avoid using the 
last six entries in the freelist trunk page array in order that database
files created by newer versions of SQLite can be read by older versions
of SQLite.

The number of freelist pages is stored as a 4-byte big-endian integer
in the database header at an offset of 36 from the beginning of the file.
The database header also stores the page number of the first freelist trunk
page as a 4-byte big-endian integer at an offset of 32 from the beginning
of the file.


c2: Database File Format
c3: 1.6. B-tree Pages
c5: fileformat.html#b_tree_pages
c4:
The b-tree algorithm provides key/data storage with unique and
ordered keys on page-oriented storage devices.
For background information on b-trees, see
Knuth, The Art Of Computer Programming, Volume 3 "Sorting
and Searching", pages 471-479.  Two variants of b-trees are used by
SQLite.  "Table b-trees" use a 64-bit signed integer key and store
all data in the leaves.  "Index b-trees" use arbitrary keys and store no
data at all.

A b-tree page is either an interior page or a leaf page.
A leaf page contains keys and in the case of a table b-tree each
key has associated data.  An interior page contains
K keys together with K+1 pointers to child b-tree pages.
A "pointer" in an interior b-tree page is just the 32-bit 
unsigned integer page number of the child page.

The number of keys on an interior b-tree page, K,
is almost always at least 2 and is usually much more than 2.
The only exception is when page 1 is an interior b-tree page.
Page 1 has 100 fewer bytes of storage space available,
due to the presence of the database header at the beginning of that page,
and so sometimes (rarely) if page 1 is an interior b-tree page, it can
end up holding just a a single key.  In all other cases, K is 2 or more.
The upper bound on K is as many keys as will fit on the page. Large keys
on index b-trees are split up into overflow pages so that no single key
uses more than one fourth of the available storage space on the page
and hence every internal page is able to store at least 4 keys.
The integer keys of table b-trees are never large enough to
require overflow, so key overflow only occurs on index b-trees.

Define the depth
of a leaf b-tree to be 1 and the depth of any interior b-tree to be one
more than the maximum depth of any of its children.  In a well-formed
database, all children of an interior b-tree have the same depth.

In an interior b-tree page, the pointers and keys logically alternate 
with a pointer on both ends. (The previous sentence is to be understood
conceptually - the actual layout of the keys and
pointers within the page is more complicated and will be described in
the sequel.)  All keys within the same page are unique and are logically
organized in ascending order from left to right.  (Again, this ordering
is logical, not physical.  The actual location of keys within the page
is arbitrary.) For any key X, pointers to the left
of a X refer to b-tree pages on which all keys are less than or equal to X.
Pointers to the right of X refer to pages where all keys are 
greater than X.

Within an interior b-tree page, each key and the pointer to its
immediate left are combined into a structure called a "cell".  The
right-most pointer is held separately.  A leaf b-tree page has no
pointers, but it still uses the cell structure to hold keys for
index b-trees or keys and content for table b-trees.  Data is also
contained in the cell.


Every b-tree page has at most one parent b-tree page.
A b-tree page without a parent is called a root page.  A root b-tree page
together with the closure of its children form a complete b-tree.
It is possible (and in fact rather common) to have a complete b-tree
that consists of a single page that is both a leaf and the root.
Because there are pointers from parents to children, every page of a
complete b-tree can be located if only the root page is known.  Hence,
b-trees are identified by their root page number.



A b-tree page is either a table b-tree page or an index b-tree page.
All pages within each complete b-tree are of the same type: either table
or index.  There is one table b-trees in the database file
for each rowid table in the database schema, including system tables
such as sqlite_schema.  There is one index b-tree
in the database file for each index in the schema, including implied indexes
created by uniqueness constraints.  There are no b-trees associated with
virtual tables.  Specific virtual table implementations might make use
of shadow tables for storage, but those shadow tables will have separate
entries in the database schema.  WITHOUT ROWID tables use index b-trees
rather than a table b-trees, so there is one
index b-tree in the database file for each WITHOUT ROWID table.
The b-tree corresponding to the sqlite_schema table is always a table
b-tree and always has a root page of 1.
The sqlite_schema table contains the root page number for every other 
table and index in the database file.

Each entry in a table b-tree consists of a 64-bit signed integer key
and up to 2147483647 bytes of arbitrary data.  (The key of a table b-tree
corresponds to the rowid of the SQL table that the b-tree implements.)
Interior table b-trees hold only keys and pointers to children.
All data is contained in the table b-tree leaves.

Each entry in an index b-tree consists of an arbitrary key of up
to 2147483647 bytes in length and no data.



Define the "payload" of a cell to be the arbitrary length section
of the cell.  For an index b-tree, the key is always arbitrary in length
and hence the payload is the key.  There are no arbitrary length elements
in the cells of interior table b-tree pages and so those cells have no
payload.  Table b-tree leaf pages contain arbitrary length content and
so for cells on those pages the payload is the content.
When the size of payload for a cell exceeds a certain threshold (to
be defined later) then only the first few bytes of the payload
are stored on the b-tree page and the balance is stored in a linked list
of content overflow pages.

A b-tree page is divided into regions in the following order:


The 100-byte database file header (found on page 1 only)
The 8 or 12 byte b-tree page header
The cell pointer array
Unallocated space
The cell content area
The reserved region.



The 100-byte database file header is found only on page 1, which is
always a table b-tree page.  All other b-tree pages in the database file
omit this 100-byte header.

The reserved region is an area of unused space at the end of every
page (except the locking page) that extensions can use to hold per-page
information.  The size of the reserved region is determined by the one-byte
unsigned integer found at an offset of 20 into the database file header.
The size of the reserved region is usually zero.

The b-tree page header is 8 bytes in size for leaf pages and 12
bytes for interior pages.  All multibyte values in the page header
are big-endian.
The b-tree page header is composed of the following fields:


B-tree Page Header Format

OffsetSizeDescription
01
The one-byte flag at offset 0 indicating the b-tree page type.
A value of 2 (0x02) means the page is an interior index b-tree page.
A value of 5 (0x05) means the page is an interior table b-tree page.
A value of 10 (0x0a) means the page is a leaf index b-tree page.
A value of 13 (0x0d) means the page is a leaf table b-tree page.
Any other value for the b-tree page type is an error.
12
The two-byte integer at offset 1 gives the start of the
first freeblock on the page, or is zero if there are no freeblocks.
32
The two-byte integer at offset 3 gives the number of cells on the page.
52
The two-byte integer at offset 5 designates the start of the cell content
area.  A zero value for this integer is interpreted as 65536.
71
The one-byte integer at offset 7 gives the number of fragmented free
bytes within the cell content area.
84
The four-byte page number at offset 8 is the right-most pointer.  This
value appears in the header of interior b-tree pages only and is omitted from
all other pages.


The cell pointer array of a b-tree page immediately follows the b-tree
page header.  Let K be the number of cells on the btree.  The cell pointer
array consists of K 2-byte integer offsets to the cell contents.  The
cell pointers are arranged in key order with left-most cell (the cell with the
smallest key) first and the right-most cell (the cell with the largest
key) last.

Cell content is stored in the cell content region of the b-tree page.
SQLite strives to place cells as far toward the end of the b-tree page as
it can, in order to leave space for future growth of the cell pointer array.
The area in between the last cell pointer array entry and the beginning of
the first cell is the unallocated region.


If a page contains no cells (which is only possible for a root page
of a table that contains no rows) then the offset to the
cell content area will equal the page size minus the bytes of reserved space.
If the database uses a 65536-byte page size and the reserved space is zero
(the usual value for reserved space) then the cell content offset of an
empty page wants to be 65536.  
However, that integer is too large to be stored in a
2-byte unsigned integer, so a value of 0 is used in its place.

A freeblock is a structure used to identify unallocated space within
a b-tree page.  Freeblocks are organized as a chain.  The first 2 bytes of
a freeblock are a big-endian integer which is the offset in the b-tree page
of the next freeblock in the chain, or zero if the freeblock is the last on
the chain.  The third and fourth bytes of each freeblock form
a big-endian integer which is the size of the freeblock in bytes, including
the 4-byte header.  Freeblocks are always connected in order 
of increasing offset.  The second field of the b-tree page header is the
offset of the first freeblock, or zero if there are no freeblocks on the
page.  In a well-formed b-tree page, there will always be at least one cell
before the first freeblock.

A freeblock requires at least 4 bytes of space.  If there is an isolated
group of 1, 2, or 3 unused bytes within the cell content area, those bytes
comprise a fragment.  The total number of bytes in all fragments is stored
in the fifth field of the b-tree page header.  In a well-formed b-tree page,
the total number of bytes in fragments may not exceed 60.

The total amount of free space on a b-tree page consists of the size
of the unallocated region plus the total size of all freeblocks plus the
number of fragmented free bytes.  SQLite may from time to time reorganize
a b-tree page so that there are no freeblocks or fragment bytes, all
unused bytes are contained in the unallocated space region, and all
cells are packed tightly at the end of the page.  This is called 
"defragmenting" the b-tree page.




A variable-length integer or "varint" is a static Huffman encoding
of 64-bit twos-complement integers that uses less space for small positive 
values. 
A varint is between 1 and 9 bytes in length.  The varint consists of either
zero or more bytes which have the high-order bit set followed by a single byte
with the high-order bit clear, or nine bytes, whichever is shorter.
The lower seven bits of each of the first eight bytes and all 8 bits of
the ninth byte are used to reconstruct the 64-bit twos-complement integer.
Varints are big-endian: bits taken from the earlier byte of the varint
are more significant than bits taken from the later bytes. 

The format of a cell depends on which kind of b-tree page the cell
appears on.  The following table shows the elements of a cell, in
order of appearance, for the various b-tree page types.


Table B-Tree Leaf Cell (header 0x0d):

A varint which is the total number of bytes of payload, including any
overflow
A varint which is the integer key, a.k.a. "rowid"
The initial portion of the payload that does not spill to overflow
pages.
A 4-byte big-endian integer page number for the first page of the
overflow page list - omitted if all payload fits on the b-tree page.


Table B-Tree Interior Cell (header 0x05):

A 4-byte big-endian page number which is the left child pointer.
A varint which is the integer key


Index B-Tree Leaf Cell (header 0x0a):

A varint which is the total number of bytes of key payload, including any
overflow
The initial portion of the payload that does not spill to overflow
pages.
A 4-byte big-endian integer page number for the first page of the
overflow page list - omitted if all payload fits on the b-tree page.


Index B-Tree Interior Cell (header 0x02):

A 4-byte big-endian page number which is the left child pointer.
A varint which is the total number of bytes of key payload, including any
overflow
The initial portion of the payload that does not spill to overflow
pages.
A 4-byte big-endian integer page number for the first page of the
overflow page list - omitted if all payload fits on the b-tree page.



The information above can be recast into a table format as follows:




B-tree Cell Format

Datatype
    Appears in...
    Description
Table Leaf (0x0d)
    Table Interior (0x05)
    Index Leaf (0x0a)
    Index Interior (0x02)
4-byte integer
    &nbsp;
    &#x2714;
    &nbsp;
    &#x2714;
    Page number of left child
varint
    &#x2714;
    &nbsp;
    &#x2714;
    &#x2714;
    Number of bytes of payload
varint
    &#x2714;
    &#x2714;
    &nbsp;
    &nbsp;
    Rowid
byte array
    &#x2714;
    &nbsp;
    &#x2714;
    &#x2714;
    Payload
4-byte integer
    &#x2714;
    &nbsp;
    &#x2714;
    &#x2714;
    Page number of first overflow page



The amount of payload that spills onto overflow pages also depends on
the page type.  For the following computations, let U be the usable size
of a database page, the total page size less the reserved space at the
end of each page.  And let P be the payload size.  In the following,
symbol X represents the maximum amount of payload that can be stored directly
on the b-tree page without spilling onto an overflow page and symbol M
represents the minimum amount of payload that must be stored on the btree
page before spilling is allowed.


Table B-Tree Leaf Cell:

Let X be U-35.  If the payload size P is less than or equal to X then
the entire payload is stored on the b-tree leaf page.
Let M be ((U-12)*32/255)-23 and let K be M+((P-M)%(U-4)).
If P is greater than X
then the number of bytes stored on the table b-tree leaf page is K
if K is less or equal to X or M otherwise.
The number of bytes stored on the leaf page is never less than M.


Table B-Tree Interior Cell:

Interior pages of table b-trees have no payload and so there is never
any payload to spill.


Index B-Tree Leaf Or Interior Cell:

Let X be ((U-12)*64/255)-23.  If the payload size P is less than
or equal to X then the entire payload is stored on the b-tree page.
Let M be ((U-12)*32/255)-23 and let K be M+((P-M)%(U-4)).
If P is greater than X then the number
of bytes stored on the index b-tree page is K if K is less than or
equal to X or M otherwise.
The number of bytes stored on the index page is never less than M.



Here is an alternative description of the same computation:


X is U-35 for table btree leaf pages or
    ((U-12)*64/255)-23 for index pages.
M is always ((U-12)*32/255)-23.
Let K be M+((P-M)%(U-4)).
If P&lt;=X then all P bytes of payload are stored directly on the 
    btree page without overflow.
If P&gt;X and K&lt;=X then the first K bytes of P are stored on the 
    btree page and the remaining P-K bytes are stored on overflow pages.
If P&gt;X and K&gt;X then the first M bytes of P are stored on the
    btree page and the remaining P-M bytes are stored on overflow pages.


The overflow thresholds are designed to give a minimum fanout of
4 for index b-trees and to make sure enough of the payload
is on the b-tree page that the record header can usually be accessed
without consulting an overflow page.  In hindsight, the designer of
the SQLite b-tree logic realized that these thresholds could have been
made much simpler.  However, the computations cannot be changed
without resulting in an incompatible file format.  And the current computations
work well, even if they are a little complex.


