952 lines
33 KiB
HTML
952 lines
33 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
<head>
|
||
<meta name="generator" content=
|
||
"HTML Tidy for HTML5 for Linux version 5.2.0">
|
||
<title>Database Access and Persistence</title>
|
||
<meta charset="utf-8">
|
||
<meta name="description" content="A collection of examples of using Common Lisp">
|
||
<meta name="viewport" content=
|
||
"width=device-width, initial-scale=1">
|
||
<link rel="stylesheet" href=
|
||
"assets/style.css">
|
||
<script type="text/javascript" src=
|
||
"assets/highlight-lisp.js">
|
||
</script>
|
||
<script type="text/javascript" src=
|
||
"assets/jquery-3.2.1.min.js">
|
||
</script>
|
||
<script type="text/javascript" src=
|
||
"assets/jquery.toc/jquery.toc.min.js">
|
||
</script>
|
||
<script type="text/javascript" src=
|
||
"assets/toggle-toc.js">
|
||
</script>
|
||
|
||
<link rel="stylesheet" href=
|
||
"assets/github.css">
|
||
|
||
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
|
||
</head>
|
||
<body>
|
||
<h1 id="title-xs"><a href="index.html">The Common Lisp Cookbook</a> – Database Access and Persistence</h1>
|
||
<div id="logo-container">
|
||
<a href="index.html">
|
||
<img id="logo" src="assets/cl-logo-blue.png"/>
|
||
</a>
|
||
|
||
<div id="searchform-container">
|
||
<form onsubmit="duckSearch()" action="javascript:void(0)">
|
||
<input id="searchField" type="text" value="" placeholder="Search...">
|
||
</form>
|
||
</div>
|
||
|
||
<div id="toc-container" class="toc-close">
|
||
<div id="toc-title">Table of Contents</div>
|
||
<ul id="toc" class="list-unstyled"></ul>
|
||
</div>
|
||
</div>
|
||
|
||
<div id="content-container">
|
||
<h1 id="title-non-xs"><a href="index.html">The Common Lisp Cookbook</a> – Database Access and Persistence</h1>
|
||
|
||
<!-- Announcement we can keep for 1 month or more. I remove it and re-add it from time to time. -->
|
||
<p class="announce">
|
||
📹 <a href="https://www.udemy.com/course/common-lisp-programming/?couponCode=6926D599AA-LISP4ALL">NEW! Learn Lisp in videos and support our contributors with this 40% discount.</a>
|
||
</p>
|
||
<p class="announce-neutral">
|
||
📕 <a href="index.html#download-in-epub">Get the EPUB and PDF</a>
|
||
</p>
|
||
|
||
|
||
<div id="content"
|
||
<p>The
|
||
<a href="https://github.com/CodyReichert/awesome-cl#database">Database section on the Awesome-cl list</a>
|
||
is a resource listing popular libraries to work with different kind of
|
||
databases. We can group them roughly in four categories:</p>
|
||
|
||
<ul>
|
||
<li>wrappers to one database engine (cl-sqlite, postmodern, cl-redis,…),</li>
|
||
<li>interfaces to several DB engines (clsql, sxql,…),</li>
|
||
<li>persistent object databases (bknr.datastore (see chap. 21 of “Common Lisp Recipes”), ubiquitous,…),</li>
|
||
<li><a href="https://en.wikipedia.org/wiki/Object-relational_mapping">Object Relational Mappers</a> (Mito),</li>
|
||
</ul>
|
||
|
||
<p>and other DB-related tools (pgloader).</p>
|
||
|
||
<p>We’ll begin with an overview of Mito. If you must work with an
|
||
existing DB, you might want to have a look at cl-dbi and clsql. If you
|
||
don’t need a SQL database and want automatic persistence of Lisp
|
||
objects, you also have a choice of libraries.</p>
|
||
|
||
<h2 id="the-mito-orm-and-sxql">The Mito ORM and SxQL</h2>
|
||
|
||
<p>Mito is in Quicklisp:</p>
|
||
|
||
<pre><code class="language-lisp">(ql:quickload "mito")
|
||
</code></pre>
|
||
|
||
<h3 id="overview">Overview</h3>
|
||
|
||
<p><a href="https://github.com/fukamachi/mito">Mito</a> is “an ORM for Common Lisp
|
||
with migrations, relationships and PostgreSQL support”.</p>
|
||
|
||
<ul>
|
||
<li>it <strong>supports MySQL, PostgreSQL and SQLite3</strong>,</li>
|
||
<li>when defining a model, it adds an <code>id</code> (serial primary key),
|
||
<code>created_at</code> and <code>updated_at</code> fields by default like Ruby’s
|
||
ActiveRecord or Django,</li>
|
||
<li>handles DB <strong>migrations</strong> for the supported backends,</li>
|
||
<li>permits DB <strong>schema versioning</strong>,</li>
|
||
<li>is tested under SBCL and CCL.</li>
|
||
</ul>
|
||
|
||
<p>As an ORM, it allows to write class definitions, to specify relationships, and
|
||
provides functions to query the database. For custom queries, it relies on
|
||
<a href="https://github.com/fukamachi/sxql">SxQL</a>, an SQL generator that provides the
|
||
same interface for several backends.</p>
|
||
|
||
<p>Working with Mito generally involves these steps:</p>
|
||
|
||
<ul>
|
||
<li>connecting to the DB</li>
|
||
<li>writing <a href="clos.html">CLOS</a> classes to define models</li>
|
||
<li>running migrations to create or alter tables</li>
|
||
<li>creating objects, saving same in the DB,</li>
|
||
</ul>
|
||
|
||
<p>and iterating.</p>
|
||
|
||
<h3 id="connecting-to-a-db">Connecting to a DB</h3>
|
||
|
||
<p>Mito provides the function <code>connect-toplevel</code> to establish a
|
||
connection to RDBMs:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:connect-toplevel :mysql :database-name "myapp" :username "fukamachi" :password "c0mon-1isp")
|
||
</code></pre>
|
||
|
||
<p>The driver type can be of <code>:mysql</code>, <code>:sqlite3</code> and <code>:postgres</code>.</p>
|
||
|
||
<p>With sqlite you don’t need the username and password:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:connect-toplevel :sqlite3 :database-name "myapp")
|
||
</code></pre>
|
||
|
||
<p>As usual, you need to create the MySQL or PostgreSQL database beforehand.
|
||
Refer to their documentation.</p>
|
||
|
||
<p>Connecting sets <code>mito:*connection*</code> to the new connection and returns it.</p>
|
||
|
||
<p>Disconnect with <code>disconnect-toplevel</code>.</p>
|
||
|
||
<p>You might make good use of a wrapper function:</p>
|
||
|
||
<pre><code class="language-lisp">(defun connect ()
|
||
"Connect to the DB."
|
||
(mito:connect-toplevel :sqlite3 :database-name "myapp"))
|
||
</code></pre>
|
||
|
||
<h3 id="models">Models</h3>
|
||
|
||
<h4 id="defining-models">Defining models</h4>
|
||
|
||
<p>In Mito, you can define a class which corresponds to a database table with the <code>deftable</code> macro:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:deftable user ()
|
||
((name :col-type (:varchar 64))
|
||
(email :col-type (or (:varchar 128) :null)))
|
||
</code></pre>
|
||
<p>Alternatively, you can specify <code>(:metaclass mito:dao-table-class)</code> in a regular class definition.</p>
|
||
|
||
<p>The <code>deftable</code> macro automatically adds some slots: a primary key named <code>id</code> if there’s no primary key, and <code>created_at</code> and <code>updated_at</code> for recording timestamps. Specifying <code>(:auto-pk nil)</code> and <code>(:record-timestamps nil)</code> in the <code>deftable</code> form will disable these behaviours. A <code>deftable</code> class will also come with initializers, named after the slot, and accessors, of form <code><class-name>-<slot-name></code>, for each named slot. For example, for the <code>name</code> slot in the above table definition, the initarg <code>:name</code> will be added to the constuctor, and the accessor <code>user-name</code> will be created.</p>
|
||
|
||
<p>You can inspect the new class:</p>
|
||
|
||
<pre><code class="language-lisp">(mito.class:table-column-slots (find-class 'user))
|
||
;=> (#<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS MITO.DAO.MIXIN::ID>
|
||
; #<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS COMMON-LISP-USER::NAME>
|
||
; #<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS COMMON-LISP-USER::EMAIL>
|
||
; #<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS MITO.DAO.MIXIN::CREATED-AT>
|
||
; #<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS MITO.DAO.MIXIN::UPDATED-AT>)
|
||
</code></pre>
|
||
|
||
<p>The class inherits <code>mito:dao-class</code> implicitly.</p>
|
||
|
||
<pre><code class="language-lisp">(find-class 'user)
|
||
;=> #<MITO.DAO.TABLE:DAO-TABLE-CLASS COMMON-LISP-USER::USER>
|
||
|
||
(c2mop:class-direct-superclasses *)
|
||
;=> (#<STANDARD-CLASS MITO.DAO.TABLE:DAO-CLASS>)
|
||
</code></pre>
|
||
|
||
<p>This may be useful when you define methods which can be applied for
|
||
all table classes.</p>
|
||
|
||
<p>For more information on using the Common Lisp Object System, see the
|
||
<a href="clos.html">clos</a> page.</p>
|
||
|
||
<h4 id="creating-the-tables">Creating the tables</h4>
|
||
|
||
<p>After defining the models, you must create the tables:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:ensure-table-exists 'user)
|
||
</code></pre>
|
||
|
||
<p>So a helper function:</p>
|
||
|
||
<pre><code class="language-lisp">(defun ensure-tables ()
|
||
(mapcar #'mito:ensure-table-exists '(user foo bar)))
|
||
</code></pre>
|
||
|
||
<p>See
|
||
<a href="https://github.com/fukamachi/mito#generating-table-definitions">Mito’s documentation</a>
|
||
for a couple more ways.</p>
|
||
|
||
<p>When you alter the model you’ll need to run a DB migration, see the next section.</p>
|
||
|
||
<h4 id="fields">Fields</h4>
|
||
|
||
<h5 id="fields-types">Fields types</h5>
|
||
|
||
<p>Field types are:</p>
|
||
|
||
<p><code>(:varchar <integer>)</code> ,</p>
|
||
|
||
<p><code>:serial</code>, <code>:bigserial</code>, <code>:integer</code>, <code>:bigint</code>, <code>:unsigned</code>,</p>
|
||
|
||
<p><code>:timestamp</code>, <code>:timestamptz</code>,</p>
|
||
|
||
<p><code>:bytea</code>,</p>
|
||
|
||
<h5 id="optional-fields">Optional fields</h5>
|
||
|
||
<p>Use <code>(or <real type> :null)</code>:</p>
|
||
|
||
<pre><code class="language-lisp"> (email :col-type (or (:varchar 128) :null))
|
||
</code></pre>
|
||
|
||
<h5 id="field-constraints">Field constraints</h5>
|
||
|
||
<p><code>:unique-keys</code> can be used like so:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:deftable user ()
|
||
((name :col-type (:varchar 64))
|
||
(email :col-type (:varchar 128))
|
||
(:unique-keys email))
|
||
</code></pre>
|
||
|
||
<p>We already saw <code>:primary-key</code>.</p>
|
||
|
||
<p>You can change the table name with <code>:table-name</code>.</p>
|
||
|
||
<h4 id="relationships">Relationships</h4>
|
||
|
||
<p>You can define a relationship by specifying a foreign class with <code>:col-type</code>:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:deftable tweet ()
|
||
((status :col-type :text)
|
||
;; This slot refers to USER class
|
||
(user :col-type user))
|
||
|
||
(table-definition (find-class 'tweet))
|
||
;=> (#<SXQL-STATEMENT: CREATE TABLE tweet (
|
||
; id BIGSERIAL NOT NULL PRIMARY KEY,
|
||
; status TEXT NOT NULL,
|
||
; user_id BIGINT NOT NULL,
|
||
; created_at TIMESTAMP,
|
||
; updated_at TIMESTAMP
|
||
; )>)
|
||
</code></pre>
|
||
|
||
<p>Now you can create or retrieve a <code>TWEET</code> by a <code>USER</code> object, not a <code>USER-ID</code>.</p>
|
||
|
||
<pre><code class="language-lisp">(defvar *user* (mito:create-dao 'user :name "Eitaro Fukamachi"))
|
||
(mito:create-dao 'tweet :user *user*)
|
||
|
||
(mito:find-dao 'tweet :user *user*)
|
||
</code></pre>
|
||
|
||
<p>Mito doesn’t add foreign key constraints for referring tables.</p>
|
||
|
||
<h5 id="one-to-one">One-to-one</h5>
|
||
|
||
<p>A one-to-one relationship is simply represented with a simple foreign
|
||
key on a slot (as <code>:col-type user</code> in the <code>tweet</code> class). Besides, we
|
||
can add a unicity constraint, as with <code>(:unique-keys email)</code>.</p>
|
||
|
||
<h5 id="one-to-many-many-to-one">One-to-many, many-to-one</h5>
|
||
|
||
<p>The tweet example above shows a one-to-many relationship between a user and
|
||
his tweets: a user can write many tweets, and a tweet belongs to only
|
||
one user.</p>
|
||
|
||
<p>The relationship is defined with a foreign key on the “many” side
|
||
linking back to the “one” side. Here the <code>tweet</code> class defines a
|
||
<code>user</code> foreign key, so a tweet can only have one user. You didn’t need
|
||
to edit the <code>user</code> class.</p>
|
||
|
||
<p>A many-to-one relationship is actually the contrary of a one-to-many.
|
||
You have to put the foreign key on the appropriate side.</p>
|
||
|
||
<h5 id="many-to-many">Many-to-many</h5>
|
||
|
||
<p>A many-to-many relationship needs an intermediate table, which will be
|
||
the “many” side for the two tables it is the intermediary of.</p>
|
||
|
||
<p>And, thanks to the join table, we can store more information about the relationship.</p>
|
||
|
||
<p>Let’s define a <code>book</code> class:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:deftable book ()
|
||
((title :col-type (:varchar 128))
|
||
(ean :col-type (or (:varchar 128) :null))))
|
||
</code></pre>
|
||
|
||
<p>A user can have many books, and a book (as the title, not the physical
|
||
copy) is likely to be in many people’s library. Here’s the
|
||
intermediate class:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:deftable user-books ()
|
||
((user :col-type user)
|
||
(book :col-type book)))
|
||
</code></pre>
|
||
|
||
<p>Each time we want to add a book to a user’s collection (say in
|
||
a <code>add-book</code> function), we create a new <code>user-books</code> object.</p>
|
||
|
||
<p>But someone may very well own many copies of one book. This is an
|
||
information we can store in the join table:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:deftable user-books ()
|
||
((user :col-type user)
|
||
(book :col-type book)
|
||
;; Set the quantity, 1 by default:
|
||
(quantity :col-type :integer)))
|
||
</code></pre>
|
||
|
||
<h4 id="inheritance-and-mixin">Inheritance and mixin</h4>
|
||
|
||
<p>A subclass of DAO-CLASS is allowed to be inherited. This may be useful
|
||
when you need classes which have similar columns:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:deftable user ()
|
||
((name :col-type (:varchar 64))
|
||
(email :col-type (:varchar 128)))
|
||
(:unique-keys email))
|
||
|
||
(mito:deftable temporary-user (user)
|
||
((registered-at :col-type :timestamp)))
|
||
|
||
(mito:table-definition 'temporary-user)
|
||
;=> (#<SXQL-STATEMENT: CREATE TABLE temporary_user (
|
||
; id BIGSERIAL NOT NULL PRIMARY KEY,
|
||
; name VARCHAR(64) NOT NULL,
|
||
; email VARCHAR(128) NOT NULL,
|
||
; registered_at TIMESTAMP NOT NULL,
|
||
; created_at TIMESTAMP,
|
||
; updated_at TIMESTAMP,
|
||
; UNIQUE (email)
|
||
; )>)
|
||
</code></pre>
|
||
|
||
<p>If you need a ‘template’ for tables which aren’t related to any
|
||
database tables, you can use <code>DAO-TABLE-MIXIN</code> in a <code>defclass</code> form. The <code>has-email</code>
|
||
class below will not create a table.</p>
|
||
|
||
<pre><code class="language-lisp">(defclass has-email ()
|
||
((email :col-type (:varchar 128)
|
||
:initarg :email
|
||
:accessor object-email))
|
||
(:metaclass mito:dao-table-mixin)
|
||
(:unique-keys email))
|
||
;=> #<MITO.DAO.MIXIN:DAO-TABLE-MIXIN COMMON-LISP-USER::HAS-EMAIL>
|
||
|
||
(mito:deftable user (has-email)
|
||
((name :col-type (:varchar 64))))
|
||
;=> #<MITO.DAO.TABLE:DAO-TABLE-CLASS COMMON-LISP-USER::USER>
|
||
|
||
(mito:table-definition 'user)
|
||
;=> (#<SXQL-STATEMENT: CREATE TABLE user (
|
||
; id BIGSERIAL NOT NULL PRIMARY KEY,
|
||
; name VARCHAR(64) NOT NULL,
|
||
; email VARCHAR(128) NOT NULL,
|
||
; created_at TIMESTAMP,
|
||
; updated_at TIMESTAMP,
|
||
; UNIQUE (email)
|
||
; )>)
|
||
</code></pre>
|
||
|
||
<p>See more examples of use in <a href="https://github.com/fukamachi/mito-auth/">mito-auth</a>.</p>
|
||
|
||
<h4 id="troubleshooting">Troubleshooting</h4>
|
||
|
||
<h5 id="cannot-change-class-objects-into-class-metaobjects">“Cannot CHANGE-CLASS objects into CLASS metaobjects.”</h5>
|
||
|
||
<p>If you get the following error message:</p>
|
||
|
||
<pre><code>Cannot CHANGE-CLASS objects into CLASS metaobjects.
|
||
[Condition of type SB-PCL::METAOBJECT-INITIALIZATION-VIOLATION]
|
||
See also:
|
||
The Art of the Metaobject Protocol, CLASS [:initialization]
|
||
</code></pre>
|
||
|
||
<p>it is certainly because you first wrote a class definition and <em>then</em>
|
||
added the Mito metaclass and tried to evaluate the class definition
|
||
again.</p>
|
||
|
||
<p>If this happens, you must remove the class definition from the current package:</p>
|
||
|
||
<pre><code class="language-lisp">(setf (find-class 'foo) nil)
|
||
</code></pre>
|
||
|
||
<p>or, with the Slime inspector, click on the class and find the “remove” button.</p>
|
||
|
||
<p>More info <a href="https://stackoverflow.com/questions/38811931/how-to-change-classs-metaclass">here</a>.</p>
|
||
|
||
<h3 id="migrations">Migrations</h3>
|
||
|
||
<p>We can run database migrations manually, as shown below, or we can
|
||
automatically run migrations after a change to the model
|
||
definitions. To enable automatic migrations, set <code>mito:*auto-migration-mode*</code> to <code>t</code>.</p>
|
||
|
||
<p>The first step is to create the tables, if needed:</p>
|
||
|
||
<pre><code class="language-lisp">(ensure-table-exists 'user)
|
||
</code></pre>
|
||
|
||
<p>then alter the tables:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:migrate-table 'user)
|
||
</code></pre>
|
||
|
||
<p>You can check the SQL generated code with <code>migration-expressions
|
||
'class</code>. For example, we create the <code>user</code> table:</p>
|
||
|
||
<pre><code class="language-lisp">(ensure-table-exists 'user)
|
||
;-> ;; CREATE TABLE IF NOT EXISTS "user" (
|
||
; "id" BIGSERIAL NOT NULL PRIMARY KEY,
|
||
; "name" VARCHAR(64) NOT NULL,
|
||
; "email" VARCHAR(128),
|
||
; "created_at" TIMESTAMP,
|
||
; "updated_at" TIMESTAMP
|
||
; ) () [0 rows] | MITO.DAO:ENSURE-TABLE-EXISTS
|
||
</code></pre>
|
||
|
||
<p>There are no changes from the previous user definition:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:migration-expressions 'user)
|
||
;=> NIL
|
||
</code></pre>
|
||
|
||
<p>Now let’s add a unique <code>email</code> field:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:deftable user ()
|
||
((name :col-type (:varchar 64))
|
||
(email :col-type (:varchar 128)))
|
||
(:unique-keys email))
|
||
</code></pre>
|
||
|
||
<p>The migration will run the following code:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:migration-expressions 'user)
|
||
;=> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
|
||
; #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)
|
||
</code></pre>
|
||
|
||
<p>so let’s apply it:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:migrate-table 'user)
|
||
;-> ;; ALTER TABLE "user" ALTER COLUMN "email" TYPE character varying(128), ALTER COLUMN "email" SET NOT NULL () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
|
||
; ;; CREATE UNIQUE INDEX "unique_user_email" ON "user" ("email") () [0 rows] | MITO.MIGRATION.TABLE:MIGRATE-TABLE
|
||
;-> (#<SXQL-STATEMENT: ALTER TABLE user ALTER COLUMN email TYPE character varying(128), ALTER COLUMN email SET NOT NULL>
|
||
; #<SXQL-STATEMENT: CREATE UNIQUE INDEX unique_user_email ON user (email)>)
|
||
</code></pre>
|
||
|
||
<h3 id="queries">Queries</h3>
|
||
|
||
<h4 id="creating-objects">Creating objects</h4>
|
||
|
||
<p>We can create user objects with the regular <code>make-instance</code>:</p>
|
||
|
||
<pre><code class="language-lisp">(defvar me
|
||
(make-instance 'user :name "Eitaro Fukamachi" :email "e.arrows@gmail.com"))
|
||
;=> USER
|
||
</code></pre>
|
||
|
||
<p>To save it in DB, use <code>insert-dao</code>:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:insert-dao me)
|
||
;-> ;; INSERT INTO `user` (`name`, `email`, `created_at`, `updated_at`) VALUES (?, ?, ?, ?) ("Eitaro Fukamachi", "e.arrows@gmail.com", "2016-02-04T19:55:16.365543Z", "2016-02-04T19:55:16.365543Z") [0 rows] | MITO.DAO:INSERT-DAO
|
||
;=> #<USER {10053C4453}>
|
||
</code></pre>
|
||
|
||
<p>Do the two steps above at once:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:create-dao 'user :name "Eitaro Fukamachi" :email "e.arrows@gmail.com")
|
||
</code></pre>
|
||
|
||
<p>You should not export the <code>user</code> class and create objects outside of
|
||
its package (it is good practice anyway to keep all database-related
|
||
operations in say a <code>models</code> package and file). You should instead use
|
||
a helper function:</p>
|
||
|
||
<pre><code class="language-lisp">(defun make-user (&key name)
|
||
(make-instance 'user :name name))
|
||
</code></pre>
|
||
|
||
<h4 id="updating-fields">Updating fields</h4>
|
||
|
||
<pre><code class="language-lisp">(setf (slot-value me 'name) "nitro_idiot")
|
||
;=> "nitro_idiot"
|
||
</code></pre>
|
||
|
||
<p>and save it:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:save-dao me)
|
||
</code></pre>
|
||
|
||
<h4 id="deleting">Deleting</h4>
|
||
|
||
<pre><code class="language-lisp">(mito:delete-dao me)
|
||
;-> ;; DELETE FROM `user` WHERE (`id` = ?) (1) [0 rows] | MITO.DAO:DELETE-DAO
|
||
|
||
;; or:
|
||
(mito:delete-by-values 'user :id 1)
|
||
;-> ;; DELETE FROM `user` WHERE (`id` = ?) (1) [0 rows] | MITO.DAO:DELETE-DAO
|
||
</code></pre>
|
||
|
||
<h4 id="get-the-primary-key-value">Get the primary key value</h4>
|
||
|
||
<pre><code class="language-lisp">(mito:object-id me)
|
||
;=> 1
|
||
</code></pre>
|
||
|
||
<h4 id="count">Count</h4>
|
||
|
||
<pre><code class="language-lisp">(mito:count-dao 'user)
|
||
;=> 1
|
||
</code></pre>
|
||
|
||
<h4 id="find-one">Find one</h4>
|
||
|
||
<pre><code class="language-lisp">(mito:find-dao 'user :id 1)
|
||
;-> ;; SELECT * FROM `user` WHERE (`id` = ?) LIMIT 1 (1) [1 row] | MITO.DB:RETRIEVE-BY-SQL
|
||
;=> #<USER {10077C6073}>
|
||
</code></pre>
|
||
|
||
<p>So here’s a possibility of generic helpers to find an object by a given key:</p>
|
||
|
||
<pre><code class="language-lisp">(defgeneric find-user (key-name key-value)
|
||
(:documentation "Retrieves an user from the data base by one of the unique
|
||
keys."))
|
||
|
||
(defmethod find-user ((key-name (eql :id)) (key-value integer))
|
||
(mito:find-dao 'user key-value))
|
||
|
||
(defmethod find-user ((key-name (eql :name)) (key-value string))
|
||
(first (mito:select-dao 'user
|
||
(sxql:where (:= :name key-value)))))
|
||
</code></pre>
|
||
|
||
<h4 id="find-all">Find all</h4>
|
||
|
||
<p>Use the macro <code>select-dao</code>.</p>
|
||
|
||
<p>Get a list of all users:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:select-dao 'user)
|
||
;(#<USER {10077C6073}>)
|
||
;#<SXQL-STATEMENT: SELECT * FROM user>
|
||
</code></pre>
|
||
|
||
<h4 id="find-by-relationship">Find by relationship</h4>
|
||
|
||
<p>As seen above:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:find-dao 'tweet :user *user*)
|
||
</code></pre>
|
||
|
||
<h4 id="custom-queries">Custom queries</h4>
|
||
|
||
<p>It is with <code>select-dao</code> that you can write more precise queries by
|
||
giving it <a href="https://github.com/fukamachi/sxql">SxQL</a> statements.</p>
|
||
|
||
<p>Example:</p>
|
||
|
||
<pre><code class="language-lisp">(select-dao 'tweet
|
||
(where (:like :status "%Japan%")))
|
||
</code></pre>
|
||
|
||
<p>another:</p>
|
||
|
||
<pre><code class="language-lisp">(select (:id :name :sex)
|
||
(from (:as :person :p))
|
||
(where (:and (:>= :age 18)
|
||
(:< :age 65)))
|
||
(order-by (:desc :age)))
|
||
</code></pre>
|
||
|
||
<p>You can compose your queries with regular Lisp code:</p>
|
||
|
||
<pre><code class="language-lisp">(defun find-tweets (&key user)
|
||
(select-dao 'tweet
|
||
(when user
|
||
(where (:= :user user)))
|
||
(order-by :object-created)))
|
||
</code></pre>
|
||
|
||
<p><code>select-dao</code> is a macro that expands to the right thing©.</p>
|
||
|
||
<div class="info-box info">
|
||
<strong>Note:</strong> if you didn't <code>use</code> SXQL, then write <code>(sxql:where …)</code> and <code>(sxql:order-by …)</code>.
|
||
</div>
|
||
<p><br /></p>
|
||
|
||
<p>You can compose your queries further with the backquote syntax.</p>
|
||
|
||
<p>Imagine you receive a <code>query</code> string, maybe composed of
|
||
space-separated words, and you want to search for books that have
|
||
either one of these words in their title or in their author’s
|
||
name. Searching for “bob adventure” would return a book that has
|
||
“adventure” in its title and “bob” in its author name, or both in the
|
||
title.</p>
|
||
|
||
<p>For the example sake, an author is a string, not a link to another table:</p>
|
||
|
||
<pre><code class="language-lisp">(mito:deftable book ()
|
||
((title :col-type (:varchar 128))
|
||
(author :col-type (:varchar 128))
|
||
(ean :col-type (or (:varchar 128) :null))))
|
||
</code></pre>
|
||
|
||
<p>You want to add a clause that searches on both fields for each word.</p>
|
||
|
||
<pre><code class="language-lisp">(defun find-books (&key query (order :desc))
|
||
"Return a list of books. If a query string is given, search on both the title and the author fields."
|
||
(mito:select-dao 'book
|
||
(when (str:non-blank-string-p query)
|
||
(sxql:where
|
||
`(:and
|
||
,@(loop for word in (str:words query)
|
||
:collect `(:or (:like :title ,(str:concat "%" word "%"))
|
||
(:like :authors ,(str:concat "%" word "%")))))))
|
||
(sxql:order-by `(,order :created-at))))
|
||
</code></pre>
|
||
|
||
<p>By the way, we are still using a <code>LIKE</code> statement, but with a non-small dataset you’ll want to use your database’s full text search engine.</p>
|
||
|
||
<h4 id="clauses">Clauses</h4>
|
||
|
||
<p>See the <a href="https://github.com/fukamachi/sxql#sql-clauses">SxQL documentation</a>.</p>
|
||
|
||
<p>Examples:</p>
|
||
|
||
<pre><code class="language-lisp">(select-dao 'foo
|
||
(where (:and (:> :age 20) (:<= :age 65))))
|
||
</code></pre>
|
||
|
||
<pre><code class="language-lisp">(order-by :age (:desc :id))
|
||
</code></pre>
|
||
|
||
<pre><code class="language-lisp">(group-by :sex)
|
||
</code></pre>
|
||
|
||
<pre><code class="language-lisp">(having (:>= (:sum :hoge) 88))
|
||
</code></pre>
|
||
|
||
<pre><code class="language-lisp">(limit 0 10)
|
||
</code></pre>
|
||
|
||
<p>and <code>join</code>s, etc.</p>
|
||
|
||
<h4 id="operators">Operators</h4>
|
||
|
||
<pre><code class="language-lisp">:not
|
||
:is-null, :not-null
|
||
:asc, :desc
|
||
:distinct
|
||
:=, :!=
|
||
:<, :>, :<= :>=
|
||
:a<, :a>
|
||
:as
|
||
:in, :not-in
|
||
:like
|
||
:and, :or
|
||
:+, :-, :* :/ :%
|
||
:raw
|
||
</code></pre>
|
||
|
||
<h3 id="triggers">Triggers</h3>
|
||
|
||
<p>Since <code>insert-dao</code>, <code>update-dao</code> and <code>delete-dao</code> are defined as generic
|
||
functions, you can define <code>:before</code>, <code>:after</code> or <code>:around</code> methods to those, like regular <a href="clos.html#qualifiers-and-method-combination">method combination</a>.</p>
|
||
|
||
<pre><code class="language-lisp">(defmethod mito:insert-dao :before ((object user))
|
||
(format t "~&Adding ~S...~%" (user-name object)))
|
||
|
||
(mito:create-dao 'user :name "Eitaro Fukamachi" :email "e.arrows@gmail.com")
|
||
;-> Adding "Eitaro Fukamachi"...
|
||
; ;; INSERT INTO "user" ("name", "email", "created_at", "updated_at") VALUES (?, ?, ?, ?) ("Eitaro Fukamachi", "e.arrows@gmail.com", "2016-02-16 21:13:47", "2016-02-16 21:13:47") [0 rows] | MITO.DAO:INSERT-DAO
|
||
;=> #<USER {100835FB33}>
|
||
</code></pre>
|
||
|
||
<h3 id="inflationdeflation">Inflation/Deflation</h3>
|
||
|
||
<p>Inflation/Deflation is a function to convert values between Mito and RDBMS.</p>
|
||
|
||
<pre><code class="language-lisp">(mito:deftable user-report ()
|
||
((title :col-type (:varchar 100))
|
||
(body :col-type :text
|
||
:initform "")
|
||
(reported-at :col-type :timestamp
|
||
:initform (local-time:now)
|
||
:inflate #'local-time:universal-to-timestamp
|
||
:deflate #'local-time:timestamp-to-universal)))
|
||
</code></pre>
|
||
|
||
<h3 id="eager-loading">Eager loading</h3>
|
||
|
||
<p>One of the pains in the neck to use ORMs is the “N+1 query” problem.</p>
|
||
|
||
<pre><code class="language-lisp">;; BAD EXAMPLE
|
||
|
||
(use-package '(:mito :sxql))
|
||
|
||
(defvar *tweets-contain-japan*
|
||
(select-dao 'tweet
|
||
(where (:like :status "%Japan%"))))
|
||
|
||
;; Getting names of tweeted users.
|
||
(mapcar (lambda (tweet)
|
||
(user-name (tweet-user tweet)))
|
||
*tweets-contain-japan*)
|
||
</code></pre>
|
||
|
||
<p>This example sends a query to retrieve a user like “SELECT * FROM user
|
||
WHERE id = ?” at each iteration.</p>
|
||
|
||
<p>To prevent this performance issue, add <code>includes</code> to the above query
|
||
which only sends a single WHERE IN query instead of N queries:</p>
|
||
|
||
<pre><code class="language-lisp">;; GOOD EXAMPLE with eager loading
|
||
|
||
(use-package '(:mito :sxql))
|
||
|
||
(defvar *tweets-contain-japan*
|
||
(select-dao 'tweet
|
||
(includes 'user)
|
||
(where (:like :status "%Japan%"))))
|
||
;-> ;; SELECT * FROM `tweet` WHERE (`status` LIKE ?) ("%Japan%") [3 row] | MITO.DB:RETRIEVE-BY-SQL
|
||
;-> ;; SELECT * FROM `user` WHERE (`id` IN (?, ?, ?)) (1, 3, 12) [3 row] | MITO.DB:RETRIEVE-BY-SQL
|
||
;=> (#<TWEET {1003513EC3}> #<TWEET {1007BABEF3}> #<TWEET {1007BB9D63}>)
|
||
|
||
;; No additional SQLs will be executed.
|
||
(tweet-user (first *))
|
||
;=> #<USER {100361E813}>
|
||
</code></pre>
|
||
|
||
<h3 id="schema-versioning">Schema versioning</h3>
|
||
|
||
<pre><code>$ ros install mito
|
||
$ mito
|
||
Usage: mito command [option...]
|
||
|
||
Commands:
|
||
generate-migrations
|
||
migrate
|
||
|
||
Options:
|
||
-t, --type DRIVER-TYPE DBI driver type (one of "mysql", "postgres" or "sqlite3")
|
||
-d, --database DATABASE-NAME Database name to use
|
||
-u, --username USERNAME Username for RDBMS
|
||
-p, --password PASSWORD Password for RDBMS
|
||
-s, --system SYSTEM ASDF system to load (several -s's allowed)
|
||
-D, --directory DIRECTORY Directory path to keep migration SQL files (default: "/Users/nitro_idiot/Programs/lib/mito/db/")
|
||
--dry-run List SQL expressions to migrate
|
||
</code></pre>
|
||
|
||
<h3 id="introspection">Introspection</h3>
|
||
|
||
<p>Mito provides some functions for introspection.</p>
|
||
|
||
<p>We can access the information of <strong>columns</strong> with the functions in
|
||
<code>(mito.class.column:...)</code>:</p>
|
||
|
||
<ul>
|
||
<li><code>table-column-[class, name, info, not-null-p,...]</code></li>
|
||
<li><code>primary-key-p</code></li>
|
||
</ul>
|
||
|
||
<p>and likewise for <strong>tables</strong> with <code>(mito.class.table:...)</code>.</p>
|
||
|
||
<p>Given we get a list of slots of our class:</p>
|
||
|
||
<pre><code class="language-lisp">(ql:quickload "closer-mop")
|
||
|
||
(closer-mop:class-direct-slots (find-class 'user))
|
||
;; (#<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS NAME>
|
||
;; #<MITO.DAO.COLUMN:DAO-TABLE-COLUMN-CLASS EMAIL>)
|
||
|
||
(defparameter user-slots *)
|
||
</code></pre>
|
||
|
||
<p>We can answer the following questions:</p>
|
||
|
||
<h4 id="what-is-the-type-of-this-column-">What is the type of this column ?</h4>
|
||
|
||
<pre><code class="language-lisp">(mito.class.column:table-column-type (first user-slots))
|
||
;; (:VARCHAR 64)
|
||
</code></pre>
|
||
|
||
<h4 id="is-this-column-nullable-">Is this column nullable ?</h4>
|
||
|
||
<pre><code class="language-lisp">(mito.class.column:table-column-not-null-p
|
||
(first user-slots))
|
||
;; T
|
||
(mito.class.column:table-column-not-null-p
|
||
(second user-slots))
|
||
;; NIL
|
||
</code></pre>
|
||
|
||
<h3 id="testing">Testing</h3>
|
||
|
||
<p>We don’t want to test DB operations against the production one. We
|
||
need to create a temporary DB before each test.</p>
|
||
|
||
<p>The macro below creates a temporary DB with a random name, creates the
|
||
tables, runs the code and connects back to the original DB connection.</p>
|
||
|
||
<pre><code class="language-lisp">(defpackage my-test.utils
|
||
(:use :cl)
|
||
(:import-from :my.models
|
||
:*db*
|
||
:*db-name*
|
||
:connect
|
||
:ensure-tables-exist
|
||
:migrate-all)
|
||
(:export :with-empty-db))
|
||
|
||
(in-package my-test.utils)
|
||
|
||
(defun random-string (length)
|
||
;; thanks 40ants/hacrm.
|
||
(let ((chars "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"))
|
||
(coerce (loop repeat length
|
||
collect (aref chars (random (length chars))))
|
||
'string)))
|
||
|
||
(defmacro with-empty-db (&body body)
|
||
"Run `body` with a new temporary DB."
|
||
`(let* ((*random-state* (make-random-state t))
|
||
(prefix (concatenate 'string
|
||
(random-string 8)
|
||
"/"))
|
||
;; Save our current DB connection.
|
||
(connection mito:*connection*))
|
||
(uiop:with-temporary-file (:pathname name :prefix prefix)
|
||
;; Bind our *db-name* to a new name, so as to create a new DB.
|
||
(let* ((*db-name* name))
|
||
;; Always re-connect to our real DB even in case of error in body.
|
||
(unwind-protect
|
||
(progn
|
||
;; our functions to connect to the DB, create the tables and run the migrations.
|
||
(connect)
|
||
(ensure-tables-exist)
|
||
(migrate-all)
|
||
,@body)
|
||
|
||
(setf mito:*connection* connection))))))
|
||
</code></pre>
|
||
|
||
<p>Use it like this:</p>
|
||
|
||
<pre><code class="language-lisp">(prove:subtest "Creation in a temporary DB."
|
||
(with-empty-db
|
||
(let ((user (make-user :name "Cookbook")))
|
||
(save-user user)
|
||
|
||
(prove:is (name user)
|
||
"Cookbook"
|
||
"Test username in a temp DB."))))
|
||
;; Creation in a temporary DB
|
||
;; CREATE TABLE "user" (
|
||
;; id BIGSERIAL NOT NULL PRIMARY KEY,
|
||
;; name VARCHAR(64) NOT NULL,
|
||
;; email VARCHAR(128) NOT NULL,
|
||
;; created_at TIMESTAMP,
|
||
;; updated_at TIMESTAMP,
|
||
;; UNIQUE (email)
|
||
;; ) () [0 rows] | MITO.DB:EXECUTE-SQL
|
||
;; ✓ Test username in a temp DB.
|
||
</code></pre>
|
||
|
||
<h2 id="see-also">See also</h2>
|
||
|
||
<ul>
|
||
<li>
|
||
<p><a href="https://sites.google.com/site/sabraonthehill/postmodern-examples/exploring-a-database">exploring an existing (PostgreSQL) database with postmodern</a></p>
|
||
</li>
|
||
<li><a href="https://github.com/fukamachi/mito-attachment">mito-attachment</a></li>
|
||
<li><a href="https://github.com/fukamachi/mito-auth">mito-auth</a></li>
|
||
<li><a href="https://github.com/fukamachi/can/">can</a> a role-based access right control library</li>
|
||
<li>an advanced <a href="drafts/defmodel.lisp.html">“defmodel” macro</a>.</li>
|
||
</ul>
|
||
|
||
<!-- # todo: Generating models for an existing DB -->
|
||
|
||
|
||
<p class="page-source">
|
||
Page source: <a href="https://github.com/LispCookbook/cl-cookbook/blob/master/databases.md">databases.md</a>
|
||
</p>
|
||
</div>
|
||
|
||
<script type="text/javascript">
|
||
|
||
// Don't write the TOC on the index.
|
||
if (window.location.pathname != "/cl-cookbook/") {
|
||
$("#toc").toc({
|
||
content: "#content", // will ignore the first h1 with the site+page title.
|
||
headings: "h1,h2,h3,h4"});
|
||
}
|
||
|
||
$("#two-cols + ul").css({
|
||
"column-count": "2",
|
||
});
|
||
$("#contributors + ul").css({
|
||
"column-count": "4",
|
||
});
|
||
</script>
|
||
|
||
|
||
|
||
<div>
|
||
<footer class="footer">
|
||
<hr/>
|
||
© 2002–2021 the Common Lisp Cookbook Project
|
||
</footer>
|
||
|
||
</div>
|
||
<div id="toc-btn">T<br>O<br>C</div>
|
||
</div>
|
||
|
||
<script text="javascript">
|
||
HighlightLisp.highlight_auto({className: null});
|
||
</script>
|
||
|
||
<script type="text/javascript">
|
||
function duckSearch() {
|
||
var searchField = document.getElementById("searchField");
|
||
if (searchField && searchField.value) {
|
||
var query = escape("site:lispcookbook.github.io/cl-cookbook/ " + searchField.value);
|
||
window.location.href = "https://duckduckgo.com/?kj=b2&kf=-1&ko=1&q=" + query;
|
||
// https://duckduckgo.com/params
|
||
// kj=b2: blue header in results page
|
||
// kf=-1: no favicons
|
||
}
|
||
}
|
||
</script>
|
||
|
||
<script async defer data-domain="lispcookbook.github.io/cl-cookbook" src="https://plausible.io/js/plausible.js"></script>
|
||
|
||
</body>
|
||
</html>
|