<astyle="font-size: 120%"href="https://www.udemy.com/course/common-lisp-programming/?couponCode=LISPY-XMAS2023"title="This course is under a paywall on the Udemy platform. Several videos are freely available so you can judge before diving in. vindarel is (I am) the main contributor to this Cookbook."> Discover our contributor's Lisp course with this Christmas coupon.</a>
📕 <ahref="index.html#download-in-epub">Get the EPUB and PDF</a>
</p>
<divid="content"
<p>For web development as for any other task, one can leverage Common Lisp’s
advantages: the unmatched REPL that even helps to interact with a running web
app, the exception handling system, performance, the ability to build a
self-contained executable, stability, good threads story, strong typing, etc. We
can, say, define a new route and try it right away, there is no need to restart
any running server. We can change and compile <em>one function at a time</em> (the
usual <code>C-c C-c</code> in Slime) and try it. The feedback is immediate. We can choose
the degree of interactivity: the web server can catch exceptions and fire the
interactive debugger, or print lisp backtraces on the browser, or display a 404
error page and print logs on standard output. The ability to build
self-contained executables eases deployment tremendously (compared to, for
example, npm-based apps), in that we just copy the executable to a server and
run it.</p>
<p>And when we have deployed our app, we can still interact with it,
allowing for hot reload, that even works when new dependencies have to
be installed. If you are careful and don’t want to use full live
reload, you might still enjoy this capability to reload, for example, a user’s
configuration file.</p>
<p>We’ll present here some established web frameworks and other common
libraries to help you getting started in developing a web
application. We do <em>not</em> aim to be exhaustive nor to replace the
upstream documentation. Your feedback and contributions are
appreciated.</p>
<!-- form creation, form validation -->
<!-- Javascript -->
<h2id="overview">Overview</h2>
<p><ahref="https://edicl.github.io/hunchentoot">Hunchentoot</a> and <ahref="https://github.com/fukamachi/clack">Clack</a> are two projects that
you’ll often hear about.</p>
<p>Hunchentoot is</p>
<blockquote>
<p>a web server and at the same time a toolkit for building dynamic websites. As a stand-alone web server, Hunchentoot is capable of HTTP/1.1 chunking (both directions), persistent connections (keep-alive), and SSL. It provides facilities like automatic session handling (with and without cookies), logging, customizable error handling, and easy access to GET and POST parameters sent by the client.</p>
</blockquote>
<p>It is a software written by Edi Weitz (“Common Lisp Recipes”,
<code>cl-ppcre</code> and <ahref="https://edicl.github.io/">much more</a>), it’s used and
proven solid. One can achieve a lot with it, but sometimes with more
friction than with a traditional web framework. For example,
dispatching a route by the HTTP method is a bit convoluted, one must
write a function for the <code>:uri</code> parameter that does the check, when it
is a built-in keyword in other frameworks like Caveman.</p>
<p>Clack is</p>
<blockquote>
<p>a web application environment for Common Lisp inspired by Python’s WSGI and Ruby’s Rack.</p>
</blockquote>
<p>Also written by a prolific lisper
(<ahref="https://github.com/fukamachi/">E. Fukamachi</a>), it actually uses
Hunchentoot by default as the server, but thanks to its pluggable
architecture one can use another web server, like the asynchronous
<ahref="https://github.com/fukamachi/woo">Woo</a>, built on the
“the fastest web server written in any programming language”.</p>
<p>We’ll cite also <ahref="https://github.com/orthecreedence/wookie">Wookie</a>, an asynchronous HTTP server, and its
companion library
<ahref="https://github.com/orthecreedence/cl-async">cl-async</a>, for general
purpose, non-blocking programming in Common Lisp, built on libuv, the
backend library in Node.js.</p>
<p>Clack being more recent and less documented, and Hunchentoot a
de-facto standard, we’ll concentrate on the latter for this
recipe. Your contributions are of course welcome.</p>
<p>Web frameworks build upon web servers and can provide facilities for
common activities in web development, like a templating system, access
to a database, session management, or facilities to build a REST api.</p>
<p>Some web frameworks include:</p>
<ul>
<li><ahref="https://github.com/fukamachi/caveman">Caveman</a>, by E. Fukamachi. It provides, out of the box,
database management, a templating engine (Djula), a project skeleton
generator, a routing system à la Flask or Sinatra, deployment options
(mod_lisp or FastCGI), support for Roswell on the command line, etc.</li>
<li><ahref="https://github.com/Shirakumo/radiance">Radiance</a>, by <ahref="https://github.com/Shinmera">Shinmera</a>
(Qtools, Portacle, lquery, …), is a web application environment,
more general than usual web frameworks. It lets us write and tie
websites and applications together, easing their deployment as a
whole. It has thorough <ahref="https://shirakumo.github.io/radiance/">documentation</a>, a <ahref="https://github.com/Shirakumo/radiance-tutorial">tutorial</a>, <ahref="https://github.com/Shirakumo/radiance-contribs">modules</a>, <ahref="https://github.com/Shirakumo?utf8=%E2%9C%93&q=radiance&type=&language=">pre-written applications</a> such as <ahref="https://github.com/Shirakumo/purplish">an image board</a> or a <ahref="https://github.com/Shirakumo/reader">blogging platform</a>, and more.
<p><ahref="https://edicl.github.io/hunchentoot/#define-easy-handler">define-easy-handler</a> allows to create a function and to bind it to an uri at once.</p>
<p>Its form follows</p>
<pre><code>define-easy-handler (function-name :uri <uri> …) (lambda list parameters)
</code></pre>
<p>where <code><uri></code> can be a string or a function.</p>
<p>Just a thought… we didn’t explicitly ask Hunchentoot to add this
route to our first acceptor of the port 4242. Let’s try another acceptor (see
previous section), on port 4444: <ahref="http://localhost:4444/yo?name=Bob">http://localhost:4444/yo?name=Bob</a> It
works too ! In fact, <code>define-easy-handler</code> accepts an <code>acceptor-names</code>
parameter:</p>
<blockquote>
<p>acceptor-names (which is evaluated) can be a list of symbols which means that the handler will only be returned by DISPATCH-EASY-HANDLERS in acceptors which have one of these names (see ACCEPTOR-NAME). acceptor-names can also be the symbol T which means that the handler will be returned by DISPATCH-EASY-HANDLERS in every acceptor.</p>
</blockquote>
<p>So, <code>define-easy-handler</code> has the following signature:</p>
<pre><code>define-easy-handler (function-name &key uri acceptor-names default-request-type) (lambda list parameters)
</code></pre>
<p>It also has a <code>default-parameter-type</code> which we’ll use in a minute to get url parameters.</p>
<p>There are also keys to know for the lambda list. Please see the documentation.</p>
<p><ahref="https://github.com/mmontone/easy-routes">easy-routes</a> is a route
handling extension on top of Hunchentoot. It provides:</p>
<ul>
<li><strong>dispatch</strong> based on the HTTP method, such as GET or POST (which is otherwise cumbersome to do in Hunchentoot)</li>
<li><strong>arguments extraction</strong> from the url path</li>
<li><strong>decorators</strong> (functions to run before the route body, typically used to add a layer of authentication or changing the returned content type)</li>
<li><strong>URL generation</strong> from route names and given URL parameters</li>
<li>visualization of routes</li>
<li>and more</li>
</ul>
<p>To use it, don’t create a server with <code>hunchentoot:easy-acceptor</code> but
with <code>easy-routes:easy-routes-acceptor</code>:</p>
(format nil "Hey~@[ ~A~] you are of type ~a" name (type-of name)))
</code></pre>
<p>Going to <ahref="http://localhost:4242/yo?name=Alice">http://localhost:4242/yo?name=Alice</a> returns</p>
<pre><code>Hey Alice you are of type (SIMPLE-ARRAY CHARACTER (5))
</code></pre>
<p>To automatically bind it to another type, we use <code>default-parameter-type</code>. It can be
one of those simple types:</p>
<ul>
<li><code>'string</code> (default),</li>
<li><code>'integer</code>,</li>
<li><code>'character</code> (accepting strings of length 1 only, otherwise it is nil)</li>
<li>or <code>'boolean</code></li>
</ul>
<p>or a compound list:</p>
<ul>
<li><code>'(:list <type>)</code></li>
<li><code>'(:array <type>)</code></li>
<li><code>'(:hash-table <type>)</code></li>
</ul>
<p>where <code><type></code> is a simple type.</p>
<h3id="accessing-a-json-request-body">Accessing a JSON request body</h3>
<h4id="hunchentoot-4">Hunchentoot</h4>
<p>To read a request body, use <code>hunchentoot:raw-post-data</code>, to which you
can add <code>:force-text t</code> to always get a string (and not a vector of
octets).</p>
<p>Then you can parse this string to JSON with the library of your choice (<ahref="https://github.com/Zulu-Inuoe/jzon/">jzon</a>, <ahref="https://github.com/yitzchak/shasht">shasht</a>…).</p>
<p>See also the generic function <code>maybe-invoke-debugger</code> if you want to
fine-tune this behaviour. You might want to specialize it on specific
condition classes (see below) for debugging purposes. The default method <ahref="http://www.lispworks.com/documentation/HyperSpec/Body/f_invoke.htm">invokes
the debugger</a>
if <code>*catch-errors-p*</code> is <code>nil</code>.</p>
<ul>
<li><code>*show-lisp-errors-p*</code>: set to <code>t</code> if you want to see errors in HTML output in the browser.</li>
<li><code>*show-lisp-backtraces-p*</code>: set to <code>nil</code> if the errors shown in HTML
output (when <code>*show-lisp-errors-p*</code> is <code>t</code>) should <em>not</em> contain
backtrace information (defaults to <code>t</code>, shows the backtrace).</li>
</ul>
<p>Hunchentoot defines condition classes. The superclass of all
conditions is <code>hunchentoot-condition</code>. The superclass of errors is <code>hunchentoot-error</code> (itself a subclass of <code>hunchentoot-condition</code>).</p>
<p>See the documentation: <ahref="https://edicl.github.io/hunchentoot/#conditions">https://edicl.github.io/hunchentoot/#conditions</a>.</p>
<h3id="clack">Clack</h3>
<p>Clack users might make a good use of plugins, like the clack-errors middleware: <ahref="https://github.com/CodyReichert/awesome-cl#clack-plugins">https://github.com/CodyReichert/awesome-cl#clack-plugins</a>.</p>
<p>It uses the Spinneret template engine by default, but we can bind any
other one of our choice.</p>
<p>To trigger an ajax event, we write lambdas in full Common Lisp:</p>
<pre><codeclass="language-lisp">...
(with-html
(:p (:input :type "checkbox"
:checked (done task)
:onclick (make-js-action
(lambda (&key &allow-other-keys)
(toggle task))))
...
</code></pre>
<p>The function <code>make-js-action</code> creates a simple javascript function
that calls the lisp one on the server, and automatically refreshes the
HTML of the widgets that need it. In our example, it re-renders one
task only.</p>
<p>Is it appealing ? Carry on this quickstart guide here: <ahref="http://40ants.com/weblocks/quickstart.html">http://40ants.com/weblocks/quickstart.html</a>.</p>
<h2id="templates">Templates</h2>
<h3id="djula---html-markup">Djula - HTML markup</h3>
<p><ahref="https://github.com/mmontone/djula">Djula</a> is a port of Python’s
Django template engine to Common Lisp. It has <ahref="https://mmontone.github.io/djula/djula/">excellent documentation</a>.</p>
<p>Caveman uses it by default, but otherwise it is not difficult to
setup. We must declare where our templates are with something like</p>
<p>Note that for efficiency Djula compiles the templates before rendering them.</p>
<p>It is, along with its companion
<ahref="https://github.com/AccelerationNet/access/">access</a> library, one of
the most downloaded libraries of Quicklisp.</p>
<h4id="djula-filters">Djula filters</h4>
<p>Filters allow to modify how a variable is displayed. Djula comes with
a good set of built-in filters and they are <ahref="https://mmontone.github.io/djula/doc/build/html/filters.html">well documented</a>. They are not to be confused with <ahref="https://mmontone.github.io/djula/doc/build/html/tags.html">tags</a>.</p>
<p>They look like this: ``, where <code>lower</code> is an
existing filter, which renders the text into lowercase.</p>
<p>Filters sometimes take arguments. For example: `` calls
the <code>add</code> filter with arguments <code>value</code> and 2.</p>
<p>Moreover, it is very easy to define custom filters. All we have to do
is to use the <code>def-filter</code> macro, which takes the variable as first
argument, and which can take more optional arguments.</p>
<p>Its general form is:</p>
<pre><codeclass="language-lisp">(def-filter :myfilter-name (value arg) ;; arg is optional
(body))
</code></pre>
<p>and it is used like this: ``.</p>
<p>Here’s how the <code>add</code> filter is defined:</p>
<p><em>Credit: <code>/u/arvid</code> on <ahref="https://www.reddit.com/r/learnlisp/comments/begcf9/can_someone_give_me_an_eli5_on_hiw_to_encrypt_and/">/r/learnlisp</a></em>.</p>
<h2id="runnning-and-building">Runnning and building</h2>
<h3id="running-the-application-from-source">Running the application from source</h3>
<p>To run our Lisp code from source, as a script, we can use the <code>--load</code>
switch from our implementation.</p>
<p>We must ensure:</p>
<ul>
<li>to load the project’s .asd system declaration (if any)</li>
<li>to install the required dependencies (this demands we have installed Quicklisp previously)</li>
<li>and to run our application’s entry point.</li>
<p>and we can ship this on Linux, Mac and Windows.</p>
<p>There is more:</p>
<blockquote>
<p>Ceramic applications are compiled down to native code, ensuring both performance and enabling you to deliver closed-source, commercial applications.</p>
<p>Then we have a command to <code>start</code> it, only now:</p>
<pre><code>sudo systemctl start my-app.service
</code></pre>
<p>and a command to install the service, to <strong>start the app after a boot
or reboot</strong> (that’s the “[Install]” part):</p>
<pre><code>sudo systemctl enable my-app.service
</code></pre>
<p>Then we can check its <code>status</code>:</p>
<pre><code>systemctl status my-app.service
</code></pre>
<p>and see our application’s <strong>logs</strong> (we can write to stdout or stderr,
and Systemd handles the logging):</p>
<pre><code>journalctl -u my-app.service
</code></pre>
<p>(you can also use the <code>-f</code> option to see log updates in real time, and in that case augment the number of lines with <code>-n 50</code> or <code>--lines</code>).</p>
<p>Systemd handles crashes and <strong>restarts the application</strong>. That’s the <code>Restart=on-failure</code> line.</p>
<p>Now keep in mind a couple things:</p>
<ul>
<li>we want our app to crash so that it can be re-started automatically:
you’ll want the <code>--disable-debugger</code> flag with SBCL.</li>
<li>Systemd will, by default, run your app as root. If you rely on your
Lisp to read your startup file (<code>~/.sbclrc</code>), especially to setup
Quicklisp, you will need to use the <code>--userinit</code> flag, or to set the
Systemd user with <code>User=xyz</code> in the <code>[service]</code> section. And if you
use a startup file, be aware that the line <code>(user-homedir-pathname)</code>
will not return the same result depending on the user, so the snippet
<p>There is nothing CL-specific to run your Lisp web app behind Nginx. Here’s an example to get you started.</p>
<p>We suppose you are running your Lisp app on a web server, with the IP
address 1.2.3.4, on the port 8001. Nothing special here. We want to
access our app with a real domain name (and eventuall benefit of other
Nginx’s advantages, such as rate limiting etc). We bought our domain
name and we created a DNS record of type A that links the domain name
to the server’s IP address.</p>
<p>We must configure our server with Nginx to tell it that all
connections coming from “your-domain-name.org”, on port 80, are to be
sent to the Lisp app running locally.</p>
<p>Create a new file: <code>/etc/nginx/sites-enabled/my-lisp-app.conf</code> and add this proxy directive:</p>
<pre><codeclass="language-lisp">server {
listen www.your-domain-name.org:80;
server_name your-domain-name.org www.your-domain-name.org; # with and without www
location / {
proxy_pass http://1.2.3.4:8001/;
}
# Optional: serve static files with nginx, not the Lisp app.
location /files/ {
proxy_pass http://1.2.3.4:8001/files/;
}
}
</code></pre>
<p>Note that on the proxy_pass directive: <code>proxy_pass
http://1.2.3.4:8001/;</code> we are using our server’s public IP
address. Oten, your Lisp webserver such as Hunchentoot directly
listens on it. You might want, for security reasons, to run the Lisp
app on localhost.</p>
<p>Reload nginx (send the “reload” signal):</p>
<pre><code>$ nginx -s reload
</code></pre>
<p>and that’s it: you can access your Lisp app from the outside through <code>http://www.your-domain-name.org</code>.</p>
<h3id="deploying-on-heroku-and-other-services">Deploying on Heroku and other services</h3>
<p>See <ahref="https://gitlab.com/duncan-bayne/heroku-buildpack-common-lisp">heroku-buildpack-common-lisp</a> and the <ahref="https://github.com/CodyReichert/awesome-cl#deployment">Awesome CL#deploy</a> section for interface libraries for Kubernetes, OpenShift, AWS, etc.</p>
for a Grafana dashboard for SBCL and Hunchentoot metrics (memory,
threads, requests per second,…).</p>
<h2id="connecting-to-a-remote-lisp-image">Connecting to a remote Lisp image</h2>
<p>This this section: <ahref="debugging.html#remote-debugging">debugging#remote-debugging</a>.</p>
<h2id="hot-reload">Hot reload</h2>
<p>This is an example from <ahref="https://github.com/stylewarning/quickutil/blob/master/quickutil-server/">Quickutil</a>. It is actually an automated version of the precedent section.</p>
