marcuskammer/emacs.d
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
emacs.d/clones/lispcookbook.github.io/cl-cookbook/scripting.html

596 lines
23 KiB
HTML
Raw Normal View History

Add clones websites
2022-08-02 12:34:59 +02:00
<!DOCTYPE html>
<html lang="en">
<head>
<meta name="generator" content=
"HTML Tidy for HTML5 for Linux version 5.2.0">
<title>Scripting. Command line arguments. Executables.</title>
<meta charset="utf-8">
<meta name="description" content="A collection of examples of using Common Lisp">
<meta name="viewport" content=
"width=device-width, initial-scale=1">
<link rel="stylesheet" href=
"assets/style.css">
<script type="text/javascript" src=
"assets/highlight-lisp.js">
</script>
<script type="text/javascript" src=
"assets/jquery-3.2.1.min.js">
</script>
<script type="text/javascript" src=
"assets/jquery.toc/jquery.toc.min.js">
</script>
<script type="text/javascript" src=
"assets/toggle-toc.js">
</script>
<link rel="stylesheet" href=
"assets/github.css">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
</head>
<body>
<h1 id="title-xs"><a href="index.html">The Common Lisp Cookbook</a> &ndash; Scripting. Command line arguments. Executables.</h1>
<div id="logo-container">
<a href="index.html">
<img id="logo" src="assets/cl-logo-blue.png"/>
</a>
<div id="searchform-container">
<form onsubmit="duckSearch()" action="javascript:void(0)">
<input id="searchField" type="text" value="" placeholder="Search...">
</form>
</div>
<div id="toc-container" class="toc-close">
<div id="toc-title">Table of Contents</div>
<ul id="toc" class="list-unstyled"></ul>
</div>
</div>
<div id="content-container">
<h1 id="title-non-xs"><a href="index.html">The Common Lisp Cookbook</a> &ndash; Scripting. Command line arguments. Executables.</h1>
<!-- Announcement we can keep for 1 month or more. I remove it and re-add it from time to time. -->
<p class="announce">
📹 <a href="https://www.udemy.com/course/common-lisp-programming/?couponCode=6926D599AA-LISP4ALL">NEW! Learn Lisp in videos and support our contributors with this 40% discount.</a>
</p>
<p class="announce-neutral">
📕 <a href="index.html#download-in-epub">Get the EPUB and PDF</a>
</p>
<div id="content"
<p>Using a program from a REPL is fine and well, but if we want to
distribute our program easily, we’ll want to build an executable.</p>
<p>Lisp implementations differ in their processes, but they all create
<strong>self-contained executables</strong>, for the architecture they are built on. The
final user doesn’t need to install a Lisp implementation, he can run
the software right away.</p>
<p><strong>Start-up times</strong> are near to zero, specially with SBCL and CCL.</p>
<p>Binaries <strong>size</strong> are large-ish. They include the whole Lisp
including its libraries, the names of all symbols, information about
argument lists to functions, the compiler, the debugger, source code
location information, and more.</p>
<p>Note that we can similarly build self-contained executables for <strong>web apps</strong>.</p>
<h2 id="building-a-self-contained-executable">Building a self-contained executable</h2>
<h3 id="with-sbcl---images-and-executables">With SBCL - Images and Executables</h3>
<p>How to build (self-contained) executables is, by default, implementation-specific (see
below for portable ways). With SBCL, as says
<a href="http://www.sbcl.org/manual/index.html#Function-sb_002dext_003asave_002dlisp_002dand_002ddie">its documentation</a>,
it is a matter of calling <code>save-lisp-and-die</code> with the <code>executable</code> argument to T:</p>
<pre><code class="language-lisp">(sb-ext:save-lisp-and-die #P"path/name-of-executable" :toplevel #'my-app:main-function :executable t)
</code></pre>
<p><code>sb-ext</code> is an SBCL extension to run external processes. See other
<a href="http://www.sbcl.org/manual/index.html#Extensions">SBCL extensions</a>
(many of them are made implementation-portable in other libraries).</p>
<p><code>:executable t</code> tells to build an executable instead of an
image. We could build an image to save the state of our current
Lisp image, to come back working with it later. This is especially useful if
we made a lot of work that is computing intensive.
In that case, we re-use the image with <code>sbcl --core name-of-image</code>.</p>
<p><code>:toplevel</code> gives the program’s entry point, here <code>my-app:main-function</code>. Don’t forget to <code>export</code> the symbol, or use <code>my-app::main-function</code> (with two colons).</p>
<p>If you try to run this in Slime, you’ll get an error about threads running:</p>
<blockquote>
<p>Cannot save core with multiple threads running.</p>
</blockquote>
<p>So we must run the command from a simple SBCL repl.</p>
<p>I suppose your project has Quicklisp dependencies. You must then:</p>
<ul>
<li>ensure Quicklisp is installed and loaded at the Lisp startup (you
completed Quicklisp installation),</li>
<li><code>load</code> the project’s .asd,</li>
<li>install the dependencies,</li>
<li>build the executable.</li>
</ul>
<p>That gives:</p>
<pre><code class="language-lisp">(load "my-app.asd")
(ql:quickload "my-app")
(sb-ext:save-lisp-and-die #p"my-app-binary" :toplevel #'my-app:main :executable t)
</code></pre>
<p>From the command line, or from a Makefile, use <code>--load</code> and <code>--eval</code>:</p>
<pre><code>build:
sbcl --load my-app.asd \
--eval '(ql:quickload :my-app)' \
--eval "(sb-ext:save-lisp-and-die #p\"my-app\" :toplevel #'my-app:main :executable t)"
</code></pre>
<h3 id="with-asdf">With ASDF</h3>
<p>Now that we’ve seen the basics, we need a portable method. Since its
version 3.1, ASDF allows to do that. It introduces the <a href="https://common-lisp.net/project/asdf/asdf.html#Convenience-Functions"><code>make</code> command</a>,
that reads parameters from the .asd. Add this to your .asd declaration:</p>
<pre><code>:build-operation "program-op" ;; leave as is
:build-pathname "&lt;binary-name&gt;"
:entry-point "&lt;my-package:main-function&gt;"
</code></pre>
<p>and call <code>asdf:make :my-package</code>.</p>
<p>So, in a Makefile:</p>
<pre><code class="language-lisp">LISP ?= sbcl
build:
$(LISP) --load my-app.asd \
--eval '(ql:quickload :my-app)' \
--eval '(asdf:make :my-app)' \
--eval '(quit)'
</code></pre>
<h3 id="with-roswell-or-buildapp">With Roswell or Buildapp</h3>
<p><a href="https://roswell.github.io">Roswell</a>, an implementation manager, script launcher and
much more, has the <code>ros build</code> command, that should work for many
implementations.</p>
<p>This is how we can make our application easily installable by others, with a <code>ros install
my-app</code>. See Roswell’s documentation.</p>
<p>Be aware that <code>ros build</code> adds core compression by default. That adds
a significant startup overhead of the order of 150ms (for a simple
app, startup time went from about 30ms to 180ms). You can disable it
with <code>ros build &lt;app.ros&gt; --disable-compression</code>. Of course, core
compression reduces your binary size significantly. See the table
below, “Size and startup times of executables per implementation”.</p>
<p>We’ll finish with a word on
<a href="http://www.xach.com/lisp/buildapp/">Buildapp</a>, a battle-tested and
still popular “application for SBCL or CCL that configures and saves
an executable Common Lisp image”.</p>
<p>Example usage:</p>
<pre><code class="language-lisp">buildapp --output myapp \
--asdf-path . \
--asdf-tree ~/quicklisp/dists \
--load-system my-app \
--entry my-app:main
</code></pre>
<p>Many applications use it (for example,
<a href="https://github.com/dimitri/pgloader">pgloader</a>), it is available on
Debian: <code>apt install buildapp</code>, but you shouldn’t need it now with asdf:make or Roswell.</p>
<h3 id="for-web-apps">For web apps</h3>
<p>We can similarly build a self-contained executable for our web appplication. It
would thus contain a web server and would be able to run on the
command line:</p>
<pre><code>$ ./my-web-app
Hunchentoot server is started.
Listening on localhost:9003.
</code></pre>
<p>Note that this runs the production webserver, not a development one,
so we can run the binary on our VPS right away and access the application from
the outside.</p>
<p>We have one thing to take care of, it is to find and put the thread of
the running web server on the foreground. In our <code>main</code> function, we
can do something like this:</p>
<pre><code class="language-lisp">(defun main ()
(start-app :port 9003) ;; our start-app, for example clack:clack-up
;; let the webserver run.
;; warning: hardcoded "hunchentoot".
(handler-case (bt:join-thread (find-if (lambda (th)
(search "hunchentoot" (bt:thread-name th)))
(bt:all-threads)))
;; Catch a user's C-c
(#+sbcl sb-sys:interactive-interrupt
#+ccl ccl:interrupt-signal-condition
#+clisp system::simple-interrupt-condition
#+ecl ext:interactive-interrupt
#+allegro excl:interrupt-signal
() (progn
(format *error-output* "Aborting.~&amp;")
(clack:stop *server*)
(uiop:quit)))
(error (c) (format t "Woops, an unknown error occured:~&amp;~a~&amp;" c))))
</code></pre>
<p>We used the <code>bordeaux-threads</code> library (<code>(ql:quickload
"bordeaux-threads")</code>, alias <code>bt</code>) and <code>uiop</code>, which is part of ASDF so
already loaded, in order to exit in a portable way (<code>uiop:quit</code>, with
an optional return code, instead of <code>sb-ext:quit</code>).</p>
<h3 id="size-and-startup-times-of-executables-per-implementation">Size and startup times of executables per implementation</h3>
<p>SBCL isn’t the only Lisp implementation.
<a href="https://gitlab.com/embeddable-common-lisp/ecl/">ECL</a>, Embeddable
Common Lisp, transpiles Lisp programs to C. That creates a smaller
executable.</p>
<p>According to
<a href="https://www.reddit.com/r/lisp/comments/46k530/tackling_the_eternal_problem_of_lisp_image_size/">this reddit source</a>, ECL produces indeed the smallest executables of all,
an order of magnitude smaller than SBCL, but with a longer startup time.</p>
<p>CCL’s binaries seem to be as fast to start up as SBCL and nearly half the size.</p>
<pre><code>| program size | implementation | CPU | startup time |
|--------------+----------------+------+--------------|
| 28 | /bin/true | 15% | .0004 |
| 1005 | ecl | 115% | .5093 |
| 48151 | sbcl | 91% | .0064 |
| 27054 | ccl | 93% | .0060 |
| 10162 | clisp | 96% | .0170 |
| 4901 | ecl.big | 113% | .8223 |
| 70413 | sbcl.big | 93% | .0073 |
| 41713 | ccl.big | 95% | .0094 |
| 19948 | clisp.big | 97% | .0259 |
</code></pre>
<p>You’ll also want to investigate the proprietary Lisps’ tree shakers capabilities.</p>
<p>Regarding compilation times, CCL is famous for being fast in that regards.
ECL is more involved and takes the longer to compile of these three implementations.</p>
<h3 id="building-a-smaller-binary-with-sbcls-core-compression">Building a smaller binary with SBCL’s core compression</h3>
<p>Building with SBCL’s core compression can dramatically reduce your
application binary’s size. In our case, we passed from 120MB to 23MB,
for a loss of a dozen milliseconds of start-up time, which was still
under 50ms!</p>
<p>Your SBCL must be built with core compression, see the documentation: <a href="http://www.sbcl.org/manual/#Saving-a-Core-Image">http://www.sbcl.org/manual/#Saving-a-Core-Image</a></p>
<p>Is it the case ?</p>
<pre><code class="language-lisp">(find :sb-core-compression *features*)
:SB-CORE-COMPRESSION
</code></pre>
<p>Yes, it is the case with this SBCL installed from Debian.</p>
<p><strong>With SBCL</strong></p>
<p>In SBCL, we would give an argument to <code>save-lisp-and-die</code>, where
<code>:compression</code></p>
<blockquote>
<p>may be an integer from -1 to 9, corresponding to zlib compression levels, or t (which is equivalent to the default compression level, -1).</p>
</blockquote>
<p>We experienced a 1MB difference between levels -1 and 9.</p>
<p><strong>With ASDF</strong></p>
<p>However, we prefer to do this with ASDF (or rather, UIOP). Add this in your .asd:</p>
<pre><code class="language-lisp">#+sb-core-compression
(defmethod asdf:perform ((o asdf:image-op) (c asdf:system))
(uiop:dump-image (asdf:output-file o c) :executable t :compression t))
</code></pre>
<p><strong>With Deploy</strong></p>
<p>Also, the <a href="https://github.com/Shinmera/deploy/">Deploy</a> library can be used
to build a fully standalone application. It will use compression if available.</p>
<p>Deploy is specifically geared towards applications with foreign
library dependencies. It collects all the foreign shared libraries of
dependencies, such as libssl.so in the <code>bin</code> subdirectory.</p>
<p>And voilà !</p>
<h2 id="parsing-command-line-arguments">Parsing command line arguments</h2>
<p>SBCL stores the command line arguments into <code>sb-ext:*posix-argv*</code>.</p>
<p>But that variable name differs from implementations, so we want a
way to handle the differences for us.</p>
<p>We have <code>uiop:command-line-arguments</code>, shipped in ASDF and included in
nearly all implementations.
From anywhere in your code, you can simply check if a given string is present in this list:</p>
<pre><code class="language-lisp">(member "-h" (uiop:command-line-arguments) :test #'string-equal)
</code></pre>
<p>That’s good, but we also want to parse the arguments, have facilities to check short and long options, build a help message automatically, etc.</p>
<p>A quick look at the
<a href="https://github.com/CodyReichert/awesome-cl#scripting">awesome-cl#scripting</a>
list made us choose the
<a href="https://github.com/mrkkrp/unix-opts">unix-opts</a> library.</p>
<pre><code>(ql:quickload "unix-opts")
</code></pre>
<p>We can call it with its <code>opts</code> alias (a global nickname).</p>
<p>As often work happens in two phases:</p>
<ul>
<li>declaring the options that our application accepts, their optional argument, defining their type
(string, integer,…), their long and short names, and the required ones</li>
<li>parsing them (and handling missing or malformed parameters).</li>
</ul>
<h3 id="declaring-arguments">Declaring arguments</h3>
<p>We define the arguments with <code>opts:define-opts</code>:</p>
<pre><code class="language-lisp">(opts:define-opts
(:name :help
:description "print this help text"
:short #\h
:long "help")
(:name :nb
:description "here we want a number argument"
:short #\n
:long "nb"
:arg-parser #'parse-integer) ;; &lt;- takes an argument
(:name :info
:description "info"
:short #\i
:long "info"))
</code></pre>
<p>Here <code>parse-integer</code> is a built-in CL function. If the argument you expect is a string, you don’t have to define an <code>arg-parser</code>.</p>
<p>Here is an example output on the command line after we build and run a binary of our application. The help message was auto-generated:</p>
<pre><code>$ my-app -h
my-app. Usage:
Available options:
-h, --help print this help text
-n, --nb ARG here we want a number argument
-i, --info info
</code></pre>
<h3 id="parsing">Parsing</h3>
<p>We parse and get the arguments with <code>opts:get-opts</code>, which returns two
values: the list of valid options and the remaining free arguments. We
then must use <code>multiple-value-bind</code> to assign both into variables:</p>
<pre><code class="language-lisp"> (multiple-value-bind (options free-args)
;; There is no error handling yet.
(opts:get-opts)
...
</code></pre>
<p>We can test this by giving a list of strings to <code>get-opts</code>:</p>
<pre><code class="language-lisp">(multiple-value-bind (options free-args)
(opts:get-opts '("hello" "-h" "-n" "1"))
(format t "Options: ~a~&amp;" options)
(format t "free args: ~a~&amp;" free-args))
Options: (HELP T NB-RESULTS 1)
free args: (hello)
NIL
</code></pre>
<p>If we put an unknown option, we get into the debugger. We’ll see
error handling in a moment.</p>
<p>So <code>options</code> is a
<a href="data-structures.html#plist">property list</a>. We
use <code>getf</code> and <code>setf</code> with plists, so that’s how we do our
logic. Below we print the help with <code>opts:describe</code> and then we <code>quit</code>
(in a portable way).</p>
<pre><code class="language-lisp"> (multiple-value-bind (options free-args)
(opts:get-opts)
(if (getf options :help)
(progn
(opts:describe
:prefix "You're in my-app. Usage:"
:args "[keywords]") ;; to replace "ARG" in "--nb ARG"
(uiop:quit)))
(if (getf options :nb)
...)
</code></pre>
<p>For a full example, see its
<a href="https://github.com/mrkkrp/unix-opts/blob/master/example/example.lisp">official example</a>
and
<a href="https://vindarel.github.io/cl-torrents/tutorial.html">cl-torrents’ tutorial</a>.</p>
<p>The example in the unix-opts repository suggests a macro to do
slightly better. Now to error handling.</p>
<h4 id="handling-malformed-or-missing-arguments">Handling malformed or missing arguments</h4>
<p>There are 4 situations that unix-opts doesn’t handle, but signals
conditions for us to take care of:</p>
<ul>
<li>when it sees an unknown argument, an <code>unknown-option</code> condition is signaled.</li>
<li>when an argument is missing, it signals a <code>missing-arg</code> condition.</li>
<li>when it can’t parse an argument, it signals <code>arg-parser-failed</code>. For example, if it expected an integer but got text.</li>
<li>when it doesn’t see a required option, it signals <code>missing-required-option</code>.</li>
</ul>
<p>So, we must create simple functions to handle those conditions, and
surround the parsing of the options with an <code>handler-bind</code> form:</p>
<pre><code class="language-lisp"> (multiple-value-bind (options free-args)
(handler-bind ((opts:unknown-option #'unknown-option) ;; the condition / our function
(opts:missing-arg #'missing-arg)
(opts:arg-parser-failed #'arg-parser-failed)
(opts:missing-required-option))
(opts:get-opts))
…
;; use "options" and "free-args"
</code></pre>
<p>Here we suppose we want one function to handle each case, but it could
be a simple one. They take the condition as argument.</p>
<pre><code class="language-lisp">(defun handle-arg-parser-condition (condition)
(format t "Problem while parsing option ~s: ~a .~%" (opts:option condition) ;; reader to get the option from the condition.
condition)
(opts:describe) ;; print help
(uiop:quit 1))
</code></pre>
<p>For more about condition handling, see <a href="error_handling.html">error and condition handling</a>.</p>
<h4 id="catching-a-c-c-termination-signal">Catching a C-c termination signal</h4>
<p>Let’s build a simple binary, run it, try a <code>C-c</code> and read the stacktrace:</p>
<pre><code>$ ./my-app
sleep…
^C
debugger invoked on a SB-SYS:INTERACTIVE-INTERRUPT in thread &lt;== condition name
#&lt;THREAD "main thread" RUNNING {1003156A03}&gt;:
Interactive interrupt at #x7FFFF6C6C170.
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [CONTINUE ] Return from SB-UNIX:SIGINT. &lt;== it was a SIGINT indeed
1: [RETRY-REQUEST] Retry the same request.
</code></pre>
<p>The signaled condition is named after our implementation:
<code>sb-sys:interactive-interrupt</code>. We just have to surround our
application code with a <code>handler-case</code>:</p>
<pre><code class="language-lisp">(handler-case
(run-my-app free-args)
(sb-sys:interactive-interrupt () (progn
(format *error-output* "Abort.~&amp;")
(opts:exit))))
</code></pre>
<p>This code is only for SBCL though. We know about
<a href="https://github.com/guicho271828/trivial-signal/">trivial-signal</a>,
but we were not satisfied with our test yet. So we can use something
like this:</p>
<pre><code class="language-lisp">(handler-case
(run-my-app free-args)
(#+sbcl sb-sys:interactive-interrupt
#+ccl ccl:interrupt-signal-condition
#+clisp system::simple-interrupt-condition
#+ecl ext:interactive-interrupt
#+allegro excl:interrupt-signal
()
(opts:exit)))
</code></pre>
<p>here <code>#+</code> includes the line at compile time depending on
the implementation. There’s also <code>#-</code>. What <code>#+</code> does is to look for
symbols in the <code>*features*</code> list. We can also combine symbols with
<code>and</code>, <code>or</code> and <code>not</code>.</p>
<h2 id="continuous-delivery-of-executables">Continuous delivery of executables</h2>
<p>We can make a Continuous Integration system (Travis CI, Gitlab CI,…)
build binaries for us at every commit, or at every tag pushed or at
whichever other policy.</p>
<p>See <a href="testing.html#continuous-integration">Continuous Integration</a>.</p>
<h2 id="credit">Credit</h2>
<ul>
<li><a href="https://vindarel.github.io/cl-torrents/tutorial.html">cl-torrents’ tutorial</a></li>
<li><a href="https://lisp-journey.gitlab.io/web-dev/">lisp-journey/web-dev</a></li>
</ul>
<p class="page-source">
Page source: <a href="https://github.com/LispCookbook/cl-cookbook/blob/master/scripting.md">scripting.md</a>
</p>
</div>
<script type="text/javascript">
// Don't write the TOC on the index.
if (window.location.pathname != "/cl-cookbook/") {
$("#toc").toc({
content: "#content", // will ignore the first h1 with the site+page title.
headings: "h1,h2,h3,h4"});
}
$("#two-cols + ul").css({
"column-count": "2",
});
$("#contributors + ul").css({
"column-count": "4",
});
</script>
<div>
<footer class="footer">
<hr/>
&copy; 2002&ndash;2021 the Common Lisp Cookbook Project
</footer>
</div>
<div id="toc-btn">T<br>O<br>C</div>
</div>
<script text="javascript">
HighlightLisp.highlight_auto({className: null});
</script>
<script type="text/javascript">
function duckSearch() {
var searchField = document.getElementById("searchField");
if (searchField && searchField.value) {
var query = escape("site:lispcookbook.github.io/cl-cookbook/ " + searchField.value);
window.location.href = "https://duckduckgo.com/?kj=b2&kf=-1&ko=1&q=" + query;
// https://duckduckgo.com/params
// kj=b2: blue header in results page
// kf=-1: no favicons
}
}
</script>
<script async defer data-domain="lispcookbook.github.io/cl-cookbook" src="https://plausible.io/js/plausible.js"></script>
</body>
</html>
Reference in a new issue Copy permalink