emacs.d/lp/lisp-babel.org

#+TITLE: lisp-babel
# #+DATE: <2013-08-31 Sat>
#+AUTHOR: Derek Feichtinger
#+EMAIL: derek.feichtinger@psi.ch
#+OPTIONS: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline
#+OPTIONS: author:t c:nil creator:comment d:(not LOGBOOK) date:t e:t
#+OPTIONS: email:nil f:t inline:t num:t p:nil pri:nil stat:t tags:t
#+OPTIONS: tasks:t tex:t timestamp:t toc:t todo:t |:t
# #+CREATOR: Emacs 24.3.1 (Org mode 8.0.7)
#+DESCRIPTION:
#+EXCLUDE_TAGS: noexport
#+KEYWORDS:
#+LANGUAGE: en
#+SELECT_TAGS: export

# By default I do not want that source code blocks are evaluated on export. Usually
# I want to evaluate them interactively and retain the original results.
#+PROPERTY: header-args :eval never-export

# #+SETUPFILE: ~/.emacs.d/git-submod/org-html-themes.git/setup/theme-readtheorg.setup
# #+SETUPFILE: ~/.emacs.d/git-submod/org-html-themes.git/setup/theme-bigblow.setup


* How to use this document
  You should look at this document in its Org mode source form. The
  PDF rendering is useful to see the results of some of the export
  options, but the syntax of the source block is only seen in the
  source text.
  
* Version information
  #+BEGIN_SRC emacs-lisp :results output :exports both
    (princ (concat (format "Emacs version: %s\n" (emacs-version))
                   (format "org version: %s\n" (org-version))))
    
  #+END_SRC

  #+RESULTS:
  : Emacs version: GNU Emacs 24.5.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8)
  :  of 2015-05-04 on dflt1w
  : org version: 8.3.2

* using a table as input for a src block
** simple example
   We first create a table from a lisp *list of lists*. Each inner list
   will form a row in the resulting table. I already insert a header
   row with the names of three columns. A separator line can be obtained
   by putting the =hline= symbol into the resulting list.

   #+NAME: make-table1
   #+BEGIN_SRC emacs-lisp :results value :exports both
     (cons '(col1 col2 col3)
           (cons 'hline
                 (loop for i from 5 to 15 collect `(,i ,(* i 5) ""))))
   #+END_SRC

   #+NAME: table1
   #+RESULTS: make-table1
   | col1 | col2 | col3 |
   |------+------+------|
   |    5 |   25 |      |
   |    6 |   30 |      |
   |    7 |   35 |      |
   |    8 |   40 |      |
   |    9 |   45 |      |
   |   10 |   50 |      |
   |   11 |   55 |      |
   |   12 |   60 |      |
   |   13 |   65 |      |
   |   14 |   70 |      |
   |   15 |   75 |      |


   We now can fill the third column by passing the table into the next
   source block.  We force babel to treat the first row as table header
   by using the *:colnames yes* header argument. This also causes the
   result table to contain the headers (as long as the new table has the
   same number of columns as the original table)

   Here I also demonstrate the use of the *-n* option that will export
   the code with line numbers.

   #+BEGIN_SRC emacs-lisp -n :results value :var tbl=table1 :colnames yes :exports both
     (let (result)
       (dolist (row tbl result)
         (setf (nth 2 row) (* 2 (nth 1 row)))
         (setq result (cons row result)))
       (reverse result))
   #+END_SRC

   #+RESULTS:
   | col1 | col2 | col3 |
   |------+------+------|
   |    5 |   25 |   50 |
   |    6 |   30 |   60 |
   |    7 |   35 |   70 |
   |    8 |   40 |   80 |
   |    9 |   45 |   90 |
   |   10 |   50 |  100 |
   |   11 |   55 |  110 |
   |   12 |   60 |  120 |
   |   13 |   65 |  130 |
   |   14 |   70 |  140 |
   |   15 |   75 |  150 |

** passing a sub-range
   It is possible to specify a sub-range for the table that is handed over through =:var=. But currently
   it does not work well with the =:colnames yes= option, as the following example shows.

   #+BEGIN_SRC emacs-lisp -n :results value :var tbl=table1[4:8] :colnames yes :exports both
     (let (result)
       (dolist (row tbl result)
         (setf (nth 2 row) (* 2 (nth 1 row)))
         (setq result (cons row result)))
       (reverse result))
   #+END_SRC

   #+RESULTS:
   |  7 | 35 |     |
   |----+----+-----|
   |  8 | 40 |  80 |
   |  9 | 45 |  90 |
   | 10 | 50 | 100 |
   | 11 | 55 | 110 |

** Investigating how tables are passed to the src block

   #+NAME: tableCheckConv
   | col1 | col2 | col3        |
   |------+------+-------------|
   |   10 | str  | two strings |
   | 20.5 | str2 | 2 strings   |

   #+BEGIN_SRC emacs-lisp :results output :var tbl=tableCheckConv :colnames yes :exports both
     (pp tbl)
   #+END_SRC

   #+RESULTS:
   : ((10 "str" "two strings")
   :  (20.5 "str2" "2 strings"))



   Note that the =raw value= output of the source block does not yield
   the same. It loses the string quotes of the single entries!
   #+BEGIN_SRC emacs-lisp :results raw value :var tbl=tableCheckConv :colnames yes :exports both
     tbl
   #+END_SRC

   #+RESULTS:
   ((10 str two strings) (20 str2 2 strings))

* TODO using a list as input for a source block

  #+NAME: mylist
  - item1
  - item2
    1. subitem2.1
    2. subitem2.2
  - item3
    - subitem3.1
  - item4

  Let's look at the resulting structure that is passed to a source block

  #+BEGIN_SRC elisp :results output :var lst=mylist
  (pp lst)
  #+END_SRC

  #+RESULTS:
  : (("item1")
  :  ("item2"
  :   (ordered
  :    ("subitem2.1")
  :    ("subitem2.2")))
  :  ("item3"
  :   (unordered
  :    ("subitem3.1")))
  :  ("item4"))

  This is different from the current entry in the Manual ([[info:org#var][info:org#var]]), where it is said that only top
  level items are passed along. But this complete passing along of the structure opens nice and
  interesting ways of using lists. Need to ask whether this interface will remain stable.

  One important point to clarify is, why is the returned structure not exactly the result of
  what the Org function =org-list-to-lisp= returns? Comparing with that output we see that the
  current handing over of a list by the =:var= argument is losing the outermost layer of information
  that describes whether the top level list is of the ordered, unordered, ... type.

  #+BEGIN_SRC elisp :results output :var lname="mylist"
    (pp (save-excursion
	  (goto-char (point-min))
	  (unless (search-forward-regexp (concat  "#\\\+NAME: .*" lname) nil t)
	    (error "No list of this name found: %s" lname))
	  (forward-line 1)
	  (org-list-to-lisp)))
  #+END_SRC

  #+RESULTS:
  #+begin_example
  (unordered
   ("item1")
   ("item2"
    (ordered
     ("subitem2.1")
     ("subitem2.2")))
   ("item3"
    (unordered
     ("subitem3.1")))
   ("item4"))
  #+end_example

  
* calling source blocks as a function
** Chaining source block execution
   I *can have another piece of code implicitly called* by using its
   name as an input variable in another code block. So, I could
   directly fill the third column of our initial example table without
   ever having to print out that table table. We can just pass into the
   next function a variable =tbl= and the name of the initial code
   block =make-table1=.

   #+BEGIN_SRC emacs-lisp  :results value :var tbl=make-table1 :colnames yes
     (let (result)
       (dolist (row tbl result)
         (setf (nth 2 row) (* 2 (nth 1 row)))
         (setq result (cons row result)))
       (reverse result))
   #+END_SRC

   #+RESULTS:
   | col1 | col2 | col3 |
   |------+------+------|
   |    5 |   25 |   50 |
   |    6 |   30 |   60 |
   |    7 |   35 |   70 |
   |    8 |   40 |   80 |
   |    9 |   45 |   90 |
   |   10 |   50 |  100 |
   |   11 |   55 |  110 |
   |   12 |   60 |  120 |
   |   13 |   65 |  130 |
   |   14 |   70 |  140 |
   |   15 |   75 |  150 |

** simple call syntax using CALL

   We first define a function in a named code block called =mydouble=. The
   variable x will be passed in by defining a header argument =:var x=

   #+NAME: mydouble
   #+header: :var x=2
   #+BEGIN_SRC emacs-lisp :results silent :exports code
   (* 2 x)
   #+END_SRC

   Now we can call this babel function by using the code block's name
   =mydouble= from any place in the document. For example:
   #+CALL: mydouble(x=5)

   #+RESULTS:
   : 10


   Another example where we pass in two variables x and y.
   #+NAME: mydivide
   #+header: :var x=2 y=3
   #+BEGIN_SRC emacs-lisp :results silent :exports code
   (/ x y)
   #+END_SRC

   Note that *you can/must pass additional header arguments* to the
   call. The ones added at the end influence the final result
   (e.g. putting it into a drawer), while the ones added in [] are
   evaluated in the context of the original definition (e.g whether to
   capture the output or return a value).
   #+CALL: mydivide(12,3) :results value

   #+RESULTS:
   : 4

   Another alternative calling syntax

   #+CALL: mydivide(y=2,x=10)

   #+RESULTS:
   : 5

** calling a source block function from elisp

   This function is also used for table formulas

   #+BEGIN_SRC elisp
   (org-sbe mydivide (x 10) (y 3))
   #+END_SRC

   #+RESULTS:
   : 3

   #+NAME: srcRepeatStrings
   #+BEGIN_SRC elisp :results output :var s1="hello" s2="world"
   (princ (concat s1 " " s2))
   #+END_SRC

   #+RESULTS: srcRepeatStrings
   : hello world

   #+BEGIN_SRC elisp :results output
     (princ (org-sbe srcRepeatStrings (s1 $"hello") (s2 $"world")))
   #+END_SRC

   #+RESULTS:
   : 
   : (s1 (quote "hello"))
   : 
   : (s2 (quote "world"))
   : 
   : (results (quote "hello world"))
   : hello world


   #+BEGIN_SRC elisp :results output
     (org-babel-execute-src-block nil
                    		     '("emacs-lisp" "results"
                    		       ((:var . "results=mydivide[](x=30, y=5)")))
                         	     '((:results . "silent")))
   #+END_SRC

   #+RESULTS:
   : 
   : (x (quote 30))
   : 
   : (y (quote 5))
   : 
   : (results (quote 6))

   #+BEGIN_SRC elisp :results value :results value
     (org-babel-execute-src-block nil
                              	   '("emacs-lisp" "(princ (format \"%s haahahaa\n\"results))"
                              	     ((:var . "results=mydivide[](x=30, y=5)")))
                                '((:results . "silent")))
   #+END_SRC

   #+RESULTS:
   : 6 haahahaa

   
** Naming an output table produced by a CALL

   If the called function produces an output table that one wants to
   use in subsequent function calls or in table formulas (using the
   =remote= keyword) one can give the CALL a name utilizing the syntax
   used for other org elements:
  
   #+NAME: new-make-table1
   #+CALL: make-table1()

   #+RESULTS: new-make-table1
   | col1 | col2 | col3 |
   |------+------+------|
   |    5 |   25 |      |
   |    6 |   30 |      |
   |    7 |   35 |      |
   |    8 |   40 |      |
   |    9 |   45 |      |
   |   10 |   50 |      |
   |   11 |   55 |      |
   |   12 |   60 |      |
   |   13 |   65 |      |
   |   14 |   70 |      |
   |   15 |   75 |      |

   
* Inline src calls

  Basic inline source calls:

  - src_emacs-lisp[:var tbl=table1]{(nth 0 (nth (- (length tbl) 1) tbl))} {{{results(=15=)}}}

  - src_emacs-lisp[:results value]{(org-table-get-remote-range "table1" "@>$1" )} {{{results(=15=)}}} 

  Note, that the result gets wrapped into an ORG MACRO syntax using three curly brackets. This allows
  org to replace the results of the evaluation if the inline call is executed multiple times.
  
  If one uses the =:results raw= option, the results are placed "as is" into the buffer, so multiple
  executions will lead to multiple insertions of the result.

  src_emacs-lisp[:results raw]{(org-table-get-remote-range "table1" "@>$1" )} 15


  Inline source code is only supposed to create one-line results. If you write code that generates
  multiple result lines, an error is raised: /Inline error: multiline result cannot be used/

  src_emacs-lisp[:results value]{(princ "hahha\nyesyesyes" )}

* Defining buffer wide variables for src blocks
  One can use a verbatim block like this. I define a named block =myvar= and
  I pass it into the variable s of the following code block. This lends itself to some nice
  ideas of inserting test in form of templates with some custom variable replacement

  #+NAME: myvar
  : world

  #+BEGIN_SRC emacs-lisp :var s=myvar :exports both
  (concat "hello " s)
  #+END_SRC

  #+RESULTS:
  : hello world
  
* Using a :post function for post-formatting and executing generated tables

  Often I produce multiple tables from a source block (e.g. printing
  several pandas data frames). These tables do not get aligned in the
  org document after the execution of the code block (even though they
  will get aligned upon exporting the document). Also, I may want to have
  table calculations using =#+TBLFM= lines executed, instead of manually
  having to execute them in the resulting tables.

  The following function can be used in a :post argument for getting
  all tables in the output aligned and their TBLFM instructions executed, as shown further below
  #+NAME: srcPostAlignTables
  #+header: :var text="|5|22222|\n|0||\n|12|45|\n|---\n|||\n#+TBLFM:@>$1=vsum(@1..@-1)\n\n|1|22222|\n|0||\n|12|45|\n"
  #+BEGIN_SRC emacs-lisp :results value :exports both
    (with-temp-buffer
      (erase-buffer)
      (cl-assert text nil "PostAlignTables received nil instead of text ")
      (insert text)
      (beginning-of-buffer)
      (org-mode)
      (while
          (search-forward-regexp org-table-any-line-regexp nil t)
        (org-table-align)
        (org-table-recalculate 'iterate)
        (goto-char (org-table-end)))
      (buffer-string))
  #+END_SRC

  #+RESULTS: srcPostAlignTables
  #+begin_example
  |  5 | 22222 |
  |  0 |       |
  | 12 |    45 |
  |----+-------|
  | 17 |       |
  #+TBLFM:@>$1=vsum(@1..@-1)

  |  1 | 22222 |
  |  0 |       |
  | 12 |    45 |
#+end_example


  Example without using the =:post= function:

  #+BEGIN_SRC emacs-lisp :results output drawer :exports both
    (princ
     (concat
      "#+CAPTION: Test1\n"
      "|A|B|C|\n"
      "|---\n"
      "|1|20|300|\n"
      "|200|30|4|\n"
      "|---\n"
      "||||\n"
      "#+TBLFM: @>$1..@>$3=vsum(@I..@II)\n"
      "\n#+CAPTION: Test2\n"
      "|A|B|C|\n"
      "|---\n"
      "|1|20|300|\n"
      "|200|30|4|\n"
      ))
  #+END_SRC

  #+RESULTS:
  :RESULTS:
  #+CAPTION: Test1
  |A|B|C|
  |---
  |1|20|300|
  |200|30|4|
  |---
  ||||
  #+TBLFM: @>$1..@>$3=vsum(@I..@II)

  #+CAPTION: Test2
  |A|B|C|
  |---
  |1|20|300|
  |200|30|4|
  :END:

  The same example with the =:post= function:

  #+BEGIN_SRC emacs-lisp :results output drawer :post srcPostAlignTables(*this*) :exports both
    (princ
     (concat
      "#+CAPTION: Test1\n"
      "|A|B|C|\n"
      "|---\n"
      "|1|20|300|\n"
      "|200|30|4|\n"
      "|---\n"
      "||||\n"
      "#+TBLFM: @>$1..@>$3=vsum(@I..@II)\n"
      "\n#+CAPTION: Test2\n"
      "|A|B|C|\n"
      "|---\n"
      "|1|20|300|\n"
      "|200|30|4|\n"
      ))
  #+END_SRC

  #+RESULTS:
  :RESULTS:
  #+CAPTION: Test1
  |   A |  B |   C |
  |-----+----+-----|
  |   1 | 20 | 300 |
  | 200 | 30 |   4 |
  |-----+----+-----|
  | 201 | 50 | 304 |
  #+TBLFM: @>$1..@>$3=vsum(@I..@II)

  #+CAPTION: Test2
  |   A |  B |   C |
  |-----+----+-----|
  |   1 | 20 | 300 |
  | 200 | 30 |   4 |
  :END:

* Library of babel

  The "Library of Babel" feature provides a kind of primitive function
  library system for org files. It allows running source blocks that
  have been added to it in every org file. The library is implemented
  as an association list with the source block names as the keys. It
  is stored in the =org-babel-library-of-babel= variable.
  
  Execute the following source block to load the source code blocks of the
  test file =lib-of-babel-test.org= into the library of babel.

  #+BEGIN_SRC elisp :exports code'
  (org-babel-lob-ingest "./lib-of-babel-test.org")
  #+END_SRC

  #+RESULTS:
  : 1
  
  For example, the post table alignment function of the last section is a useful generic function. I renamed it in the
  =lib-of-babel-test.org= file to =srcPostAlignTablesLIB= to demonstrate that it indeed is the definition from that file.

  I can call the function like any normally defined named source code block which produces:

  #+CALL: srcPostAlignTablesLIB()

  #+RESULTS:
  #+begin_example
  |  5 | 22222 |
  |  0 |       |
  | 12 |    45 |
  |----+-------|
  | 17 |       |
  ,#+TBLFM:@>$1=vsum(@1..@-1)

  |  1 | 22222 |
  |  0 |       |
  | 12 |    45 |
  #+end_example


  But more interesting for this example, I can also use it in the =:post= block:
  
  #+header: :var text="|A|22222|\n|B||\n|C|45|\n|---\n|||\n#+TBLFM:@>$2=vsum(@1..@-1)\n\n|X|3|\n|Y|4|\n|Z|5|\n|---\n|||\n#+TBLFM:@>$2=vsum(@1..@-1)\n"
  #+BEGIN_SRC elisp :results output raw drawer :post srcPostAlignTablesLIB(*this*) :exports both
  (princ text)
  #+END_SRC

  #+RESULTS:
  :RESULTS:
  | A | 22222 |
  | B |       |
  | C |    45 |
  |---+-------|
  |   | 22267 |
  #+TBLFM:@>$2=vsum(@1..@-1)

  | X |  3 |
  | Y |  4 |
  | Z |  5 |
  |---+----|
  |   | 12 |
  #+TBLFM:@>$2=vsum(@1..@-1)
  :END:

  Note: Originally, I thought I could have the babel library as a
  local variable by executing the =org-babel-lob-ingest= on a file
  local variable in the local variable section of the file (using
  first make-local-variable and the using the ingest). But it turns
  out that during the ingest the buffer associated with the sourced
  file is active, so the local variable in this buffer remains
  unset. This is regrettable, because this means that the library of
  babel is always global. One could set the
  =org-babel-library-of-babel= variable directly to the final value
  instead of using the ingest function, but this would break the
  abstraction.


  
* COMMENT org babel settings

Local variables:
org-confirm-babel-evaluate: nil
End: