cl-hssg/HACKING.rst

.. default-role:: code

###############################
 Hacking on the HSSG internals
###############################

This is interim documentation about the project. Eventually it will become a
proper user manual, for now I am using this document to keep notes before they
become lost forever.


Project overview
################

The idea of HSSG is have a static site generator which acts as a library. Each
website project is its own Common Lisp project that depends on HSSG. It is up
to the website generator how to actually structure and build the website. This
makes it a little harder to set up the website initially, but in return there
are close to no limits as to what kind of website to build. Want a blog? Want
multiple blogs? Multiple languages? Sub-sites? All of that is possible.

Commonly used features like a blog are provided as plugins. These are not
necessarily actual plugins that hook into HSSG (although they can be), they are
generally additional libraries which also depend on HSSG. Two plugins are
provided as part of this project: the blog plugin and the CommonMark reader
plugin. They are not part of core because they have extra dependencies which
not all users of HSSG might need.


Project structure
=================

All source code is in the `src` directory. HSSG and the plugins each have their
own directory there.

Each system has a `package.lisp` file which serves as the initial entry point.
This file defines (almost) all systems and packages, but some files may define
small one-off packages which are not used anywhere else


Core concepts
#############

Artifacts
=========

An artifact is an abstract representation of something that will produce output
when it is written. It can be something like a file that will be copied
verbatim, or a potential HTML page. Artifact can also be of higher order,
meaning they wrap around other artifacts. The compound artifact is a simple
case of just a plain container artifact. On the other hand, the blog is a
higher-order artifact that also has its own data such as the blog title.

Metadata
========

Metadata is an association list of key-value pairs. The key is a keyword symbol
which names the kind of data, while the value is whatever. Metadata gets
transformed by templates.

There are no fixed rules as to what metadata is supposed to represent. Usually
it is information about a web page, but it can also represent a CSS file, or
even something abstract like a product showcase. A product showcase could then
be transformed into metadata representing multiple HTML pages, turning an
abstract domain-specific concept into a more general concept.

Templates
=========

A template is a function which transforms metadata to metadata. There are no
further rules, it is the responsibility of the user to chain templates in a
meaningful way.

There are convenience macros (e.g. `DEFTEMPLATE`) for defining templates and
there are combinator functions which combine templates into other templates
(e.g. `CHAIN-TEMPLATES`).

Readers
=======

We have metadata to represent information, we have templates to transform them,
but where do we get metadata from? In theory metadata can come from anywhere,
but in practice it will usually be read from a file. A reader is a function
which takes a path to a file in a certain forma and returns metadata.

Usually readers will be stored in a key-value variable where the key is the
file format. Another function can then dispatch on the file format to the
correct reader.

File systems and instructions
=============================

Ultimately every artifact will be written to a file. Hard-coding this would
require intermixing the concern of "represents some content" and "will produce
a file". The lack of separation of concerns makes the artifact classes harder
to reason about and harder to test. The act of producing an actual on-disk file
needs to be separate from the individual artifacts. The usual solution would be
to have some sort of "file system service" object which we can inject it as a
dependency into a function, but this is not an elegant solution.

My solution are file systems and instructions. A file system is an object which
abstracts away access to an actual file system. An instruction is an object
which represents an action to perform, such as copying a file or writing a
string to a file.

.. code:: lisp

   (defclass base-file-system (file-system)
     ((directory :initarg :directory :reader file-system-directory :type pathname
        :documentation "Actual directory within the file system."))
     (:documentation
      "A file system which accesses files of the host OS relative to a given base
   directory."))

   (defclass write-string-contents (file-system-instruction)
     ((path :initarg :path :reader instruction-path :type pathname
         :documentation "Path to the output file")
      (contents :initarg :contents :initform (make-string 0) :type string
        :documentation "The file content as a string"))
     (:documentation
      "Instruction which creates a file with the given content."))

We can apply an instruction to a file system, which carries out the action in
the directory of the file system. We use the fact that CLOS supports multiple
dispatch to dispatch on both the file system and the instruction.

.. code:: lisp

   (defmethod write-to-filesystem ((instruction write-string-contents)
                                   (file-system base-file-system))
     "A primitive implementation producing one file for fixed contents and an
     absolute file system."
     (let ((path (fad:merge-pathnames-as-file
                   (fad:pathname-as-directory (file-system-directory file-system))
                   (instruction-path instruction))))
       (with-slots (contents) instruction
         (write-string-to-file contents path))))

But wait, if we have `n` file systems and `m` instructions, does that mean we
need `n * m` implementations? No, most file systems and instructions are
actually of a higher-order and reduce down to the most elemental ones. Consider
the compound instruction, a container instruction which wraps other
instructions:

.. code:: lisp

   (defmethod write-to-filesystem ((instruction compound-instruction)
                                   file-system)
     (with-slots (instructions) instruction
       (dolist (instruction instructions)
         (write-to-filesystem instruction file-system))))

Instructions a produced by deriving an artifact; e.g. to derive an HTML
artifact we generate the file contents as a string and produce a
`WRITE-STRING-CONTENTS` instruction with the content and file name. We do not
care where the file is written to, that part is the responsibility of the file
system.

The most elemental file system simply references an on-disc directory. A
higher-order file system is the overlay file system which adds a path on top of
another file system. We are not bound by on-disc directories though: an FTP
file system could abstract away access to an FTP server, a ZIP file system
might abstract away access to a ZIP file.


The blog plugin
###############

This plugin implements blogging. Each blog is one giant artifact which wraps
around other artifacts. Those other artifacts are private though.


The CommonMark reader plugin
############################

This plugin provides a reader for CommonMark files. Metadata is given as
key-value pairs before a separator line `---`. Everything after the separator
becomes the content of the file.