Computational notebooks allow arguing from evidence by mixing prose with executable code. For a good overview of problems users encounter in traditional notebooks like Jupyter, see I don't like notebooks and What’s Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities.
Specifically Clerk wants to address the following problems:
Clerk is a notebook library for Clojure that aims to address these problems by doing less, namely:
When you're not yet familiar with Clerk, we recommend cloning and playing with the nextjournal/clerk-demo repo.
Then open dev/user.clj
from the project in your favorite editor start a REPL into the project, see
To use Clerk in your project, add the following dependency to your deps.edn
:
Require and start Clerk as part of your system start, e.g. in user.clj
:
You can then access Clerk at http://localhost:7777.
You can load, evaluate, and present a file with the clerk/show! function, but in most cases it's easier to start a file watcher with something like:
... which will automatically reload and re-eval any clj or md files that change, displaying the most recently changed one in your browser.
To make this performant enough to feel good, Clerk caches the computations it performs while evaluating each file. Likewise, to make sure it doesn't send too much data to the browser at once, Clerk paginates data structures within an interactive viewer.
A recommended alternative to the file watcher is setting up a hotkey in your editor to save & clerk/show!
the active file.
Emacs
In Emacs, add the following to your config:
IntelliJ/Cursive
In IntelliJ/Cursive, you can set up REPL commands via:
Tools→REPL→Add New REPL Command
, then(show! "~file-path")
;nextjournal.clerk
namespace;Settings→Keymap
Neovim + Conjure
With neovim + conjure one can use the following vimscript function to save the file and show it with Clerk:
Clerk comes with a number of useful built-in viewers e.g. for Clojure data, html & hiccup, tables, plots &c.
When showing large data structures, Clerk's default viewers will paginate the results.
The default set of viewers are able to render Clojure data.
Viewers can handle lazy infinite sequences, partially loading data by default with the ability to load more data on request.
In addition, there's a number of built-in viewers that can be called explicity using functions.
The html
viewer interprets hiccup
when passed a vector.
Alternatively you can pass it an HTML string.
You can style elements, using Tailwind CSS.
The html
viewer is also able to display SVG, taking either a hiccup vector or a SVG string.
Clerk provides a built-in data table viewer that supports the three most common tabular data shapes out of the box: a sequence of maps, where each map's keys are column names; a seq of seq, which is just a grid of values with an optional header; a map of seqs, in with keys are column names and rows are the values for that column.
1 | 2 |
3 | 4 |
odd numbers | even numbers |
---|---|
1 | 2 |
3 | 4 |
odd numbers | even numbers |
---|---|
1 | 2 |
3 | 4 |
odd numbers | even numbers |
---|---|
1 | 2 |
3 | 4 |
Internally the table viewer will normalize all of the above to a map with :rows
and an optional :head
key, also giving you control over the column order.
odd numbers | even numbers |
---|---|
1 | 2 |
3 | 4 |
As we've already seen, all comment blocks can contain TeX (we use KaTeX under the covers). In addition, you can call the TeX viewer programmatically. Here, for example, are Maxwell's equations in differential form:
Clerk also has built-in support for Plotly's low-ceremony plotting. See Plotly's JavaScript docs for more examples and options.
But Clerk also has Vega Lite for those who prefer that grammar.
You can provide a map of embed options to the vega viewer via the :embed/opts
key.
The code viewer uses clojure-mode for syntax highlighting.
Clerk now has built-in support for the java.awt.image.BufferedImage
class, which is the native image format of the JVM.
When combined with javax.imageio.ImageIO/read
, one can easily load images in a variety of formats from a java.io.File
, an java.io.InputStream
, or any resource that a java.net.URL
can address.
For example, we can fetch a photo of De zaaier, Vincent van Gogh's famous painting of a farmer sowing a field from Wiki Commons like this:
We've put some effort into making the default image rendering pleasing. The viewer uses the dimensions and aspect ratio of each image to guess the best way to display it in classic DWIM fashion. For example, an image larger than 900px wide with an aspect ratio larger then two will be displayed full width:
On the other hand, smaller images are centered and shown using their intrinsic dimensions:
The same Markdown support Clerk uses for comment blocks is also available programmatically:
Layouts can be composed via row
s and col
s
Passing :width
, :height
or any other style attributes to ::clerk/opts
will assign them on the row or col that contains your items. You can use this to size your containers accordingly.
Laying out stuff is not limited to images. You can use it to lay out any Clerk viewer. E.g. combine it with HTML viewers to render nice captions:
Alternative notations
By default, row
and col
operate on & rest
so you can pass any number of items to the functions. But the viewers are smart enough to accept any sequential list of items.
Viewers compose, so you can for example use the plotly viewer inside the grid viewers.
Metadata Notation
In the examples above, we've used convience helper functions like clerk/html
or clerk/plotly
to wrap values in a viewer. If you call this on the REPL, you'll notice a given value gets wrapped in a map under the :nextjournal/value
key with the viewer being in the :nextjournal/viewer
key.
You can also select a viewer using Clojure metadata in order to avoid Clerk interfering with the value.
:temperature | :date |
---|---|
41 | 2022-08-01" ] |
39 | 2022-08-01" ] |
34 | 2022-08-01" ] |
29 | 2022-08-01" ] |
Let's explore how Clerk viewers work and how you create your own to gain better insight into your problem at hand.
These are the default viewers that come with Clerk.
Each viewer is a simple Clojure map.
We have a total of 41 viewers in the defaults. Let's start with a simple example and explain the different extensions points in the viewer api.
Clerk's rendering happens in the browser. On the Clojure-side, a given document is presented. Presenting takes a value and transforms it such that Clerk can send it to the browser where it will be rendered.
Let's start with one of the simplest examples. You can see that present
takes our value 1
and transforms it into a map, with 1
under a :nextjournal/value
key and the number viewer assigned under the :nextjournal/viewer
key. We call this map a wrapped-value
.
This data structure is is sent over Clerk's websocket to the browser, where it will be displayed using the :render-fn
found in the :nextjournal/viewer
key.
Now onto something slightly more complex, #{1 2 3}
.
Here, we're giving it a set with 1, 2, 3 in it. In its generalized form, present
is a function that does a depth-first traversal of a given tree, starting at the root node. It will select a viewer for this root node, and unless told otherwise, descend further down the tree to present its child nodes.
Compare this with the simple 1
example above! You should recognize the leaf values. Also note that the container is no longer a set, but it has been transformed into a vector. This transformation exists to support pagination of long unordered sequences like maps and sets and so we can efficiently access a value inside this tree using get-in
.
You might ask yourself why we don't just send the unmodified value to the browser. For one, we could easily overload the browser with too much data. Secondly we will look at examples of being able to select viewers based on Clojure and Java types, which cannot be serialized and sent to the browser.
When writing your own viewer, the first extention point you should reach for is :tranform-fn
.
{:nextjournal/value "Exploring the viewer api", :nextjournal/viewers [{:pred #object[sicmutils.expression$literal_QMARK_ 0x510ba0cf "sicmutils.expression$literal_QMARK_@510ba0cf"], :transform-fn #object[clojure.core$comp$fn__5825 0x1911d197 "clojure.core$comp$fn__5825@1911d197"], :render-fn (fn [label->val] (v/html (reagent/with-let [!selected-label (reagent/atom (ffirst label->val))] [:<> (into [:div.flex.items-center.font-sans.text-xs.mb-3 [:span.text-slate-500.mr-2 "View-as:"]] (map (fn [label] [:button.px-3.py-1.font-medium.hover:bg-indigo-50.rounded-full.hover:text-indigo-600.transition {:class (if (= (clojure.core/deref !selected-label) label) "bg-indigo-100 text-indigo-600" "text-slate-500"), :on-click (fn* [] (reset! !selected-label label))} label])) (keys label->val)) [v/inspect-presented (get label->val (clojure.core/deref !selected-label))]])))} {:pred #object[clojure.core$char_QMARK___5425 0x4106651f "clojure.core$char_QMARK___5425@4106651f"], :render-fn (fn [c] (v/html [:span.cmt-string.inspected-value "\\" c]))} {:pred #object[clojure.core$string_QMARK___5427 0x3113a37 "clojure.core$string_QMARK___5427@3113a37"], :render-fn v/quoted-string-viewer, :page-size 80} {:pred #object[clojure.core$number_QMARK_ 0x5002fc11 "clojure.core$number_QMARK_@5002fc11"], :render-fn v/number-viewer} {:name :number-hex, :render-fn (fn [num] (v/number-viewer (str "0x" (.toString (js/Number. num) 16))))} {:pred #object[clojure.core$symbol_QMARK_ 0x1e11bc55 "clojure.core$symbol_QMARK_@1e11bc55"], :render-fn (fn [x] (v/html [:span.cmt-keyword.inspected-value (str x)]))} {:pred #object[clojure.core$keyword_QMARK_ 0x3c403b36 "clojure.core$keyword_QMARK_@3c403b36"], :render-fn (fn [x] (v/html [:span.cmt-atom.inspected-value (str x)]))} {:pred #object[clojure.core$nil_QMARK_ 0x9acee30 "clojure.core$nil_QMARK_@9acee30"], :render-fn (fn [_] (v/html [:span.cmt-default.inspected-value "nil"]))} {:pred #object[clojure.core$boolean_QMARK_ 0x133e019b "clojure.core$boolean_QMARK_@133e019b"], :render-fn (fn [x] (v/html [:span.cmt-bool.inspected-value (str x)]))} {:pred #object[clojure.core$map_entry_QMARK_ 0x7d28297b "clojure.core$map_entry_QMARK_@7d28297b"], :name :map-entry, :render-fn (fn [xs opts] (v/html (into [:<>] (comp (v/inspect-children opts) (interpose " ")) xs))), :page-size 2} {:pred #object[nextjournal.clerk.viewer$get_safe$fn__12031 0x70bd0c3 "nextjournal.clerk.viewer$get_safe$fn__12031@70bd0c3"], :transform-fn #object[nextjournal.clerk.viewer$update_val$fn__12177 0x2fa7dc36 "nextjournal.clerk.viewer$update_val$fn__12177@2fa7dc36"]} {:name :read+inspect, :render-fn (fn [x] (try (v/html [v/inspect (v/read-string x)]) (catch js/Error _e (v/unreadable-edn-viewer x))))} {:pred #object[clojure.core$vector_QMARK___5431 0x7cd91400 "clojure.core$vector_QMARK___5431@7cd91400"], :render-fn v/coll-viewer, :opening-paren "[", :closing-paren "]", :page-size 20} {:pred #object[clojure.core$set_QMARK_ 0x2b7e2212 "clojure.core$set_QMARK_@2b7e2212"], :render-fn v/coll-viewer, :opening-paren "#{", :closing-paren "}", :page-size 20} {:pred #object[clojure.core$sequential_QMARK_ 0x54398386 "clojure.core$sequential_QMARK_@54398386"], :render-fn v/coll-viewer, :opening-paren "(", :closing-paren ")", :page-size 20} {:pred #object[clojure.core$map_QMARK___5429 0xb0a1535 "clojure.core$map_QMARK___5429@b0a1535"], :name :map, :render-fn v/map-viewer, :opening-paren "{", :closing-paren "}", :page-size 10} {:pred #object[clojure.core$some_fn$sp1__8670 0x79d6b7d6 "clojure.core$some_fn$sp1__8670@79d6b7d6"], :transform-fn #object[clojure.core$comp$fn__5825 0xccdbaa6 "clojure.core$comp$fn__5825@ccdbaa6"], :render-fn (fn [x] (v/html [:span.inspected-value [:span.cmt-meta "#'" (str x)]]))} {:pred #object[nextjournal.clerk.viewer$fn__12219 0x45892a66 "nextjournal.clerk.viewer$fn__12219@45892a66"], :name :error, :render-fn v/throwable-viewer, :transform-fn #object[clojure.core$comp$fn__5825 0x55f1d5ff "clojure.core$comp$fn__5825@55f1d5ff"]} {:pred #object[nextjournal.clerk.viewer$fn__12222 0x1cdd67dc "nextjournal.clerk.viewer$fn__12222@1cdd67dc"], :transform-fn #object[nextjournal.clerk.viewer$fn__12225 0x621c72b7 "nextjournal.clerk.viewer$fn__12225@621c72b7"], :render-fn (fn [blob] (v/html [:figure.flex.flex-col.items-center.not-prose [:img {:src (v/url-for blob)}]]))} {:pred #object[nextjournal.clerk.viewer$fn__12230 0x47a56acf "nextjournal.clerk.viewer$fn__12230@47a56acf"], :transform-fn #object[nextjournal.clerk.viewer$update_val$fn__12177 0x75168a0a "nextjournal.clerk.viewer$update_val$fn__12177@75168a0a"]} {:pred #object[clojure.core$partial$fn__5857 0xb010dea "clojure.core$partial$fn__5857@b010dea"], :transform-fn #object[nextjournal.clerk.viewer$fn__12235 0x146f0684 "nextjournal.clerk.viewer$fn__12235@146f0684"]} {:pred #object[clojure.core$constantly$fn__5689 0x7d7dcf6 "clojure.core$constantly$fn__5689@7d7dcf6"], :transform-fn #object[nextjournal.clerk.viewer$update_val$fn__12177 0x4f6a67db "nextjournal.clerk.viewer$update_val$fn__12177@4f6a67db"]} {:name :elision, :render-fn v/elision-viewer, :transform-fn #object[nextjournal.clerk.viewer$mark_presented 0x4e623d4b "nextjournal.clerk.viewer$mark_presented@4e623d4b"]} {:name :latex, :render-fn v/katex-viewer, :transform-fn #object[nextjournal.clerk.viewer$mark_presented 0x4e623d4b "nextjournal.clerk.viewer$mark_presented@4e623d4b"]} {:name :mathjax, :render-fn v/mathjax-viewer, :transform-fn #object[nextjournal.clerk.viewer$mark_presented 0x4e623d4b "nextjournal.clerk.viewer$mark_presented@4e623d4b"]} {:name :html, :render-fn v/html, :transform-fn #object[clojure.core$comp$fn__5825 0xf65a857 "clojure.core$comp$fn__5825@f65a857"]} {:name :plotly, :render-fn v/plotly-viewer, :transform-fn #object[nextjournal.clerk.viewer$mark_presented 0x4e623d4b "nextjournal.clerk.viewer$mark_presented@4e623d4b"]} {:name :vega-lite, :render-fn v/vega-lite-viewer, :transform-fn #object[nextjournal.clerk.viewer$mark_presented 0x4e623d4b "nextjournal.clerk.viewer$mark_presented@4e623d4b"]} {:name :markdown, :transform-fn #object[nextjournal.clerk.viewer$fn__12241 0x63d2d419 "nextjournal.clerk.viewer$fn__12241@63d2d419"]} {:name :code, :render-fn v/code-viewer, :transform-fn #object[clojure.core$comp$fn__5825 0x7f1a2307 "clojure.core$comp$fn__5825@7f1a2307"]} {:name :code-folded, :render-fn v/foldable-code-viewer, :transform-fn #object[clojure.core$comp$fn__5825 0x20cdc1ca "clojure.core$comp$fn__5825@20cdc1ca"]} {:name :reagent, :render-fn v/reagent-viewer, :transform-fn #object[nextjournal.clerk.viewer$mark_presented 0x4e623d4b "nextjournal.clerk.viewer$mark_presented@4e623d4b"]} {:name :row, :render-fn (fn [items opts] (let [item-count (count items)] (v/html (into [:div {:class "md:flex md:flex-row md:gap-4 not-prose", :style opts}] (map (fn [item] [:div.flex.items-center.justify-center.flex-auto (v/inspect-presented opts item)])) items))))} {:name :col, :render-fn (fn [items opts] (v/html (into [:div {:class "md:flex md:flex-col md:gap-4 clerk-grid not-prose", :style opts}] (map (fn [item] [:div.flex.items-center.justify-center (v/inspect-presented opts item)])) items)))} {:name :table, :transform-fn #object[nextjournal.clerk.viewer$fn__12256 0x2c8f294 "nextjournal.clerk.viewer$fn__12256@2c8f294"]} {:name :table-error, :render-fn v/table-error, :page-size 1} {:name :clerk/code-block, :transform-fn #object[nextjournal.clerk.viewer$fn__12268 0x77cf9cc8 "nextjournal.clerk.viewer$fn__12268@77cf9cc8"]} {:name :clerk/result-block, :transform-fn #object[clojure.core$comp$fn__5825 0x59638b2d "clojure.core$comp$fn__5825@59638b2d"]} {:name :tagged-value, :render-fn (fn [{:keys [tag value space?]} opts] (v/html (v/tagged-value {:space? (:nextjournal/value space?)} (str "#" (:nextjournal/value tag)) [v/inspect-presented value]))), :transform-fn #object[nextjournal.clerk.viewer$mark_preserve_keys 0x17eec25a "nextjournal.clerk.viewer$mark_preserve_keys@17eec25a"]} {:name :clerk/result, :render-fn v/result-viewer, :transform-fn #object[nextjournal.clerk.viewer$mark_presented 0x4e623d4b "nextjournal.clerk.viewer$mark_presented@4e623d4b"]} {:name :clerk/notebook, :render-fn v/notebook-viewer, :transform-fn #object[nextjournal.clerk.viewer$fn__12276 0x52f7de61 "nextjournal.clerk.viewer$fn__12276@52f7de61"]} {:name :hide-result, :transform-fn #object[nextjournal.clerk.viewer$fn__12139 0x59d5312b "nextjournal.clerk.viewer$fn__12139@59d5312b"]}], :!budget #object[clojure.lang.Atom 0x6173f46a {:status :ready, :val 200}], :path [], :current-path []}
As you can see the argument to the :transform-fn
isn't just the string we're passing it, but a wrapped-value
. We will look at what this enables in a bit. But let's look at one of the simplest examples first.
A first simple example
For this simple greet-viewer
we're only doing a simple value transformation. For this, clerk/update-val
is a small helper function which takes a function f
and returns a function to update only the value inside a wrapped-value
, a shorthand for #(update % :nextjournal/val f)
The :transform-fn
runs on the JVM, which means you can explore what it does at your REPL by calling v/present
on such a value.
Passing modified viewers down the tree
Column: A | Column: B | Column: C |
---|---|---|
1 | 1 | 1 |
2 | 2 | 2 |
3 | 3 | 3 |
4 | N/A | N/A |
Column: A | Column: B | Column: C |
---|---|---|
1 | 1 | 1 |
2 | 2 | 2 |
3 | 3 | 3 |
4 | N/A | N/A |
But this presentation and hence tranformation of nodes further down the tree isn't always what you want. For example, the plotly
or vl
viewers want to receive the child value unaltered in order to use it as a spec.
To stop Clerk's presentation from descending into child nodes, use clerk/mark-presented
as a :transform-fn
. Compare the result below in which [1 2 3]
appears unaltered with what you see above.
Clerk's presentation will also transform maps into sequences in order to paginate large maps. When you're dealing with a map that you know is bounded and would like to preserve its keys, there's clerk/mark-preserve-keys
. This will still transform (and paginate) the values of the map, but leave the keys unaltered.
As we've just seen, you can also do a lot with :transform-fn
and using clerk/html
on the JVM. When you want to run code in the browser where Clerk's viewers are rendered, reach for :render-fn
. As an example, we'll write a multiviewer for a sicmutils literal expression that will compute two alternative representations and let the user switch between them in the browser.
We start with a simple function that takes a such an expression and turns it into a map with two representation, one TeX and the original form.
Our literal-viewer
calls this transform-literal
function and also calls clerk/mark-preserve-keys
. This tells Clerk to leave the keys of the map as-is.
In our :render-fn
, which is called in the browser we will recieve this map. Note that this is a quoted form, not a function. Clerk will send this form to the browser for evaluation. There it will create a reagent/atom
that holds the selection state. Lastly, v/inspect-presented
is a component that takes a wrapped-value
that ran through v/present
and show it.
Now let's see if this works. Try switching to the original representation!
Without a viewer specified, Clerk will go through the sequence of viewers and apply the :pred
function in the viewer to find a matching one. Use v/viewer-for
to select a viewer for a given value.
If we select a specific viewer (here the v/html-viewer
using clerk/html
) this is the viewer we will get.
Instead of specifying a viewer for every value, we can also modify the viewers per namespace. Here, we add the literal-viewer
from above to the whole namespace.
As you can see we now get this viewer automatically, without needing to explicitly select it.
Notice that for the string?
viewer above, there's a :page-size
of 80
. This is the case for all collection viewers in Clerk and controls how many elements are displayed. So using the default string?-viewer
above, we're showing the first 80 characters.
If we change the viewer and set a different :n
in :page-size
, we only see 10 characters.
Or, we can turn off eliding, by dissoc'ing :page-size
alltogether.
The operations above were changes to a single viewer. But we also have a function update-viewers
to update a given viewers by applying a select-fn->update-fn
map. Here, the predicate is the keyword :page-size
and our update function is called for every viewer with :page-size
and is dissoc'ing them.
Here's the updated-viewers:
Now let's confirm these modified viewers don't have :page-size
on them anymore.
And compare it with the defaults:
Now let's display our clojure-data
var from above using these modified viewers.
This is a custom viewer for Mermaid, a markdown-like syntax for creating diagrams from text. Note that this library isn't bundles with Clerk but we use a component based on d3-require to load it at runtime.
We can then use the above viewer using with-viewer
.
Visibility for code and results can be controlled document-wide and per top-level form. By default, Clerk will always show code and results for a notebook.
You can use a map of the following shape to set the visibility of code and results individually:
The above example will hide the code and show only results.
Valid values are :show
, :hide
and :fold
(only available for :code
). Using {:code :fold}
will hide the code cell initially but show an indicator to toggle its visibility:
The visibility map can be used in the following ways:
Set document defaults via the ns
form
The above example will hide all code cells by default but show an indicator to toggle their visibility instead. In this case results will always show because that’s the default.
As metadata to control a single top-level form
This will hide the code but only show the result:
Setting visibility as metadata will override the document-wide visibility settings for this one specific form.
As top-level form to change the document defaults
Independently of what defaults are set via your ns
form, you can use a top-level map as a marker to override the visibility settings for any forms following it.
Example: Code is hidden by default but you want to show code for all top-level forms after a certain point:
This comes in quite handy for debugging too!
First, we parse a given Clojure file using rewrite-clj
.
Then, each expression is analysed using tools.analyzer
. A dependency graph, the analyzed form and the originating file is recorded.
This analysis is done recursively, descending into all dependency symbols.
Then we can use this information to hash each expression.
Clerk uses the hashes as filenames and only re-evaluates forms that haven't been seen before. The cache is using nippy.
We can look up the cache key using the var name in the hashes map.
As an escape hatch, you can tag a form or var with ::clerk/no-cache
to always re-evaluate it. The following form will never be cached.
For side effectful functions that should be cached, like a database query, you can add a value like this #inst
to control when evaluation should happen.
:tracks/AlbumId | :tracks/Bytes | :tracks/Name | :tracks/TrackId | :tracks/UnitPrice |
---|---|---|---|---|
1 | 11170334 | For Those About To Rock (We Salute You) | 1 | 0.99 |
2 | 5510424 | Balls to the Wall | 2 | 0.99 |
3 | 3990994 | Fast As a Shark | 3 | 0.99 |
3 | 4331779 | Restless and Wild | 4 | 0.99 |
3 | 6290521 | Princess of the Dawn | 5 | 0.99 |
1 | 6713451 | Put The Finger On You | 6 | 0.99 |
1 | 7636561 | Let's Get It Up | 7 | 0.99 |
1 | 6852860 | Inject The Venom | 8 | 0.99 |
1 | 6599424 | Snowballed | 9 | 0.99 |
1 | 8611245 | Evil Walks | 10 | 0.99 |
1 | 6566314 | C.O.D. | 11 | 0.99 |
1 | 8596840 | Breaking The Rules | 12 | 0.99 |
1 | 6706347 | Night Of The Long Knives | 13 | 0.99 |
1 | 8817038 | Spellbound | 14 | 0.99 |
4 | 10847611 | Go Down | 15 | 0.99 |
4 | 7032162 | Dog Eat Dog | 16 | 0.99 |
4 | 12021261 | Let There Be Rock | 17 | 0.99 |
4 | 8776140 | Bad Boy Boogie | 18 | 0.99 |
4 | 10617116 | Problem Child | 19 | 0.99 |
4 | 12066294 | Overdose | 20 | 0.99 |
3483 more elided |