Because I can, I recently added support for testnet Bitcoin Lightning payments into this website. The beauty of Freedom and privacy respecting software is that you can do anything you want with it. There are many ways to do it. You can always use a service, provider for that, but the beauty of Bitcoin, the reason it is valuable, is that you can do it yourself.

Yes, there are common tools, the most established would be to use BTCPay. For my goals, it was an overkill, I didn’t want their entire infrastructure requirements, I didn’t want to work on C# and I really wanted to try Common Lisp for this project.

This project was more a pain than a delight. That is the nature of learning, you need inefficiencies and time waste to learn. I at least learn by mistakes, mistakes force me to pay attention. When things work, it does feel great, but I don’t learn anything new. I want to share this non trivial example, I always find the web full of toy examples that leave me wanting. That is why, when I find a project with a closed and manageable scope that was a pain for me to solve, I write about them , for you to learn from my mistakes.

Why was it a pain?

It is not to long ago that I learned Lisp, and I really love it. I wish I had learned about it earlier in my life. It is so flexible and expressive and simple. Its interactive developing process has no competition and today it is the language I use to try new ideas. Unfortunately that doesn’t mean it is the language I can use to do anything I want.

The language isn’t everything, there is also tooling, libraries, documentation and if I ever manage to become more social to take advantage of it: the community. I really like Common Lisp, the language, I feel the tools I managed to use are great. Libraries seem of high quality, yet they are really hard to find and there isn’t one for everything. Documentation is extremely hard to find. My well developed work mode of search based development is useless. There isn’t enough reachable content on the Internet to figure out how to do the things I want to do.

Yet, that wasn’t the main pain of this project. My pain was the front-end, how to deal with the user interface. Thus, it wasn’t Common Lisp’s fault, it was the lack of a good tool for that part of the job. It wasn’t any better than doing it in JavaScript, which nobody does this days. For a user interface, you need some kind of framework, and I couldn’t really find something for it. This is the unfair comparison, but ClojureScript+Reagent+Reframe are the only combination I have tried, that makes working on a front-end a manageable experience. It is not the language, it is the tools & libraries that really make the difference and I couldn’t find a similar experience in Common Lisp land for this project.

If you can’t find a good tool for the small job, don’t go big! Look how to do the minimum amount of work to get it done, without committing to a technology.

Front-end and Back-end total around 250 lines of code. The Back-end was a delight to write in Common Lisp, it beats any Python experience I have. There are good enough libraries to address my needs.

The Front-end a pain. I used Parenscript, which is a Lisp flavored JavaScript. That means, you are most of the time still in the JavaScript land of pain, and without a library to manage the User Interface any endeavor you take feels hopeless.

What does my solution look like?

The backend is a set of services, the frontend is a form and some JavaScript for the interactive parts. It is amazing that the Bitcoin & lightning networks require so little hardware to function and yet they let you to partake in a global trade network. Thanks to their design decisions around management of local state, keeping global consensus to the minimum, it is very possible for every individual to run your infrastructure and verify.

The next figure shows how little you need for this project. I use Bitcoin Core to connect to the Bitcoin network and participate on the global consensus. On top of that the LND daemon manages my wallet and manages my lightning channels, it too connects to the lightning network. LND exposes a REST API which is what my payment backend uses to request lightning invoices and listens to until the payment settles. That means the Payment Backend acts as a middle-ware exposing those same 2 API endpoints to the frontend user interface.

I’ll explain the system following the Lisp interactive development way. That means, building little by little each peace, refactor it then finally package everything. Simply summarizing the final code misses the advantage of doing it in Lisp.

I’ll jump between frontend and backend development. Interactive development for me starts with wishful thinking. Visualizing what you want and starting there, then continue to make it work.

Front-end: The user needs a form to request an invoice

The way payments in the Lightning Network work is by offering the user an invoice. That invoice tells the user’s lightning client where to pay and how much. I want to make it possible for the readers of this blog to endorse/reward my work on this blog by sending me a lightning payment for an amount they choose and with a message for which post earned their appreciation.

Thus take your favorite path for web development and craft a form with 2 fields, one for text, one numeric and a submit button. I personally develop directly the html and use on Emacs the impatient-mode to refresh the webpage as I type my changes in. I do this in isolation for speed when developing individual components. My general preference is to open an .html file load it with tachyons and fontawesome from their CDN and live code the html.

1<link rel="stylesheet" href="https://unpkg.com/[email protected]/css/tachyons.min.css"/>
2<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.3.0/css/all.min.css" />

My resulting form, which you can see at the end of every post on this blog, is this simple html code.

 1<div id="app" class="pa4 black-80">
 2  <div class="mw7 center br3 ba b--black-20 pa4" id="ln-payment">
 3    <fieldset class="ph0 mh0 bn">
 4      <legend class="f4 fw6 ph0 mh0">
 5        Tip me with <i class="fas fa-flask green"></i>Testnet
 6        <i class="fas fa-bolt gold"></i>Lightning
 7        <i class="fab fa-bitcoin green"></i>Bitcoin
 8      </legend>
 9      <div class="mt3">
10        <label class="db fw6 lh-copy f6">Message:</label>
11        <input
12          class="input pa2 input-reset ba bg-transparent w-100"
13          maxlength="64"
14          id="memo"
15          type="text"
16          value="Tip for this article"
17        />
18      </div>
19      <div class="mt3">
20        <label class="db fw6 lh-copy f6"><small>test</small> Sats:</label>
21        <input
22          class="pa2 input-reset ba bg-transparent w-100"
23          id="sat-value"
24          type="number"
25          value="2345"
26          min="2121"
27          max="400000"
28        />
29      </div>
30    </fieldset>
31    <input
32      class="b mt3 ph3 pv2 input-reset ba b--black bg-transparent grow pointer f6 dib"
33      id="get-invoice"
34      onclick="javascript:invoice()"
35      type="submit"
36      value="Get lightning invoice"
37    />
38  </div>
39</div>

This is great, because I can then use this same code as a partial template in hugo which powers this blog. I did suffer to get the styling right, because CSS is a pain, no matter how much tachyons has helped me to be declarative in styling, I have not yet a trained mind for visual design. But secondly, because this blog uses different CSS stylesheets and I had to deal with the conflicts.

Front-end: Reacting to user input, the JavaScript problem

The previous HTML code is the form, and the first sign for interactivity for the user is the onclick = "javascript:invoice()" on the submit button. That is where the nightmare starts. No matter how much I appreciate simple tools, few dependencies, managing the user interaction with the bare language is horrible.

The challenge was to do it in Common Lisp. Parenscript is the Common Lisp answer for the JavaScript problem. It gives you a Lisp flavored JavaScript. I do find it better than JavaScript, but not better enough, you only get to program in a bit more expressive JavaScript. Secondly, the language is not enough for managing a user interface, using the bare language is akin to a bare hands fight in warfare. It is not possible to win. Yet, because this is a small self-hosted project I stuck to it. I didn’t find the React wrapper documented enough to venture that alternative, neither were other options interesting enough. Documentation, examples, tutorials barely exist in this space. With this experience I know to stick to ClojureScript+Reagent+Reframe for any relevant project in my future.

Rant over, what is my solution? Open a .lisp file and start hacking the front end client. I use Emacs as my editor, I use doomemacs as main configuration manager for it. Doomemacs uses sly to work with Common Lisp code. As soon as you open a .lisp file, it fires a Lisp repl for you to develop. I use SBCL as my Common Lisp implementation, and I have Quicklisp for package management. Quicklisp is wonderful as I can load new libraries while I develop. There is no need to restart lisp and thus loose my program state at any time I need a new library. That means I can discover, and try which dependencies I need to get my system to work. For the interactive web development beyond design I drop impatient-mode and move to skewer, and in this particular case with Parenscript I use trident-mode to push my JavaScript changes into the browser, without the need to manually refresh it. Because of doomemacs’ use of sly I did have to patch trident-mode to use it, thus be warned. That is the price and reward of using Free Software, you get the code to change it to your needs and load it into your machine, but you have to do it yourself.

This is the starting code, I include the Emacs code comments to emphasize, that tools are necessary for development, it is not simply code. The editor is your assistant. In this case I declare this file using the lisp-mode which helps me to interactively use the lisp repl and evaluate code. I also have a save hook that evaluates the code in this file using trident-mode, that means it translates this lisp code into JavaScript and loads it into the browser. Instantaneous code reload, no need to refresh the page, no pain of loosing state due to refresh.

 1;; -*- mode: lisp; -*-
 2;; evaluate this once to load parenscript
 3;; (ql:quickload :parenscript)
 4;; Evaluate this line once on the repl workaround so that output from
 5;; trident doesn't get truncated.
 6;; (pushnew '(SLYNK:*STRING-ELISION-LENGTH*) slynk:*slynk-pprint-bindings*
 7;;          :test #'equal)
 8(defpackage ln-pay-js
 9  (:use :cl :parenscript))
10(in-package :ln-pay-js)
11
12;; Local Variables:
13;; eval: (add-hook 'after-save-hook #'trident-eval-buffer :append :local)
14;; End:

Now let’s write effectful code. collect-invoice-data gets the value inside the memo and value fields in the form and packs them into an object. You immediately notice that no client side validation has taken place. A very bad practice, but I have to cut corners to get this done. I will directly send the strings of those fields to the backend, as you can recognize by the use of this function inside invoice, which the user calls from the form.

 1(in-package :ln-pay-js)
 2(defvar *__PS_MV_REG*) ;; Work around parenscript multiple value return
 3(defvar +ln-api-uri+ "")
 4(defun collect-invoice-data ()
 5  (create :value (chain document (get-element-by-id "sat-value") value)
 6          :memo (chain document (get-element-by-id "memo") value)))
 7
 8(defun invoice ()
 9  (chain
10   (fetch (stringify +ln-api-uri+ "/make-invoice")
11          (create :method "POST"
12                  :headers (create "Content-Type" "application/json")
13                  :body (chain -J-S-O-N (stringify (collect-invoice-data)))))
14   (then (lambda (resp)
15           (if (eq 200 (@ resp status))
16                 (chain resp (json))
17                 (throw (@ resp status-text)))))))

As you see, this is JavaScript with Lisp syntax. I’m forced to use JavaScript functions and put the parenthesis elsewhere. There is an advantage of structural editing, which becomes ever less relevant with the use of tree-sitter. Yet, even then I find lisp syntax quite useful. The stand alone function stringify concatenates strings. +ln-api-uri+ is a variable containing the backend server address. At the moment is the same development server and thus the empty string, but that will change in production. create is to declare a JavaScript object specified by key value paired arguments. -J-S-O-N is an abomination of syntax, which translates to JSON in JavaScript, and here stringify has actually a different meaning as it is the method of JSON. I’ll avoid this pointless explanation of the code and let you see to what JavaScript it translates to. It is an advantage of ParenScript that it produces readable JavaScript, which is itself its limitation, as it can then only be a lisp flavored JavaScript.

 1var __PS_MV_REG;
 2if ("undefined" === typeof LNAPIURI) {
 3  var LNAPIURI = "";
 4}
 5function collectInvoiceData() {
 6    return { 'value' : document.getElementById('sat-value').value,
 7             'memo' : document.getElementById('memo').value };
 8};
 9function invoice() {
10  __PS_MV_REG = [];
11  return fetch([LNAPIURI, "/make-invoice"].join(""), {
12    method: "POST",
13    headers: { "Content-Type": "application/json" },
14    body: JSON.stringify(collectInvoiceData()),
15  }).then(function (resp) {
16    if (200 === resp.status) {
17      return resp.json();
18    } else {
19      throw resp.statusText;
20    }
21  });
22}

__PS_MV_REG shows up to track multiple value returns. I don’t use that beautiful lisp capability, that is why it remains empty, but I need to define that variable for the JavaScript code not to crash. ParenScript doesn’t help you with that.

Backend: Processing the invoice request

First load the dependencies into the lisp image. hunchentoot is the webserver, data-format-validation helps to validate input, dexador is an http client to query LND’s REST API, cl-json to work with JSON and finally arrows which provides some syntactic sugar to “compose functions”.

Then define a package ln-pay and for the first development instances hard code everything in variables. The reason I do this during development is because using values directly is easy. If I need to load secrets, I can commit them to code, because they are useless development secrets. The moment I need to move to production, I voluntarily go over the pain of making each variable configurable and consciously only create and properly account for the secrets I do need. In this program I only need the authentication macaroon in a header and where the LND REST endpoint is. Finally, I start the HTTP server, which I can check works on http://localhost:4646 . The :document-root option is a relative path to where the .html files reside, start serving an index.html file containing the earlier from .

 1(ql:quickload '(hunchentoot data-format-validation dexador cl-json arrows))
 2
 3(defpackage ln-pay
 4  (:use :cl :hunchentoot))
 5
 6(in-package :ln-pay)
 7
 8(defun encode-hex (bytes)
 9  (with-output-to-string (o)
10    (loop for byte across bytes
11          do (format o "~2,'0x" byte))))
12
13(defvar *auth-header*
14  (with-open-file (input "/path/to/lnd.macaroon"
15                         :element-type '(unsigned-byte 8))
16    (let ((v (make-array 1024 :element-type '(unsigned-byte 8))))
17      (read-sequence v input)
18      (cons "Grpc-Metadata-macaroon" (encode-hex v)))))
19
20(defvar *lnd-api-endpoint* "https://localhost:8481/v1")
21
22(defvar *server* (start (make-instance 'easy-acceptor :port 4646
23                                       :document-root #p"www/")))

Now to the actual code to process the invoice request. The threading last macro (->>) from arrows is useful for readability, each output of a function is the input for the next as its last argument. The sequence of steps is easily readable, it declares: Get the POST data, validate it is a non-nil string, decode that string assuming it is JSON, then validate that JSON data structure, because that is what we want to use to create the invoice. The valid data goes into the request to LND and its result is finally returned.

1(define-easy-handler (pay-req :uri "/make-invoice") ()
2  (setf (content-type*) "application/json")
3  (multiple-value-bind (response)
4      (arrows:->> (raw-post-data :force-text t)
5                  (dfv:parse-input 'string)
6                  (cl-json:decode-json-from-string)
7                  (validate-invoice-request)
8                  (request-invoice))
9    response))

The function for data validation is quite simple. I’m however unhappy that I specify my desired data inside the function, I would really prefer to specify a complete data model elsewhere, but for this simple example this works.

1(defun validate-invoice-request (alist)
2  (flet ((entry (key) (cdr (assoc key alist))))
3    (list
4     :value (dfv:parse-input '(integer :min 1 :max 400000) (entry :value))
5     :memo (dfv:parse-input '(string :max-length 64) (entry :memo))
6     :private t
7     :expiry 600
8     :is_amp t)))

The request to LND is well encapsulated in the request-invoice function.

1(defun request-invoice (content)
2  (dex:post (format nil "~a/invoices" *lnd-api-endpoint*)
3            :headers `(,*auth-header*
4                       ("Content-Type" . "application/json"))
5            :content (cl-json:encode-json-plist-to-string content)
6            :insecure t))

You might have noticed that at this point, all the functions only consider the happy path, where every input is good and produced the desired result. During prototyping, I don’t really care about exception handling. I deal with that as I develop.

Frontend: Updating DOM according to the response

Now that the frontend has a response, the sincere nightmare of dealing with a user interface begins, things must change dynamically. The reply is JSON and once into a JavaScript object, There is a single key I care about: payment_request. It is a string I want to display and generate a QR code for it too.

To generate the QR code, I use the library from Project Nayuki , which is simple and without dependencies. Yet, you must still implement the renderer. Following their examples, I wrote one in Parenscript to render on a HTML canvas.

 1(in-package :ln-pay-js)
 2(defun qr-canvas (payload canvas &key (scale 4) (border 0)
 3                                   (light-color "#FFFFFF")
 4                                   (dark-color "#000000"))
 5  "Draws the QR Code on CANVAS with given PAYLOAD
 6
 7Renders the CANVAS of width and height equal to:
 8  (qr.size + BORDER * 2) * SCALE.
 9
10The drawn image is purely DARK-COLOR and LIGHT-COLOR, and fully opaque.
11
12The SCALE must be a positive integer and the BORDER must be
13  a non-negative integer."
14  (when (or (<= scale 0) (< border 0))
15    (throw (new (-range-error "Value out of range"))))
16  (let* ((ctx (chain canvas (get-context "2d")))
17         (qr (chain qrcodegen -qr-code
18                    (encode-text payload
19                                 (@ qrcodegen -qr-code -ecc "MEDIUM"))))
20         (width (* scale (+ (@ qr size) (* 2 border)))))
21    (setf (@ canvas width) width)
22    (setf (@ canvas height) width)
23    (loop for y from (- border) below (- width border) do
24      (loop for x from (- border) below (- width border) do
25        (setf (@ ctx fill-style) (if (chain qr (get-module x y))
26                                     dark-color light-color))
27        (chain ctx (fill-rect (* scale (+ x border))
28                              (* scale (+ y border))
29                              scale scale))))))

I want to place the QR code inside a modal, it should pop up and cover the screen. Design the modal again in isolation until you are happy with the design. Once the html is ready, I convert it into lisp form. The reason is I want to serve it directly from the JavaScript client.

 1(in-package :ln-pay-js)
 2(defun payment-modal-html ()
 3  (who-ps-html
 4   (:div :id "qr-send"
 5         :class "fixed top-0 left-0 w-100 h-100 flex items-center justify-center bg-black-80"
 6         (:div :class "modal-background absolute top-0 left-0 w-100 h-100")
 7         (:div :class "w-60-l relative"
 8               (:div :class "flex items-center justify-between pa3 bb b--moon-gray bg-near-white br3 br--top"
 9                        (:legend :class "f4 fw6 ph0 mh0"
10                                 "Tip me with"
11                                 (:i :class "fas fa-flask green") "Testnet"
12                                 (:i :class "fas fa-bolt gold") "Lightning"
13                                 (:i :class "fab fa-bitcoin green") "Bitcoin")
14                        (:button :id "close-modal"
15                                 :class "f4 bg-transparent bn pointer" "x"))
16               (:div :id "invoice-body" :class "pa4 bg-white tc"
17                         (:canvas :id "qr-canvas")
18                         (:div :class "bg-moon-gray mv2 relative overflow-hidden br2"
19                               (:div :class "progress-bar indeterminate h1 ma0 br3 bg-green"))
20                         (:textarea :id "ln-req"
21                                    :class "mw9 w-100 mv2 h5"
22                                    "Generate an invoice first."))
23               (:div :class "flex pa3 bt b--moon-gray bg-near-white br3 br--bottom")))))

I also add to the CSS file the styling for an indeterminate progress bar.

 1.progress-bar {
 2  background-image: linear-gradient(
 3    -45deg,
 4    rgba(255, 255, 255, 0.35) 25%,
 5    rgba(255, 255, 255, 0) 25%,
 6    rgba(255, 255, 255, 0) 50%,
 7    rgba(255, 255, 255, 0.35) 50%,
 8    rgba(255, 255, 255, 0.35) 75%,
 9    rgba(255, 255, 255, 0) 75%,
10    rgba(255, 255, 255, 0)
11  );
12  background-repeat: repeat-x;
13  background-size: 2rem 2rem;
14  animation: stripes 2s linear infinite;
15}
16
17@keyframes stripes {
18  to {
19    background-position: 20px 0;
20  }
21}
22.progress-bar.indeterminate {
23  position: relative;
24  animation: progress-indeterminate 5s linear infinite;
25}
26
27@keyframes progress-indeterminate {
28  from {
29    left: -35%;
30    width: 35%;
31  }
32  to {
33    left: 100%;
34    width: 25%;
35  }
36}

From this moment on I truly missed using ClojureScript and Reframe. Yet my solution is as follow. Always clear all DOM state to have a clean workplace. Use the function that generates the HTML text and inject it into an element using the insert-adjacent-h-t-m-l function. Hook new event listeners to clear elements that were just added. Inject into the modal the payment_request string and render the QR code in the designated canvas node.

 1(in-package :ln-pay-js)
 2(defun clear-state-nodes ()
 3  "Remove from DOM all nodes which represent state from previous request.
 4That is error boxes and modal containers of payment."
 5  (chain document (query-selector-all ".invoice-error,#qr-send")
 6         (for-each (lambda (x) (chain x (remove))))))
 7
 8(defun payment-modal (node-id)
 9  "Dress NODE-ID element with a payment modal.
10   Add corresponding close events"
11  (clear-state-nodes)
12  (chain document (get-element-by-id node-id)
13         (insert-adjacent-h-t-m-l "beforeend" (payment-modal-html)))
14  (chain document (query-selector-all ".modal-background,#close-modal")
15         (for-each
16          (lambda (node)
17            (chain node
18                   (add-event-listener "click" #'clear-state-nodes))))))
19
20(defun load-qr (lnreq)
21  (payment-modal "app")
22  (let ((text-con (chain document (get-element-by-id "ln-req")))
23        (canvas (chain document (get-element-by-id "qr-canvas")))
24        (payment-request (@ lnreq payment_request)))
25    (setf (@ text-con value) payment-request)
26    (qr-canvas payment-request canvas)))

It is of course necessary to update the invoice function to handle the response from the backend and use these functions.

 1(in-package :ln-pay-js)
 2(defun invoice ()
 3  (clear-state-nodes)
 4  (chain
 5   (fetch (stringify +ln-api-uri+ "/make-invoice")
 6          (create :method "POST"
 7                  :headers (create "Content-Type" "application/json")
 8                  :body (chain -J-S-O-N (stringify (collect-invoice-data)))))
 9   (then (lambda (resp)
10           (if (eq 200 (@ resp status))
11                 (chain resp (json))
12                 (throw (@ resp status-text)))))
13   (then #'load-qr)))

At this point, it works for us. The user sees the modal pop up with the invoice request. He could pay and close the modal himself. However, on our side, we only treated the happy path. Things could go wrong and handling those exceptions is necessary.

Handling errors

There are many things which can go wrong. Invalid User input, general bad input, the LND service could reply unexpected errors among other things. Let’s first handle those cases on the backend using a macro. Everything is taken care of at the same level, only changing the response message for the user. The operation either succeeds or fails as a bad request.

 1(defun err (message &rest args &key &allow-other-keys)
 2  (setf (hunchentoot:return-code*) 400)
 3  (cl-json:encode-json-plist-to-string
 4   `(:error ,message ,@args)))
 5
 6(defmacro handle-all-errors (&body body)
 7  `(handler-case
 8       (progn ,@body)
 9     (dex:http-request-failed ()
10       (err "Communication to Lightning Network failed"))
11     (dfv:invalid-format (c)
12       (err (dfv:invalid-format-reason c)))
13     ((or cl-json:json-syntax-error
14       cl-json:unencodable-value-error) ()
15       (err "Invalid JSON input"))
16     ((or type-error cl-base64:base64-error) ()
17       (err "Invalid input"))
18     (error (e)
19       (format t "ERROR: UNKNOWN ~A~%" e)
20       (err "Unknown error"))))
21
22(define-easy-handler (pay-req :uri "/make-invoice") ()
23  (setf (content-type*) "application/json")
24  (handle-all-errors
25    (multiple-value-bind (response)
26        (arrows:->> (raw-post-data :force-text t)
27                    (dfv:parse-input 'string)
28                    (cl-json:decode-json-from-string)
29                    (validate-invoice-request)
30                    (request-invoice))
31      response)))

On the front-end errors will appear inside error boxes instead of triggering the modal.

 1(in-package :ln-pay-js)
 2(defun signal-error (error-msg)
 3  (chain document (get-element-by-id "ln-payment")
 4         (insert-adjacent-h-t-m-l
 5          "beforeend"
 6          (who-ps-html
 7           (:div :class "invoice-error bg-washed-red ba br2 ma3 pa3 red"
 8                 (:div :class "message-body" error-msg))))))
 9
10(defun invoice ()
11  (clear-state-nodes)
12  (chain
13   (fetch (stringify +ln-api-uri+ "/make-invoice")
14          (create :method "POST"
15                  :headers (create "Content-Type" "application/json")
16                  :body (chain -J-S-O-N (stringify (collect-invoice-data)))))
17   (then (lambda (resp)
18           (let ((parsed (chain resp (json))))
19             (if (@ resp ok)
20                 parsed
21                 (chain parsed (then (lambda (json)
22                                       (throw (@ json error)))))))))
23   (then #'load-qr)
24   (catch #'signal-error)))

Listening to payment

Once error handling is properly set and the user is properly warned about errors, it is time to add the last convenience for the user. After he pays, the user interface must update. The waiting modal changes to thank the user for the payment and then closes.

The correct way would be to subscribe to LND for payment updates, and keep the user subscribed to the server for it to know when the payment has been registered. That is however more problems than I want to solve today. The easy solution is, with a given timeout, query the backend for the payment status. The backend in turn queries LND very frequently and as soon as the payment has settled it replies to the client. If the timeout is reached and the payment is not settled the backend still replies, yet with an OPEN state. On the client, this re-triggers a request, and so it keeps waiting for the update. Would the back end reply CANCELLED (invoice expired), then the modal changes to notify the user about that too.

This method saves requests between client and backend, and loads the backend. Because of that asymmetry it exposes the backed to unnecessary load. For the first iteration and as long as readers of this blog aren’t maliciously creating requests to my server, this is good enough. I monitor my server to track its health, and I have enough time to implement the correct solution latter on, should I have a lot of traffic, which is anyway a nice problem to have.

My solution on the front end is:

 1(in-package :ln-pay-js)
 2(defun clear-invoice-modal (success-p)
 3  (setf (inner-html (chain document (get-element-by-id "invoice-body")))
 4        (if success-p
 5            (who-ps-html
 6             (:div :class "center bg-washed-green green w-90 ma2 f4 br3"
 7                   (:div :class "pa3 bb b--moon-gray bg-green br3 br--top white fw6"
 8                         "Payment Completed!")
 9                   (:div :class "pa3"
10                         "Thank you for your kind support!")))
11            (who-ps-html
12             (:div :class "center bg-washed-red red w-90 ma2 f4 br3"
13                   (:div :class "pa3 bb b--moon-gray bg-red br3 br--top white fw6"
14                         "Invoice has expired!")
15                   (:div :class "pa3"
16                         "Please request a new one.")))))
17  (set-timeout #'clear-state-nodes (if success-p 1500 5000)))
18
19(defun json-or-error (response)
20  (let ((parsed (chain response (json))))
21    (if (@ response ok)
22        parsed
23        (chain parsed (then (lambda (json)
24                              (throw (@ json error))))))))
25(defun listen-for-payment (lnreq)
26  (chain
27   (fetch (stringify +ln-api-uri+ "/listen-payment")
28          (create :method "POST"
29                  :headers (create "Content-Type" "plain/text")
30                  :body (@ lnreq r_hash)))
31   (then #'json-or-error)
32   (then (lambda (resp)
33           (let ((state (@ resp :state)))
34             (cond
35               ((equal "CANCELED" state)
36                (clear-invoice-modal nil))
37               ((equal "SETTLED" state)
38                (clear-invoice-modal t))
39               ((listen-for-payment lnreq))))))
40   (catch (lambda (error)
41            (clear-state-nodes)
42            (signal-error error)))))
43
44(defun invoice ()
45  (clear-state-nodes)
46  (chain
47   (fetch (stringify +ln-api-uri+ "/make-invoice")
48          (create :method "POST"
49                  :headers (create "Content-Type" "application/json")
50                  :body (chain -J-S-O-N (stringify (collect-invoice-data)))))
51   (then #'json-or-error)
52   (then (lambda (response)
53           (load-qr response)
54           (listen-for-payment response)))
55   (catch #'signal-error)))

Notice that once again I extended invoice to now trigger the listen-for-payment event.

The back end is intentionally simple:

 1(defun invoice-state (hex-rhash)
 2  (arrows:->>
 3   (dex:get (format nil "~a/invoice/~a" *lnd-api-endpoint* hex-rhash)
 4            :headers `(,*auth-header*
 5                       ("Content-Type" . "application/json"))
 6            :insecure t)
 7   (cl-json:decode-json-from-string)
 8   (assoc :state) (cdr)))
 9
10(defun payment-state (hex-rhash &optional (repeat 30) (sleep 2))
11  (do ((request 1 (1+ request))
12       (state (invoice-state hex-rhash) (invoice-state hex-rhash)))
13      ((or (member state '("SETTLED" "CANCELED") :test #'equal)
14           (> request repeat)) state)
15    (sleep sleep)))
16
17(define-easy-handler (listen-payment :uri "/listen-payment") ()
18  (setf (content-type*) "application/json")
19  (handle-all-errors
20    (multiple-value-bind (response)
21        (arrows:->> (raw-post-data :force-text t)
22                    (dfv:parse-input 'string)
23                    (cl-base64:base64-string-to-usb8-array)
24                    (encode-hex)
25                    (payment-state)
26                    (list :state)
27                    (cl-json:encode-json-plist-to-string))
28      response)))

Doing it again: going to production

Now that all the moving pieces are in place, and everything works on my machine it is time to do it again. Review the entire code, change all hard coded values for environment variables. Refactor, create an executable, setup a service, include CORS headers because the payment processor is a different server than my blog server.

Serving the JavaScript is certainly the ugliest. During development I dynamically pushed JavaScript using the trident-mode, yet it is useful to serve it, because I do refresh from time to time the page. In the production environment, I use it in different ways. I have manually copied the compiled file to my blog, and on top of that I serve a frozen version of the file, with its server configuration.

 1(defun serve-js (env)
 2  (if (equal env "DEV")
 3      (push
 4       (hunchentoot:create-regex-dispatcher
 5        "/invoices\.js"
 6        (lambda ()
 7          (setf (hunchentoot:content-type*) "text/javascript")
 8          (ps:ps-compile-file
 9           (asdf:system-relative-pathname "ln-pay" "src/invoices.paren"))))
10       hunchentoot:*dispatch-table*)
11
12      (let* ((js-code #.(ps:ps-compile-file
13                         (asdf:system-relative-pathname
14                          "ln-pay" "src/invoices.paren")))
15             (api-uri (format nil "~a:~d" (uiop:getenv "LN_PAY_HOST")
16                              (uiop:getenv "LN_PAY_PORT")))
17             (conf-code (ps:ps (setf +ln-api-uri+ (ps:lisp api-uri))))
18             (out-file (concatenate 'string js-code conf-code)))
19        (push
20         (hunchentoot:create-regex-dispatcher
21          "/invoices\.js"
22          (lambda ()
23            (setf (hunchentoot:content-type*) "text/javascript")
24            out-file))
25         hunchentoot:*dispatch-table*))))

Starting the server has now more steps, all the hard coded variables are now queried from environment variables. I also support SSL on the production server. At the moment, I dare to expose the service to the wild, wild internet. That might change in the future and I’ll offer the services through a reverse proxy.

 1(defvar *auth-header*)
 2(defvar *lnd-api-endpoint*)
 3(defvar *server*)
 4(defun start-server (&key port)
 5  (setf *auth-header*
 6        (with-open-file (input (uiop:getenv "LND_MACAROON_PATH")
 7                               :element-type '(unsigned-byte 8))
 8          (let ((v (make-array 1024 :element-type '(unsigned-byte 8))))
 9            (read-sequence v input)
10            (cons "Grpc-Metadata-macaroon" (encode-hex v))))
11        *lnd-api-endpoint* (format nil "~a/v1" (uiop:getenv "LND_SERVER")))
12  (let ((host (quri:uri (uiop:getenv "LN_PAY_HOST"))))
13    (setf *ac-origin*
14          (format nil "~a://~a"
15                  (quri:uri-scheme host)
16                  (quri:uri-domain host))))
17  (setf *server* (if (equal (uiop:getenv "ENV") "DEV")
18                     (make-instance
19                      'hunchentoot:easy-acceptor
20                      :port port
21                      :document-root (asdf:system-relative-pathname
22                                      "ln-pay" "www/"))
23                     (make-instance
24                      'hunchentoot:easy-ssl-acceptor
25                      :port port
26                      :document-root (asdf:system-relative-pathname
27                                      "ln-pay" "www/")
28                      :ssl-certificate-file (uiop:getenv "SSL_CERT")
29                      :ssl-privatekey-file (uiop:getenv "SSL_KEY"))))
30  (serve-js (uiop:getenv "ENV"))
31  (hunchentoot:start *server*))

Because my blog and my payment processor service exist on separate domains I must configure CORS. I can solve this with a macro that wraps the definition of my API endpoints. Set the OPTIONS request for the preflight to respond with empty string. The else clause in the conditional is for the POST request, which is the actual request.

 1(defvar *ac-origin*)
 2(defmacro cors-handler (description lambda-list &body body)
 3  `(define-easy-handler ,description ,lambda-list
 4     (setf (header-out "Access-Control-Allow-Origin") *ac-origin*)
 5     (setf (header-out "Access-Control-Allow-Methods") "POST,OPTIONS")
 6     (setf (header-out "Access-Control-Max-Age") 86400)
 7     (setf (header-out "Access-Control-Allow-Headers") "Content-Type")
 8     (setf (header-out "Content-Type") "application/json")
 9     (setf (content-type*) "application/json")
10     (if (eq :options (hunchentoot:request-method*))
11         ""
12         (handle-all-errors ,@body))))
13
14(cors-handler (pay-req :uri "/make-invoice") ()
15  (multiple-value-bind (response)
16      (arrows:->> (raw-post-data :force-text t)
17                  (dfv:parse-input 'string)
18                  (cl-json:decode-json-from-string)
19                  (validate-invoice-request)
20                  (request-invoice))
21    response))
22
23(cors-handler (listen-payment :uri "/listen-payment") ()
24  (arrows:->> (raw-post-data :force-text t)
25              (dfv:parse-input 'string)
26              (cl-base64:base64-string-to-usb8-array)
27              (encode-hex)
28              (payment-state)
29              (list :state)
30              (cl-json:encode-json-plist-to-string)))

To run a standalone binary, you must incorporate an entry point. The main function starts everything, joins the webserver thread to the foreground, otherwise the executable will finish.

I also sneak a slynk server. This is the beauty of LISP. I can even after deployment connect to the running image and make changes to it if I need to. That is not an open connection to the wild internet, I must go over SSH to the server and then locally connect to it.

Once the webserver is on the main thread, it needs to listen to interrups to kill the server.

 1(defun main ()
 2  (start-server :port (parse-integer (uiop:getenv "LN_PAY_PORT")))
 3  (format *standard-output* "Hunchentoot server started.~&")
 4  (slynk:create-server
 5   :port (parse-integer (uiop:getenv "SLYNK_PORT")) :dont-close t)
 6  (handler-case (bt:join-thread
 7                 (find-if (lambda (th)
 8                            (search "hunchentoot" (bt:thread-name th)))
 9                          (bt:all-threads)))
10    ;; Catch a user's C-c
11    (#+sbcl sb-sys:interactive-interrupt
12     #+ccl  ccl:interrupt-signal-condition
13     #+clisp system::simple-interrupt-condition
14     #+ecl ext:interactive-interrupt
15     #+allegro excl:interrupt-signal
16     () (progn
17          (format *error-output* "Aborting.~&")
18          (hunchentoot:stop *server*)
19          (uiop:quit)))
20    (error (c) (format t "Woops, an unknown error occured:~&~a~&" c))))

Conclusion

You made it till the end, I’m glad you find this an interesting read. While learning Common Lisp I constantly struggled to find resources, this is my contribution to improve that.

After everything is done, it doesn’t look too bad. The code looks simple enough for a simple problem. The interactive programming model of LISP is incredibly helpful. Common Lisp beets Python, even as a beginner in Lisp and quite experienced in Python I already dare to say there is nothing Python as a Language does better. As an ecosystem, Python has every imaginable library, some domains like Data Science are even dominated by Python. There just a massive amount of work put into the Python ecosystem which you can leverage. That means, unless I’m really working on a complete novel algorithm I would prefer Python to Common Lisp. It is not the language is the libraries.

The true pain for me in this project was the JavaScript problem, and Python would have not been able to help me here either. Trying to keep the User Interface manageable from a software point of view is a hard problem, that is why so many JavaScript frameworks and tools exist. More interactions, immediate client side data validation, managing state, are necessary for any decent application and I skipped over them in this project. All those things feel incredibly hard when all you have is the bare JavaScript(ParenScript) language. Doing a Web Application without a Framework that helps you manage that complexity is a lost battle. From my experience ClojureScript+Reagent+Reframe is the most convenient developer experience, I fail to see why that hasn’t taken over the world.

Common Lisp, despite all its language quirks feels to me better than Clojure. Just by the fact that starting the SBCL image and loading new libraries at runtime is possible and fast, the developer experience is so much better. Clojure is just getting there. On top of that Sly, which is only an aggressive improvement over slime, which has years of development, is just better tooling to interact with the lisp image. CIDER has still a lot to catch up. They probably will, because the Clojure community is bigger. Until then, I’ll probably will keep jumping between Clojure and Common Lisp.

Maybe there is a hidden gem in the Common Lisp world to develop web clients, I personally coudn’t find it. Yes, there are some attempts at that, but they neither fit my mental model nor those models I already know. Yes, I’ll always want to learn new ones, but mentoring documentation is really hard to find in the land of Common Lisp all that you have is the HyperSpec, which is not beginner friendly. You can’t expect to change someones mental models without writing beginner friendly tutorials.